Month: December 2013

redis sentinel with ruby (on rails)

by Tom

In the last article I introduced how to install and use redis sentinel. As I’m using ruby, I need to use this new redis configuration with ruby (on rails).

For ruby on rails use redis-sentinel gem.

Then your redis initializer will look like

sentinels = [
  { host: '10.0.0.1', port: 17700 },
  { host: '10.0.0.2', port: 17700 },
  { host: '10.0.0.3', port: 17700 },
  { host: '10.0.0.4', port: 17700 }
]
# redis master name from sentinel.conf is 'master'
Redis.current = Redis.new(master_name: 'master', sentinels: sentinels)

You can use your redis then as usual.

When using sidekiq, configuration is pretty simple too

require 'sidekiq/web'
require 'redis-sentinel'
require 'sidetiq/web'

rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
rails_env = ENV['RAILS_ENV'] || 'development'

sentinels = [
  { host: '10.0.0.1', port: 17700 },
  { host: '10.0.0.2', port: 17700 },
  { host: '10.0.0.3', port: 17700 },
  { host: '10.0.0.4', port: 17700 }
]

redis_conn = proc { 
  Redis.current = Redis.new(master_name: 'master', sentinels: sentinels) 
}
redis = ConnectionPool.new(size: 10, &redis_conn)

Sidekiq.configure_server do |config|
  config.redis = redis
end

Sidekiq.configure_client do |config|
  config.redis = redis
end

You can test your configuration. Run rails console and test with 

Loading production environment (Rails 3.2.16)
1.9.3p448 :001 > Redis.current.keys("*").count
 => 746
1.9.3p448 :002 > Redis.current
 => #<Redis client v3.0.5 for redis://10.0.0.2:6379/0>

if you see “127.0.0.1:6379”, something is probably wrong. Then try to set/get some key and check Redis.current once again.

redis sentinel setup

by Tom

Prerequisities

  • multiple clients with redis 2.8.2+ installed

Do I need sentinel? If you want to have some kind of redis failover (there’s no cluster yet) – yes. Sentinels continuously monitor every redis instance and change configuration of given redis node(s) – if specified number of sentinels decided whether master is down, then they elect and promote new master and set other nodes as a slave of this master.

Looks interesting? Yes. It is. But. There’s a little time gap between electing and switching to the new master. You have to resolve this on application level.

Basically. Initial setup expects all nodes running as a master, with manual set slaveof ip port in redis-cli on meaned redis slaves. Then run sentinel and it does the rest.

sample redis configururation files follow:

daemonize yes
pidfile /usr/local/var/run/redis-master.pid
port 6379
bind 10.0.0.1
timeout 0
loglevel notice
logfile /opt/redis/redis.log
databases 1
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename master.rdb

dir /usr/local/var/db/redis/
slave-serve-stale-data yes
slave-read-only no
slave-priority 100
maxclients 2048
maxmemory 256mb

# act as binary log with transactions
appendonly yes

appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes

client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60

and sentinel configuration file:

port 17700
daemonize yes
logfile "/opt/redis/sentinel.log"

sentinel monitor master 10.0.0.1 6379 2
sentinel down-after-milliseconds master 4000
sentinel failover-timeout master 180000
sentinel parallel-syncs master 4

Start all of your redis nodes with redis config and choose master. Then run redis console and set all other nodes as a slave of given master, using command slaveof 10.0.0.1 6379. Then you can connect to your master and verify, if there are all of your slave nodes, connected and syncing – run info command in your master redis console. Output should show you something like this

- snip -

# Replication
role:master
connected_slaves:3
slave0:ip=10.0.0.2,port=6379,state=online,offset=17367254333,lag=1
slave1:ip=10.0.0.3,port=6379,state=online,offset=17367242971,lag=1
slave2:ip=10.0.0.4,port=6379,state=online,offset=17367222381,lag=1

- snip-

To test, if your sentinel works, just shutdown your redis master and watch sentinel log. You should see something like this

[17240] 04 Dec 07:56:16.289 # +sdown master master 10.24.37.144 6379
[17240] 04 Dec 07:56:16.551 # +new-epoch 1386165365
[17240] 04 Dec 07:56:16.551 # +vote-for-leader 185301a20bdfdf1d5316f95bae0fe1eb544edc58 1386165365
[17240] 04 Dec 07:56:17.442 # +odown master master 10.0.0.1 6379 #quorum 4/2
[17240] 04 Dec 07:56:18.489 # +switch-master master 10.0.0.1 6379 10.0.0.2 6379
[17240] 04 Dec 07:56:18.489 * +slave slave 10.0.0.3:6379 10.0.0.3 6379 @ master 10.0.0.2 6379
[17240] 04 Dec 07:56:18.490 * +slave slave 10.0.0.4:6379 10.0.0.4 6379 @ master 10.0.0.2 6379
[17240] 04 Dec 07:56:28.680 * +convert-to-slave slave 10.0.0.1:6379 10.0.0.1 6379 @ master 10.0.0.2 6379

explained line by line

+sdown master master 10.24.37.144 6379

master is subjectively down (maybe)

+odown master master 10.0.0.1 6379 #quorum 4/2

master is objectively down (oh, really), two of four sentinels have the same opinion

+switch-master master 10.0.0.1 6379 10.0.0.2 6379

so we switch to another master – chosen 10.0.0.2

+slave slave 10.0.0.3:6379 10.0.0.3 6379 @ master 10.0.0.2 6379

reconfigure 10.0.0.3 as a slave of new master 10.0.0.2

+convert-to-slave slave 10.0.0.1:6379 10.0.0.1 6379 @ master 10.0.0.2 6379

sorry, former master, you have to serve as a slave now

+sdown, -odown? + means ‘is’, – means ‘is no longer’. Then “+sdown” can be translated as “is subjectively down” and “-odown” like “is no longer objectively down”. Simple, huh? :)

PS: take my configuration files as a sample. Feel free to modify to match your need and check redis/sentinel configuration docs to get deeper knowledge about configuration options.

Kaštanový koláč Gordona Ramseye

by Tom

Ingredience

  • 250g loupaných, vařených kaštanů
  • 250g kvalitní hořké čokolády
  • 250g másla, nesoleného
  • 125g cukru (krupice, krystal)
  • 4 vejce
  • 125ml smetany
  • 125ml plnotučného mléka

Příprava je primitivní. Ale musí se dělat pečlivě :)

Máslo a čokoládu rozehřejeme ve vodní lázni. Cukr utřeme se žloutky. Smetanu, mléko a nakrájené kaštany dáme vařit – jakmile směs přejde do varu, sundáme a rozmixujeme. Ušleháme sníh ze 4 bílků. Čokoládovou směs, cukr a kaštany smícháme a vymícháme dohladka. Opatrně vmícháme ušlehaný sníh. Dáme péct do vyhřáté trouby (170oC) na 20-30minut.

Po pečení vyndáme, necháme chvíli chladnout – dort se stáhne a popraská – a podáváme buď ještě teplé nebo necháme úplně vychladnout.

Pár tipů:
Pokud máte vejce v lednici, nechte je pár hodin ohřát a stabilizovat se. Sníh musí být lesklý a pevný – ideálně šlehat v kovové misce, přidat špetku soli.

20131219-222524.jpg