If you've ever tried setting up a standard Redis Replication cluster with 1 Master node handling read/write and 2 Slave nodes backing up data, the word Sentinel probably isn't unfamiliar to you. Sentinel is extremely easy to install and configure, and how it works is very easy to understand. Let's dive in.
What is Redis Sentinel?
Redis Sentinel is used to monitor nodes in a Redis cluster — nodes running the redis-sentinel service monitor the Master and Slave nodes in the Replication cluster, then log activity like restart, stop, start of the redis-server service.
Sounds nice, so what's the point of logging this? This log serves the Fail-over process. If you previously only installed a replication cluster, when the Master node has an issue (goes down), you'd have to log into the server and manually fix it by promoting some slave to Master. But with Sentinel, this service automates that for you.
After a certain amount of time, if Sentinel can't connect to the Master node by sending a PING command and not receiving a PONG back, it marks this Node as down. Next, to ensure accuracy, this node asks other Sentinel nodes "hey, do you also see this Master is down?". If a certain number of nodes agree that this Master is down (the number of nodes needed is determined by the quorum parameter in the config, with the formula number of sentinel nodes/2 + 1, equivalent to more than half), the Sentinel nodes together elect a new Master. If the number of nodes isn't enough as required, the process of electing a new Master gets postponed.
Okay, now let's figure out which Slave node is suitable to turn into Master. So among the remaining Slave nodes, which one gets chosen as Master? Based on what criteria? Fairly simple — choosing the next master is based on the slave-priority parameter configured in the config file /etc/redis/redis.conf; if this parameter isn't configured, Sentinel generates a random ID, and whichever node has a lower ID gets chosen as Master.
Installation and configuration
Installation
apt install redis-sentinel -y (Ubuntu and Debian)
yum install redis-sentinel -y (CentOS)Configuration
Configuring Sentinel is also fairly simple — here's my sample config:
daemonize yes
pidfile "/var/run/redis/redis-sentinel.pid"
logfile "/var/log/redis/redis-sentinel.log"
bind 0.0.0.0
port 26379
dir "/var/lib/redis"
sentinel monitor mymaster 10.5.9.198 6379 2
sentinel down-after-milliseconds mymaster 3000
sentinel failover-timeout mymaster 10000
sentinel auth-pass mymaster password
maxclients 4064
sentinel current-epoch 1Explaining the main parameters:
- bind: The address you want Sentinel to listen on (here 0.0.0.0 means you can access it from any IP)
- port: The port Sentinel runs on
- sentinel monitor: Declares the master node's address to monitor (the parameters in order are <cluster name> <Master_IP> <Master_Port> <quorum>)
- sentinel down-after-milliseconds: Declares the time after which Sentinel determines the Master node has gone down (here, after 3s of failing to connect to Master, Sentinel considers this node down).
- sentinel failover-timeout: The time after which the fail-over process gets canceled if not yet successful.
- sentinel auth-pass: If your Master Node has a password, add this config so it can be accessed — if there's no password, remove this line.
Trying it out
Once configured, Sentinel connects to the Master Node to get info about the Slave Node and Sentinel Node to monitor. Along with that, the Sentinel nodes also cross-monitor each other.
Let's try stopping the Master node — on the Sentinel nodes, use the command tail -f /var/log/redis/redis-sentinel.log to watch the fail-over process.
- First, Sentinel determines the master node is down, so it initiates a vote-for-leader
- It determines the number of sentinel nodes needed to vote for master, 3 > 2 (satisfied)
- It updates the config, proceeding to change the Master, switch-master, to node 10.5.9.126
- It updates the new Slaves for the Master node and the Master for the Slave nodes. The fail-over process finishes.
On top of that, redis sentinel can also monitor multiple masters at once using the command sentinel monitor <name> <MASTER_IP> <MASTER_PORT> <quorum>
Wrapping up
Sentinel is a simple yet extremely effective tool for doing Fail-over. By combining it with HAProxy + KeepAlived, which I walked through in a previous post, you can create a Redis cluster with high durability and fault tolerance.
Wishing you a successful setup!
References
If you're running into technical challenges or need help with systems, DevOps tools, I'm confident I can help. Get in touch at hoangviet.io.vn
Thank you all! Have a nice day!