Redis+Keepalived高可用环境部署记录 – 散尽浮华 – 博客园

https://www.cnblogs.com/kevingrace/p/9001975.html

Keepalived 实现VRRP（虚拟路由冗余）协议，从路由级别实现VIP切换，可以完全避免类似heartbeat脑裂问题，可以很好的实现主从、主备、互备方案，尤其是无状态业务，有状态业务就需要额外花些功夫了。既然Mysql可以使用Keepalived很好的做到主从切换，那么Redis自然可以使用这种方式实现高可用。

Redis主从实现完全没有Mysql成熟，仅仅是可用而已，经过测试主从也不是那么完全不靠谱，主要问题在于同步连接断开之后需要重新全量同步，如果频繁进行会对主服务带来很大性能影响。但现实中主从机器往往要求放在一个机柜同一台交换设备下，网络闪断情况极低；再者主从同步在同步数量量大情况下，需要将缓存区调得足够大，不然也容易造成连接断开。
实现切换逻辑如下：A和B两台机器
1）A 、B机器依次启动，A机作为主、B机为从。
2）主A机挂掉，B机接管业务并作为主。
3）A机起来，作为从SLAVEOF B。
4）B机挂掉，A机再切回主。

在Keepalived 有两个角色：Master(一个)、Backup（多个），如果设置一个为Master，但Master挂了后再起来，必然再次业务又一次切换，这对于有状态服务是不可接受的。解决方案就是两台机器都设置为Backup，而且优先级高的Backup设置为nopreemt 不抢占。

部署记录：

0）服务器信息

192.168.10.205 redis-master 需要安装redis(3.2.0版本)、keepalived(1.3.2版本)

192.168.10.206 redis-slave 需要安装redis(3.2.0版本)、keepalived(1.3.2版本)

192.168.10.230 VIP

关闭两个节点机器的iptables和selinux（两个节点上都要操作）

[root@redis-master ~]

......

SELINUX=disabled

[root@redis-master ~]

Permissive

1）安装redis服务及主从配置（两个节点机都要操作）

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

[root@redis-master ~]

[root@redis-master src]

[root@redis-master redis-3.2.0]

添加相关文件及命令

[root@redis-master redis-3.2.0]

[root@redis-master src]

[root@redis-master redis-3.2.0]

添加redis启动脚本

[root@redis-master redis-3.2.0]

#!/bin/bash

REDISPORT=6379

EXEC=/usr/local/redis/bin/redis-server

REDIS_CLI=/usr/local/redis/bin/redis-cli

PIDFILE=/var/run/redis.pid

CONF="/etc/redis.conf"

case "$1" in

start)

if [ -f $PIDFILE ]

then

echo "$PIDFILE exists, process is already running or crashed"

else

echo "Starting Redis server..."

$EXEC $CONF

fi

if [ "$?"="0" ]

then

echo "Redis is running..."

fi

;;

stop)

if [ ! -f $PIDFILE ]

then

echo "$PIDFILE does not exist, process is not running"

else

PID=$(cat $PIDFILE)

echo "Stopping ..."

$REDIS_CLI -p $REDISPORT SHUTDOWN

while [ -x ${PIDFILE} ]

do

echo "Waiting for Redis to shutdown ..."

sleep 1

done

echo "Redis stopped"

fi

;;

restart|force-reload)

${0} stop

${0} start

;;

*)

echo "Usage: /etc/init.d/redis {start|stop|restart|force-reload}" >&2

exit 1

esac

添加执行权限

[root@redis-master redis-3.2.0]

设置开机自启动

[root@redis-master redis-3.2.0]

创建redis状态日志

[root@redis-master redis-3.2.0]

redis主从配置（先看下redis-master主节点的配置）

[root@redis-master redis-3.2.0]

.......

port 6379

.......

daemonize yes

.......

bind 0.0.0.0

.......

pidfile /var/run/redis.pid

.......

logfile /var/log/redis/redis.log

.......

dir /var/redis/redis

.......

appendonly yes

appendfilename "appendonly.aof"

appendfsync everysec

.....

save 900 1

save 300 10

save 60 10000 ＃即在多少秒的时间内，有多少key被改变的数据添加到.rdb文件里

.......

slave-serve-stale-data yes

slave-read-only yes

......

dbfilename dump.rdb ＃快照文件名称

......

另一个从节点redis-slave的redis.conf配置和上面基本差不多，只是多了下面一行配置：

slaveof 192.168.10.205 6379

接着创建redis的数据目录

[root@redis-master redis-3.2.0]

然后启动两个节点的redis服务

[root@redis-master redis-3.2.0]

Starting Redis server...

Redis is running...

[root@redis-master redis-3.2.0]

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

redis-ser 17265 root 4u IPv4 59068 0t0 TCP *:6379 (LISTEN)

2）Keepalived安装（两个节点机都要操作）

redis主从配置简单说明

redis的主从复制实现简单却功能强大，其具有以下特点：

1）一个master支持多个slave连接，slave可以接受其他slave的连接

2）主从同步时，master和slave都是非阻塞的

redis主从复制可以用来：

1）data redundancy（数据冗余）

2）slave作为master的扩展，提供一些read-only的服务

3）可以将数据持久化放在slave做，从而提升master性能

通过简单的配置slave（master端无需配置），用户就能使用redis的主从复制，即只需在slave端的redis.conf文件中配置下面一行：

slaveof <masterip> <masterport>

表示该redis服务作为slave，masterip和masterport分别为master 的ip和port

其他配置：

masterauth <master-password>

如果master设置了安全密码，则此处设置为相应的密码

slave-serve-stale-data yes 当slave丢失master或者同步正在进行时，如果发生对slave的服务请求：

slave-serve-stale-data设置为yes则slave依然正常提供服务

slave-serve-stale-data设置为no则slave返回client错误："SYNC with master in progress"

repl-ping-slave-period 10

slave发送PINGS到master的时间间隔

repl-timeout 60

IO超时时间

3）redis+keepalived配置

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

a）先进行redis-master主节点的高可用配置

[root@redis-master ~]

! Configuration File for keepalived

global_defs {

router_id redis-master

}

vrrp_script chk_redis {

script "/usr/local/keepalived/scripts/redis_check.sh 127.0.0.1 6379"

interval 2

timeout 2

fall 3

}

vrrp_instance redis {

state BACKUP

interface eth0

lvs_sync_daemon_interface eth0

virtual_router_id 202

priority 150

nopreempt

advert_int 1

authentication {

auth_type PASS

auth_pass 1111

}

virtual_ipaddress {

192.168.10.230

}

track_script {

chk_redis

}

notify_master "/usr/local/keepalived/scripts/redis_master.sh 127.0.0.1 192.168.10.206 6379"

notify_backup "/usr/local/keepalived/scripts/redis_backup.sh 127.0.0.1 192.168.10.206 6379"

notify_fault /usr/local/keepalived/scripts/redis_fault.sh

notify_stop /usr/local/keepalived/scripts/redis_stop.sh

}

b）接着进行redis-slave从节点的高可用配置

[root@redis-slave ~]

! Configuration File for keepalived

global_defs {

router_id redis-slave

}

vrrp_script chk_redis{

script "/usr/local/keepalived/scripts/redis_check.sh 127.0.0.1 6379"

interval 2

timeout 2

fall 3

}

vrrp_instance redis {

state BACKUP

interface eth0

lvs_sync_daemon_interface eth0

virtual_router_id 202

priority 100

nopreempt

advert_int 1

authentication {

auth_type PASS

auth_pass 1111

}

virtual_ipaddress {

192.168.10.230

}

track_script {

chk_redis

}

notify_master "/usr/local/keepalived/scripts/redis_master.sh 127.0.0.1 192.168.10.205 6379"

notify_backup "/usr/local/keepalived/scripts/redis_backup.sh 127.0.0.1 192.168.10.205 6379"

notify_fault /usr/local/keepalived/scripts/redis_fault.sh

notify_stop /usr/local/keepalived/scripts/redis_stop.sh

}

c）在redis-master和redis-slave两个节点机器上都要创建监控脚本（下面几个脚本，在两个节点上都要同样配置）

首先配置监控脚本

[root@redis-master ~]

#!/bin/bash

ALIVE=`/usr/local/redis/bin/redis-cli -h $1 -p $2 PING`

LOGFILE="/var/log/keepalived-redis-check.log"

echo "[CHECK]" >> $LOGFILE

date >> $LOGFILE

if [ $ALIVE == "PONG" ]; then :

echo "Success: redis-cli -h $1 -p $2 PING $ALIVE" >> $LOGFILE 2>&1

exit 0

else

echo "Failed:redis-cli -h $1 -p $2 PING $ALIVE " >> $LOGFILE 2>&1

exit 1

fi

需要注意的是：

以下负责运作的关键脚本：

notify_master /usr/local/keepalived/scripts/redis_master.sh

notify_backup /usr/local/keepalived/scripts/redis_backup.sh

notify_fault /usr/local/keepalived/scripts/redis_fault.sh

notify_stop /usr/local/keepalived/scripts/redis_stop.sh

因为Keepalived在转换状态时会依照状态来呼叫：

当进入Master状态时会呼叫notify_master

当进入Backup状态时会呼叫notify_backup

当发现异常情况时进入Fault状态呼叫notify_fault

当Keepalived程序终止时则呼叫notify_stop

温馨提示：
以上的keepalived.conf文件中的切换模式设置为nopreempt，意思是：
不抢占VIP资源，此种模式要是所有的节点都必须设置为state BACKUP模式！
需要注意无论主备服务器都需要设置为BACKUP，与以往KeepAlived的配置不同，其目的就是防止主服务器恢复后重新抢回VIP，导致Redis切换从而影响稳定。

100

101

接着在redis-master主节点上创建notity_master与notify_backup脚本：

[root@redis-master ~]

#!/bin/bash

REDISCLI="/usr/local/redis/bin/redis-cli -h $1 -p $3"

LOGFILE="/var/log/keepalived-redis-state.log"

echo "[master]" >> $LOGFILE

date >> $LOGFILE

echo "Being master...." >> $LOGFILE 2>&1

echo "Run SLAVEOF cmd ... " >> $LOGFILE

$REDISCLI SLAVEOF $2 $3 >> $LOGFILE 2>&1

sleep 10

echo "Run SLAVEOF NO ONE cmd ..." >> $LOGFILE

$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1

[root@redis-master ~]

#!/bin/bash

REDISCLI="/usr/local/redis/bin/redis-cli"

LOGFILE="/var/log/keepalived-redis-state.log"

echo "[BACKUP]" >> $LOGFILE

date >> $LOGFILE

echo "Being slave...." >> $LOGFILE 2>&1

echo "Run SLAVEOF cmd ..." >> $LOGFILE 2>&1

$REDISCLI SLAVEOF $2 $3 >> $LOGFILE

sleep 100

exit(0)

[root@redis-master ~]

#!/bin/bash

LOGFILE="/var/log/keepalived-redis-state.log"

echo "[fault]" >> $LOGFILE

date >> $LOGFILE

[root@redis-master ~]

#!/bin/bash

LOGFILE="/var/log/keepalived-redis-state.log"

echo "[stop]" >> $LOGFILE

date >> $LOGFILE

[root@redis-master ~]

total 20

-rwxr-xr-x. 1 root root 283 May 7 07:20 redis_backup.sh

-rwxr-xr-x. 1 root root 360 May 7 07:12 redis_check.sh

-rwxr-xr-x. 1 root root 102 May 7 07:22 redis_fault.sh

-rwxr-xr-x. 1 root root 445 May 7 07:16 redis_master.sh

-rwxr-xr-x. 1 root root 101 May 7 07:23 redis_stop.sh

将redis-master主节点上的上面5个脚本直接复制到redis-slave节点上即可。

[root@redis-master ~]

到redis-slave从节点上查看脚本：

[root@redis-slave ~]

total 20

-rwxr-xr-x. 1 root root 283 May 7 07:20 redis_backup.sh

-rwxr-xr-x. 1 root root 360 May 7 07:12 redis_check.sh

-rwxr-xr-x. 1 root root 102 May 7 07:22 redis_fault.sh

-rwxr-xr-x. 1 root root 445 May 7 07:16 redis_master.sh

-rwxr-xr-x. 1 root root 101 May 7 07:23 redis_stop.sh

d）设置环境变量（两个节点上都要设置）

[root@redis-master ~]

......

export PATH=$PATH:/usr/local/redis/bin

[root@redis-master ~]

e）启动两个节点上的keepalived服务

[root@redis-master ~]

Starting keepalived: [ OK ]

[root@redis-master ~]

root 32509 1 0 07:29 ? 00:00:00 keepalived -D

root 32510 32509 0 07:29 ? 00:00:00 keepalived -D

root 32512 32509 0 07:29 ? 00:00:00 keepalived -D

root 32515 32512 0 07:29 ? 00:00:00 keepalived -D

root 32517 32515 0 07:29 ? 00:00:00 /bin/bash /usr/local/keepalived/scripts/redis_backup.sh 127.0.0.1 192.168.10.206 6379

root 32529 14122 0 07:29 pts/1 00:00:00 grep keepalived

[root@redis-slave ~]

Starting keepalived: [ OK ]

[root@redis-slave ~]

root 22277 1 0 07:29 ? 00:00:00 keepalived -D

root 22278 22277 0 07:29 ? 00:00:00 keepalived -D

root 22279 22277 0 07:29 ? 00:00:00 keepalived -D

root 22283 22279 0 07:29 ? 00:00:00 keepalived -D

root 22284 22283 0 07:29 ? 00:00:00 /bin/bash /usr/local/keepalived/scripts/redis_backup.sh 127.0.0.1 192.168.10.205 6379

root 22289 10868 0 07:29 pts/1 00:00:00 grep keepalived

查看下redis-master主节点，发现vip资源已经有了

[root@redis-master ~]

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 52:54:00:b1:9c:93 brd ff:ff:ff:ff:ff:ff

inet 192.168.10.205/24 brd 192.168.10.255 scope global eth0

inet 192.168.10.230/32 scope global eth0

inet6 fe80::5054:ff:feb1:9c93/64 scope link

valid_lft forever preferred_lft forever

4）redis+keepalived主从高可用故障切换测试

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

a）分别启动redis-mastr和redis-slave两个节点的redis和keepalived服务（如上已启动）

b）尝试通过VIP连接Redis:

[root@redis-master ~]

role:master

[root@redis-master ~]

role:master

[root@redis-master ~]

role:slave

连接成功，Slave也连接上来了。

c）尝试插入一些数据：

[root@redis-master ~]

OK

从VIP读取数据

[root@redis-master ~]

"Redis"

从redis-master主节点读取数据

[root@redis-master ~]

"Redis"

从redis-slave从节点读取数据

[root@redis-master ~]

"Redis"

e）然后开始模拟故障产生：

将redis-master主节点上的redis进程杀死：

[root@redis-master ~]

root 4500 14122 0 08:04 pts/1 00:00:00 grep redis

root 17265 1 0 04:00 ? 00:00:07 /usr/local/redis/bin/redis-server 0.0.0.0:6379

[root@redis-master ~]

root 4514 14122 0 08:04 pts/1 00:00:00 grep redis

查看redis-master主节点上的Keepalived日志

[root@redis-master ~]

OK

[master]

Mon May 7 07:29:17 CST 2018

Being master....

Run SLAVEOF cmd ...

OK Already connected to specified master

Run SLAVEOF NO ONE cmd ...

OK

[fault]

Mon May 7 08:05:00 CST 2018

同时redis-slave从节点上的日志显示：

[root@redis-slave ~]

Being slave....

Run SLAVEOF cmd ...

OK

[master]

Mon May 7 08:05:02 CST 2018

Being master....

Run SLAVEOF cmd ...

OK Already connected to specified master

Run SLAVEOF NO ONE cmd ...

OK

然后我们可以发现，redis-slave从节点已经接管服务，并且担任Master的角色了。

redis-slave从节点上已经接管过来VIP资源了（大概需要等待2秒左右的时间，vip资源就切过来了）

[root@redis-slave ~]

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 52:54:00:dd:84:6b brd ff:ff:ff:ff:ff:ff

inet 192.168.10.206/24 brd 192.168.10.255 scope global eth0

inet 192.168.10.230/32 scope global eth0

inet6 fe80::5054:ff:fedd:846b/64 scope link

valid_lft forever preferred_lft forever

[root@redis-slave ~]

role:master

[root@redis-slave ~]

Could not connect to Redis at 192.168.10.205:6379: Connection refused

[root@redis-slave ~]

role:master

=======================================================================

然后再恢复redis-master主节点的redis进程

[root@redis-master ~]

/var/run/redis.pid exists, process is already running or crashed

Redis is running...

[root@redis-master ~]

Starting Redis server...

Redis is running...

[root@redis-master ~]

root 4969 1 0 08:08 ? 00:00:00 /usr/local/redis/bin/redis-server 0.0.0.0:6379

root 4977 4976 0 08:08 ? 00:00:00 /bin/bash /usr/local/keepalived/scripts/redis_backup.sh 127.0.0.1 192.168.10.206 6379

root 4987 14122 0 08:08 pts/1 00:00:00 grep redis

[root@redis-master ~]

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

redis-ser 4969 root 4u IPv4 93698 0t0 TCP *:6379 (LISTEN)

redis-ser 4969 root 6u IPv4 93709 0t0 TCP 192.168.10.205:43299->192.168.10.206:6379 (ESTABLISHED)

查看redis-master上的Keepalived日志

[root@redis-master ~]

OK Already connected to specified master

Run SLAVEOF NO ONE cmd ...

OK

[fault]

Mon May 7 08:05:00 CST 2018

[BACKUP]

Mon May 7 08:08:34 CST 2018

Being slave....

Run SLAVEOF cmd ...

OK

查看redis-slave上的Keepalived日志

[root@redis-slave ~]

Being slave....

Run SLAVEOF cmd ...

OK

[master]

Mon May 7 08:05:02 CST 2018

Being master....

Run SLAVEOF cmd ...

OK Already connected to specified master

Run SLAVEOF NO ONE cmd ...

OK

[root@redis-master ~]

role:master

[root@redis-master ~]

role:slave

[root@redis-master ~]

role:master

发现redis-master的redis服务再次启动后，redis-master主节点成为salve角色了，redis-slave从节点还是master角色。

当redis-slave节点宕机或redis服务关闭后，redis-master节点再次接管服务成为master角色，如此循环~~

关闭redis-slave从节点的reids服务

[root@redis-slave ~]

root 15407 1 0 04:00 ? 00:00:10 /usr/local/redis/bin/redis-server 0.0.0.0:6379

root 22900 10868 0 08:11 pts/1 00:00:00 grep redis

[root@redis-slave ~]

root 22902 10868 0 08:11 pts/1 00:00:00 grep redis

查看redis-slave上的Keepalived日志

[root@redis-slave ~]

.......

[stop] //测试时发现，当redis-slave的redis服务关闭后，还需要重启或关闭keepalived，才能将vip资源漂到redis-master节点上，所以日志里也就会出现这个stop信息

Mon May 7 09:25:03 CST 2018

[BACKUP]

Mon May 7 09:25:04 CST 2018

Being slave....

Run SLAVEOF cmd ...

OK

查看redis-master上的Keepalived日志

[root@redis-master ~]

.......

[master]

Mon May 7 09:25:03 CST 2018

Being master....

Run SLAVEOF cmd ...

OK Already connected to specified master

Run SLAVEOF NO ONE cmd ...

OK

查看redis-master，发现VIP资源已经接管过来了（如果没有按时切过来的话，只需重启或关闭redis-slave节点那边的keepalived服务即可）

[root@redis-master ~]

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 52:54:00:b1:9c:93 brd ff:ff:ff:ff:ff:ff

inet 192.168.10.205/24 brd 192.168.10.255 scope global eth0

inet 192.168.10.230/32 scope global eth0

inet6 fe80::5054:ff:feb1:9c93/64 scope link

valid_lft forever preferred_lft forever

[root@redis-master ~]

role:master

[root@redis-master ~]

role:master

[root@redis-master ~]

Could not connect to Redis at 192.168.10.206:6379: Connection refused

发现redis-maste节点已经转变为master角色了。

同样，当reids-slave节点的redis服务重新启动后，它将成为slave角色。

[root@redis-slave ~]

/var/run/redis.pid exists, process is already running or crashed

Redis is running...

[root@redis-slave ~]

Starting Redis server...

Redis is running...

[root@redis-slave ~]

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

redis-ser 23244 root 4u IPv4 3049509 0t0 TCP *:6379 (LISTEN)

redis-ser 23244 root 6u IPv4 3049513 0t0 TCP dns.kevin.cn:44931->192.168.10.205:6379 (ESTABLISHED)

[root@redis-master ~]

role:master

[root@redis-master ~]

role:master

[root@redis-master ~]

role:slave

***************当你发现自己的才华撑不起野心时，就请安静下来学习吧***************

Log@X.X.B

奇客の遊園地

Redis+Keepalived高可用环境部署记录 – 散尽浮华 – 博客园

October 2018
M	T	W	T	F	S	S
« Sep
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30	31