故障演练

基础环境

主机名	SID	角色
master1	1	Follower
master2	2	Leader
node1	3	Follower

直接删除 node1 服务数据，看会出现什么效果？

删除前。

--------启动 master1 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
 --------启动 master2 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader
 --------启动 node1 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower

执行删除。

rm -rf /home/bigdata/module/apache-zookeeper-3.5.7-bin/data

执行下面步骤看是否有影响。

create /mynode1 "Node Data"
create /mynode2 "Node Data"
create /mynode3 "Node Data"

get -s /mynode3

删除以后对基本操作看着没有影响，然后进行集群的关闭重启。

 --------停止 master1 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
 --------停止 master2 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
 --------停止 node1 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Stopping zookeeper ... no zookeeper to stop (could not find file /home/bigdata/module/apache-zookeeper-3.5.7-bin/data/zookeeper_server.pid)

由于刚才node1删除了data,data下面有zookeeper_server.pid，导致node1没有关闭，那么手动关闭下。

netstat -tulnp | grep 2181

kill -9 15952

然后重启集群。

 --------启动 master1 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
 --------启动 master2 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader
 --------启动 node1 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Error contacting service. It is probably not running.

恢复步骤

同步其他没有挂的节点数据到node1。

./node1xsync.sh /home/bigdata/module/apache-zookeeper-3.5.7-bin/data

修改 node 的 myid。

echo "3" > myid

启动 node1。

./zkServer.sh start

查看集群的状态。

 --------启动 master1 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
 --------启动 master2 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader
 --------启动 node1 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower

create /mynode5 "Node Data"
create /mynode6 "Node Data"
create /mynode7 "Node Data"

get -s /mynode7

恢复成功。

直接删除Leader节点，看会出现什么情况？

没有删除前。

 --------启动 master1 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
 --------启动 master2 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader
 --------启动 node1 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower

删除Leader节点。

rm -rf /home/bigdata/module/apache-zookeeper-3.5.7-bin/data
rm -rf /home/bigdata/module/apache-zookeeper-3.5.7-bin/log

客户端执行 sync 以后删除的数据没有落到磁盘。

[zk: localhost:2181(CONNECTED) 6] sync /

现在重启集群。可以看到leader变化了。

 --------启动 master1 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
 --------启动 master2 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Error contacting service. It is probably not running.
 --------启动 node1 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader

恢复步骤

还是上面删除node1的操作同步以后数据以后，修改myid重启集群，可以看到还是恢复成功了。

 --------启动 master1 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
 --------启动 master2 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
 --------启动 node1 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader

现在删除一个 follower 和一个 leader 情况会怎么样？

可以发现集群已经启动不了了。

ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Error contacting service. It is probably not running.
 --------启动 master2 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Error contacting service. It is probably not running.
 --------启动 node1 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Error contacting service. It is probably not running.

恢复步骤

同步最后有数据的data目录到所有的节点。

./xsync.sh /home/bigdata/module/apache-zookeeper-3.5.7-bin/data

修改myid。

echo "2" > myid
echo "3" > myid

可以看到启动成功

 --------启动 master1 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader
 --------启动 master2 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
 --------启动 node1 zookeeper-------
ZooKeeper JMX enabled by default
Using config: /home/bigdata/module/apache-zookeeper-3.5.7-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower

获取节点数据

get /mynode5
get /mynode6
get /mynode7

可以看到数据还是可以访问的。

[zk: localhost:2181(CONNECTED) 7] get /mynode5
Node Data
[zk: localhost:2181(CONNECTED) 8] get /mynode6
Node Data
[zk: localhost:2181(CONNECTED) 9] get /mynode7
Node Data
[zk: localhost:2181(CONNECTED) 10]

基础环境​

直接删除 node1 服务数据，看会出现什么效果？​

恢复步骤​

直接删除Leader节点，看会出现什么情况？​

恢复步骤​

现在删除一个 follower 和 一个 leader 情况会怎么样？​

恢复步骤​

基础环境

直接删除 node1 服务数据，看会出现什么效果？

恢复步骤

直接删除Leader节点，看会出现什么情况？

恢复步骤

现在删除一个 follower 和一个 leader 情况会怎么样？

恢复步骤