mongodb集群采用的replication set模式,至少3个节点以上。有个问题,当mongodb主节点down的时候,如何让ceilometer自动连接重新选举出来的主节点?
这里有两种解法,至于采用哪种,看需求。
1、采用vip来定位mongodb主节点
使用keepalived来起vip,这里有两个vip,vrrp_instance VI_1是专门给mongodb用的vip,vrrp_instance VI_2是OpenStack HA用的vip
vrrp script chk_mongo_primary脚本专门来检测该节点是不是mongo master节点,如果是的话权重+2,而且vrrp_instance VI_1的vip是可以抢占的;
当mongodb主节点发生切换,vrrp_instance VI_1的vip也会跟着切换。
keepalived.conf配置文件
! Configuration File for keepalived
vrrp_script chk_haproxy {
script "killall -0 haproxy"
interval 2
weight 2
}
# Define the script used to check if mongod is running
vrrp_script chk_mongod {
script "killall -0 mongod"
interval 2 # every two seconds
weight 2
}
# Define the script to see if the local node is the primary
vrrp_script chk_mongo_primary {
script "mongo --eval '(!!db.runCommand("ismaster")["ismaster"])?quit(0):quit(1)'"
interval 2 # every two seconds
weight 2
}
# Configuation for the virtual interface
vrrp_instance VI_1 {
interface br-ex
state node MASTER # SLAVE on the other nodes
priority 101 # 100 on other nodes
virtual_router_id 55
authentication {
auth_type PASS
auth_pass secret # Set this to some secret phrase
}
# The virtual ip address shared between the two nodes
virtual_ipaddress {
172.16.140.251/24
}
# Use the script above to check if we should fail over
track_script {
chk_mongod
chk_mongo_primary
}
}
vrrp_instance VI_2 {
interface br-ex
virtual_router_id 51
state BACKUP
priority 200
# if use it,the openstack api do not response normally
# use_vmac virtualmac
#
advert_int 1
dont_track_primary
nopreempt
authentication {
auth_type PASS
auth_pass password
}
virtual_ipaddress {
172.16.140.250/24
}
track_script {
chk_haproxy
}
notify /usr/local/bin/keepalivednotify.sh
}
2、使用haproxy的tcp-check来检测
haproyx mongodb配置
listen mongodb_cluster
bind openstack_vip:27017
option tcpka
option tcplog
option tcp-check
# MongoDB Wire Protocol
tcp-check send-binary 3a000000 # Message Length (58)
tcp-check send-binary EEEEEEEE # Request ID (random value)
tcp-check send-binary 00000000 # Response To (nothing)
tcp-check send-binary d4070000 # OpCode (Query)
tcp-check send-binary 00000000 # Query Flags
tcp-check send-binary 61646d696e2e # fullCollectionName (admin.$cmd)
tcp-check send-binary 24636d6400 # continued
tcp-check send-binary 00000000 # NumToSkip
tcp-check send-binary FFFFFFFF # NumToReturn
# Start of Document
tcp-check send-binary 13000000 # Document Length (19)
tcp-check send-binary 10 # Type (Int32)
tcp-check send-binary 69736d617374657200 # ismaster:
tcp-check send-binary 01000000 # Value : 1
tcp-check send-binary 00 # Term
tcp-check expect binary 69736d61737465720001 #ismaster True
server controller1 controller1:27017 check inter 2000 rise 2 fall 3
server controller2 controller2:27017 check inter 2000 rise 2 fall 3
这里有个问题,在haproxy监控页面上看到的非master节点都是down的,因为tcp-check的缘故。
上面send-binary为什么那么写?(用tcpdum抓包出来,拿到wireshark分析)
详情参考这里:https://blog.danman.eu/mongodb-haproxy/
http://serverfault.com/questions/625492/how-to-construct-a-mongodb-health-check-in-haproxy
最后发现mongodb的uri本身支持multi host,形如:mongodb://ceilometer:ceilometer@controller2,controller1,controller3/ceilometer?readPreference=primaryPreferred&replicaSet=rs0
详情参考mongodb官方:https://docs.mongodb.com/manual/reference/connection-string/