high availabilty

I have used a tool called HAProxy which can redirect a web call to another node. However ... what happens if the node running HAPROXY Fails ?

HA Image

As the Application needs to be pointed at 1 Server ... this solution provides Load Balencing but does not provide redundancy.

Test

I have 2 Servers

  • 192.168.59.4 rmq1
  • 192.168.59.5 rmq2

I would like to access them via a common Ip say of

  • 192.168.59.10

This IP Address will be registered in the DNS Service (in this case /etc/hosts)

Disable the Firewall

There is a Multicast setting that I do not currently understand - so to mitigate this I am disabling the firewall

service firewalld stop

these are the commands that need to be understood

sudo ufw allow in from 198.51.100.20 to 224.0.0.18 # on 198.51.100.10
sudo ufw allow in from 198.51.100.10 to 224.0.0.18 # on 198.51.100.20 

Allow Network card to have shared IP Addresses

This needs to be done on both machines.

  • edit /etc/sysctl.conf
  • net.ipv4.ip_nonlocal_bind=1
sudo sysctl -p # reload config change

Install Keepalived

This needs to be done on both machines.

mount the RHEL distro - and then

yum install keepalived

There are a couple of dependancies ... just accept them. The output I see is like this.

Resolving Dependencies
--> Running transaction check
---> Package keepalived.x86_64 0:1.2.13-6.el7 will be installed
--> Processing Dependency: libnetsnmpagent.so.31()(64bit) for package: keepalived-1.2.13-6.el7.x86_64
--> Processing Dependency: libnetsnmpmibs.so.31()(64bit) for package: keepalived-1.2.13-6.el7.x86_64
--> Running transaction check
---> Package net-snmp-agent-libs.x86_64 1:5.7.2-20.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                 Arch       Version              Repository        Size
================================================================================
Installing:
 keepalived              x86_64     1.2.13-6.el7         InstallMedia     223 k
Installing for dependencies:
 net-snmp-agent-libs     x86_64     1:5.7.2-20.el7       InstallMedia     698 k

Transaction Summary
================================================================================
Install  1 Package (+1 Dependent package)

Total download size: 922 k
Installed size: 2.7 M
Is this ok [y/d/N]: Y
Downloading packages:
--------------------------------------------------------------------------------
Total                                              6.2 MB/s | 922 kB  00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : 1:net-snmp-agent-libs-5.7.2-20.el7.x86_64                    1/2 
  Installing : keepalived-1.2.13-6.el7.x86_64                               2/2 
  Verifying  : keepalived-1.2.13-6.el7.x86_64                               1/2 
  Verifying  : 1:net-snmp-agent-libs-5.7.2-20.el7.x86_64                    2/2 

Installed:
  keepalived.x86_64 0:1.2.13-6.el7                                              

Dependency Installed:
  net-snmp-agent-libs.x86_64 1:5.7.2-20.el7                                     

Complete!

KeepAlive Config

Each machine

  • 192.168.59.4 rmq1
  • 192.168.59.5 rmq2

Needs a different config file.... however the change is VERY VERY small.

Config for rmq1

Edit the file in /etc/keepalived/keepalived.conf on rmq1

vrrp_script chk_haproxy {      # Requires keepalived-1.1.13
       script "killall -0 haproxy"  # cheaper than pidof
       interval 2 # check every 2 seconds
       weight 2 # add 2 points of priority if OK
     }
     vrrp_instance VI_1 {
       interface eth0
       state MASTER
       virtual_router_id 51
       priority 101 # 101 on primary, 100 on secondary
       virtual_ipaddress {
         192.168.59.10 
       }
       track_script {
         chk_haproxy
       }
     }

Config for rmq2

Edit the file in /etc/keepalived/keepalived.conf on rmq1.

Note: If the RabitMq is the only interface you need.... then just have this defintion you need nothing else.

Please Change the Interface value very carefully

vrrp_script chk_haproxy {      # Requires keepalived-1.1.13
       script "killall -0 haproxy"  # cheaper than pidof
       interval 2 # check every 2 seconds
       weight 2 # add 2 points of priority if OK
     }
     vrrp_instance VI_1 {
       interface eth0
       state MASTER
       virtual_router_id 51
       priority 100 # 101 on primary, 100 on secondary
       virtual_ipaddress {
         192.168.59.10 
       }
       track_script {
         chk_haproxy
       }
     }

Post Keepalived Config

On both machines please restart the services by

service keepalived restart

And make sure they are ok by

service keepalived status

Keepalived SELinux Issues

When starting keepalived I noticed there were some Security issues.

I quickly did

 service keepalived restart
 service keepaived status -l

This showed me some exceptions. This means something bad is happening. It should start cleanly.

keepalived Check

If all has worked then keepalived will create a seperate IP Address for you service. You can see this IP address like this

ip addr show | grep 192

I see back

inet 192.168.59.4/24 brd 192.168.59.255 scope global enp0s8
inet 192.168.59.10/32 scope global enp0s8

It looks like it is working.

You need this on both nodes.

HAProxy Config

On both Machines

We now need to install HAProxy

yum install haproxy

Then we create a HAProxy for RabbitMq

global
daemon

defaults
mode tcp
maxconn 10000
timeout connect 5s
timeout client 100s
timeout server 100s

listen rabbitmq 192.168.59.10:5670
mode tcp
balance roundrobin
server rmq1 192.168.59.4:5672 check inter 5s rise 2 fall 3
server rmq2 192.168.59.5:5672 check inter 5s rise 2 fall 3

Now we Restart HAProxy

service haproxy restart

And Check it

service haproxy status

:"