Failsafe Cluster for critical apps

Fails safe clusters are becoming critical components of the

datacenter, in an effort to maintain uptime. Basically these are nothing but a

set of servers who keep a continuous eye on each other and if by any case, one

goes down, the other starts doing its job. To the outside world the job at hand

continues uninterrupted. For instance in a  failsafe cluster, if you have a

web server running with the IP address 10.10.10.1, and it goes down, then

immediately another machine in the cluster will be alerted to this fact, and

it will change its IP to 10.10.10.1 and start acting like the server that went

down, so that web pages continue to be served uninterrupted.

Configuring the network interfaces of your failsafe cluster.

This article is about setting up such a cluster using

PCQLinux 2006. The software and prerequisites are included in PCQLinux 2006.

Before you start



In this article, we will explain how to create a two node fail-safe cluster

using Heartbeat. So, the first thing which you will need is two machines: one to

be the master node and the other to be the backup node. We set the fully

qualified domain name of our master node as master.pcquest.local and the name of

the backup node as “backup.pcquest.local'.  You can get the two machines

networked for failsafe clustering using two network cards on each machine and a

cross-wired CAT 5 cable. Obviously, this is just for demonstration and as you

add more machines to the cluster, you will need a more elaborate clustering

network.  Two of the network cards, one on each machine, will connect the

nodes to the network, while the other two will connect the nodes to each other

with the cross-wired cable. This will be used for determining the Heartbeat.

This is the one that senses when a machine in the cluster is down and brings the

other up.

Once you have the hardware up, install PCQLinux 2006 on

both the machines, choosing installation type 'Advanced Installation'. On

the next screen, select the Fail Safe Cluster option and follow the installation

normally. The installation will ask for the first and the second CD only. Let's

say, the IP address of your master node's network card, which is connected to

the other failsafe node with the cross-wire is 10.10.10.1 (/dev/eth1) and for

the other node it is 10.10.10.2. And on the network cards, which are connected

to the main network you can either take the IP address from the DHCP server or

you can give the IP address manually. We gave 192.168.3.27 (/dev/eth0) to public

network interface on the master node and 192.168.3.28 to the public network

interface to the backup node.  Now, you will need to configure an alias for

the public network card of the master node, (which is connected to the main

network) and give another IP address to it. To do so, start the

redhat-config-network applet and add a new LAN card. Here, when asked, select

the same network card which is connected to your main network. This card will be

given a device name, such as eth0:0. Assign an IP address to it such as

192.168.3.199. But make sure the IP is free and not taken up by any other

machine in the network. This will be the IP address of your cluster and your

service will respond to it.

Configuring the cluster



This involves modifying three files: /etc/ha.cf, /etc/haresources and /etc/authkeys.

But the default files are copied to /usr/ share/doc/heartbeat-2.0.2/. So, you

have to first copy all these files from here to /etc and then start modifying

them. The first file called ha.cf consists of all the settings for your nodes.

You have to modify the following tags in this file for both machines.

node master



node backup


deadtime 30


warntime 10


bcast eth0


auto_failback on

If you have more than one backup node in the cluster, then in the ha.cf file

on each machine, you will enter the full list of machines in the cluster

node master



node backup


node backup_two


node backup_three







and so on

Next open haresources from /etc and provide the IP address

as follows:

master.pcqlabs.net

192.168.3.199 httpd smb

This file should be identical in both machines. You will

notice that only one line is added, in which we are assigning the DNS name of

the master node and IP address of the cluster and services, which will run over

Heartbeat. This means that now your network will have a failsafe cluster of

Apache and Samba running over 192.168.3.199 (the IP allocated to the alias of

the network card connected to the main network). Finally, copy the file authkeys

from the documents directory to the /etc directory in both machines and restart

them.

Testing the cluster



Start up the servers and once they are fully functional, just shut down your

master node and wait for 30 seconds. Then go to the backup node and run the

ifconfig command. You will find that the alias eth0:0 is already created and the

same IP (192.168.3.199) is assigned to it.

Note: In this article we haven't discussed how to configure

your individual services such as SMB and httpd. For more details on them you can

read the article setting up Network Services, page 56, PCQuest, March 2004. The

only thing to keep in mind is that you need exactly the same configurations for

all the services in all the nodes. Plus, this setup will only create the backup

of the IP settings of the server. So you have to have some machinists to

replicate all the crucial data in all the cluster nodes as well. for instance if

you have a File server running in Samba and you are using it over heartbeat for

failsafe then you also have to have an exact replication of the directory

structure of the shares and user rights. So when the master node fails the data

will also be available n the backup node/s. For this you can either use rsync

over crond or use any other backup mechanism. For more details on rsync read Aug

2003 issue of PCQuest.