Supercomputing in Linux

To keep it simple, we start with a cluster of three machines. One will be the server and the other two will be the nodes. However, plugging in additional nodes is easy and we will tell you the modification to accommodate additional nodes. Instead of two nodes, you can have a single node. So, even if you have two PCs, you can build a cluster. We suggest that you go through the article Understanding Clustering, page 42, which explains what a cluster is and what server and nodes mean in a cluster before you get started.

Set up server hardware



You should have at least a 2 GB or bigger hard disk on the server. It should have a graphics card that is supported by PCQLinux 7.1 and a floppy drive. You also need to plug in two network cards–preferably the faster PCI cards instead of ISA–supported by

PCQLinux. 

Why two network cards? Adhering to the standards for cluster setups, if the server node needs to be connected to the outside (external) network–Internet or your private network–the nodes in the cluster must be on a separate network. This is needed if you want to remotely execute programs on the server. If not, you can do away with a second network card for the external network. For example, at PCQ Labs, we have our regular machines plugged in the 192.168.1.0 network. We selected the network 172.16.0.0 for the cluster nodes. Hence, on the server, one network card (called external interface) will be connected to the Labs network and the other network card (internal interface) will be connected to a switch. We used a 100/10 Mbps switch. A 100 Mbps switch is recommended because the faster the speed of the network, the faster is the message passing. All cluster nodes will also be connected to this switch.

PCQLinux on server



If you already have a machine with PCQLinux 7.1, including the X Window (KDE or GNOME), installed you can use it as a server machine. In this case you may skip the following steps for installation. If this machine has a firewall (ipchains or iptables) setup, remove all strict restrictive rules, as it will hinder communication between the server and the nodes. The ‘medium’ level of firewall rules in PCQLinux is suitable. After the cluster set up, you may selectively enable the rules, if required. 

If you haven’t installed PCQLinux on the machine, opt for custom system install and manual partitioning. Create the swap and / (ROOT) partitions. If you are shown the 1024 cylinder limit problem, you may also have to create a /boot partition of about 50 MB. In the network configuration, fill in the hostname (say, server. cluster.net), IP address of the gateway/router on your network, and the IP of a DNS server (if any) running on your network. Leave other field to their defaults. We will set up the IP addresses for network cards after the installation. Select ‘Medium’ for the firewall configuration. We now come to the package-selection wizard. You don’t need to install all the packages. Besides the packages selected by default, select ‘Development’ and ‘Kernel Development’ packages. These provide various libraries and header files for writing programs and are useful if you will develop applications on the cluster. You will need the X Window system because we will use a graphical tool for cluster set up and configuration. By default, GNOME is selected as the Window Manager. If you are comfortable using KDE, select it instead. By suggesting that you select only a few packages for install, we aim at a minimal installation. However, if you wish to install other packages like your favorite text editor, network management utilities or a Web server, then you can select them. Make sure that you set up your graphics card and monitor correctly. 

After the installation finishes, reboot into PCQLinux. Log in as root. 

Set up OSCAR



Mount this month’s CD and copy the file oscar-1.2.1.tar.gz from the directory system/cdrom/ unltdlinux/linux on the CD to /root. Uncompress and extract the archive as:

tar -zxvf oscar-1.2.1.tar.gz

This will extract the files in a directory named oscar-1.2.1 within /root directory. 

OSCAR installs Linux on the nodes from the server across the network. For this, it constructs an image file from RPM packages. This image file is in turn picked up by the nodes to install PCQLinux onto them. The OSCAR version we’ve given on the CD is customized for RedHat 7.1. Though PCQLinux 7.1 is also based on RedHat 7.1, some RPMs with PCQLinux are of more recent versions than the ones required by OSCAR. OSCAR constructs the image out of a list of RPMs specified in sample.rpmlist in the subdirectory oscarsamples in oscar-1.2.1. You have to replace this file with the one customized for PCQLinux RPMs. We have given a file named sample.rpmlist on this month’s CD in the directory system/cdrom/unltdlinux /linux. Overwrite the file sample.rpmlist in the oscarsamples directory with this file. 

Copy PCQLinux RPMs to /tftpboot/rpm



For creating the image, OSCAR will look for the PCQLinux RPMs in the directory /tftpboot/rpm. Create a directory /tftpboot and a subdirectory named rpm within it

mkdir /tftpboot



mkdir /tftpboot/rpm

Next, copy all the PCQLinux RPMs from both the CDs to /tftpboot/rpm directory. Insert CD 1 (PCQLinux CD 1, given with our July 2001 issue) and issue the following commands:

mount /mnt/cdrom



cd /mnt/cdrom/RedHat/RPMS


cp *.rpm /tftpboot/ rpm 


cd


umount /mnt/cdrom

Insert CD 2 (given with the July 2001 issue) and issue the above commands again. 

Note. If you are tight at the disk space, you don’t need to copy all the RPMs to /tftpboot/rpm. You can copy only the RPMs listed in sample.rpmlist file. Copy only the required

RPMs. 

Copy required RPMs



Type the following in a Linux text editor and save the file as copyrpms.sh

#!/bin/bash



rpms_path=”/mnt/cdrom/RedHat/RPMS/”


rpms_list=”/root/oscar-1.2.1/oscarsamples/sample.rpmlist”


mount /mnt/cdrom


while read line


do file=”$rpms_path$line.i386.rpm”


if < -f $file >


then


cp $file /tftpboot/rpm


else file=”$rpms_path$line.noarch.rpm”


if < -f $file >


then


cp $file /tftpboot/rpm


else file=”$rpms_path$line.i586.rpm”


if < -f $file >


then


cp $file /tftpboot/rpm


else file=”$rpms_path$line.i686.rpm”


if < -f $file >


then


cp $file /tftpboot/rpm


fi


fi


fi 


fi 


done < $rpms_list 


eject

Give executable permissions to the file as:

chmod +x copyrpms.sh

Assuming that you have created the directory /tftpboot/rpm, insert PCQLinux CD 1 (don’t mount it) and issue:



./copyrpms

When all the RPMs from the CD are copied, the CD drive will eject. Next, insert CD 2 and issue ./copyrpms again. 

Fix glitch in PCQLinux 



On this month’s CD we have carried the zlib rpm–zlib-1.1.3-22.i386.rpm–which you can find in the directory system/cdrom/ unltdlinux/linux on the CD. (We had given this on our July CD as well, but the file was corrupt.) Install the RPM as:

rpm —ivh zlib-1.1.3-22.i386.rpm

Copy this file to /tftpboot/rpm directory. This will prompt you to overwrite the corrupted zlib RPM, already in the directory. Go for it. 

Set up networking



Linux names network cards or interfaces as eth0, eth1, eth2. In our case eth0 is the internal interface and eth1 is the external interface. We assign eth0, an IP address of 172.16.0.1. Since we are running a DHCP server on the PCQ Labs network, we will set eth1 to obtain IP address from the DHCP server. If you are using a single network card for the cluster network, skip setting up the second card. 

Launch X Window. Launch a terminal window within GNOME or KDE and issue the command netcfg. This will pop up a graphical network configurator. Click on the Interfaces tab. To set up the internal interface, click on eth0 and then on edit. For IP address, enter 172.16.0.1 and for the netmask enter 255.255.255.0. Click on ‘Activate interface at boot time’. For ‘Interface configuration protocol’ select ‘none’ from the drop-down list. 

To set up the external interface, select eth1 and click on edit. If you are running a DHCP server, select dhcp from the drop down list. Else, enter a free IP address (say, 192.168.1.23), the associated netmask (say, 255.255.255.0) and select none from the drop-down list. In either case, make sure to click on ‘Activate interface at boot time’. 

Highlight eth0 and click on the button ‘Activate’. Do the same for eth1. Finally, click on save and quit the

configurator.  

Issue the command, ifconfig to check whether the network interfaces are up and have been given the correct IP addresses.



You are now ready to start Oscar.

Run OSCAR



In the terminal window, change to oscar-1.2.2 directory and issue the command:

./install_cluster eth0

Replace eth0 with the name of the internal interface in your case. You will see text flowing in the window. After a couple of minutes, the graphical wizard of OSCAR will pop up. OSCAR installation calls cluster nodes as clients

Build image from RPMs



Click on ‘Build Oscar Client Image’. We assume that all the node machines will have IDE hard disks. If you are using SCSI hard disk in the nodes, you need to change the Disk Partition File. Refer to the OSCAR installation documentation on the CD. When finished, a message ‘Successfully created image oscarimage’ will pop up. 

Tell OSCAR about the nodes



Click on the button ‘Define OSCAR clients’. Here you should see the domain name, starting IP and subnet mast, pre-filled with cluster.net, 172.16.0.2 and 255.255. 255.0. With ‘Number of hosts’ you specify the number of nodes. As per the OSCAR documentation, OSCAR supports up to 100 nodes or may be more. But it hasn’t been experimented with arbitrary large number of nodes. In our case we fill in two. If you are experimenting with two machines, one server and the other the node, then fill in one. 

In OSCAR once you define the number of nodes you cannot change it after the cluster is installed. You need to again start from the beginning, ie, from the step when we issued ‘install_cluster‘

Note. If for any reason you need to start again, before issuing ./install_cluster, execute the script named start_over located in the subdirectory scripts as:

/root/oscar-1.2.1/script/start_over’

Clicking on the ‘Add clients’ button will show ‘Successfully created clients’ after a couple of seconds.

Set up the nodes 



Before carrying out the subsequent steps in OSCAR installation, connect the network cards of the node machines to the switch and set them up to boot from floppy from their BIOS.

Set up nodes to network



We come back to OSCAR installation wizard running on the server machine. Click on the button ‘Set up Networking’. In the right frame you will see a tree-like structure as shown in the screenshot. In our case, the two nodes are given a hostname of oscarnode1.cluster.net and oscarnode2. cluster.net. They are assigned IP addresses 172.16.0.2 and 172.16. 0.3 respectively. Next, we assign the MAC (Media Access Control) address of the nodes to the listed IP addresses. This can be done by booting the nodes using a floppy created by OSCAR or by networking booting them. For the latter refer to the OSCAR documentation given on the CD.

Click on the button ‘Build AutoInstall Floppy’. This will pop up a terminal window. Insert a blank floppy in the server and click ‘y’ to continue. After the terminal window disappears, click on the button ‘Collect MAC addresses’ in the OSCAR window. Insert the floppy in one of the node machines and power it on. The machine will boot from the floppy. Press enter at the boot: prompt. After some time, the MAC address of the node will show up in the left frame. Suppose we want to assign the IP address 172.16.0.1 to this node. Click on the MAC address in the left and on the ‘osacrnde1.cluster.net’ in the right frame. Then, click on ‘Assign MAC to node’. 

Assign IP addresses to the nodes of the cluster



Switch off the node machine. Now boot the second node machine from the same floppy. As before, the MAC address of the second node will appear in the left frame. Assign it to oscarnode2.

cluster.net. 

If you want to plug in more node machines, repeat the above process for them. When done, click on the button ‘Stop collecting’ on the OSCAR window. 

After shutting down all the node machines, click on the button ‘Configure DHCP Server’. Then click on the close button in the ‘MAC address collection’ window.

PCQLinux on the nodes



Next, boot the first node machine again from the floppy. This time the node machine will install PCQLinux 7.1 from the network. When done, a message, as following, will be shown:

I have done for … seconds. Reboot me already

Take out the floppy and reboot the node machine. This time it should boot from the hard disk. If everything has gone well, you will boot into PCOLinux 7.1. While booting, PCQLinux will detect and prompt you to set up hardware like mouse, graphics card, sound card etc on the nodes. 

Problem: No active partition



If you are shown an error during booting which says no active partition, then boot from a Windows bootable floppy or CD. Launch fdisk and select option2 (Set active partition). Set partition 1 of type non-dos and about 31 MB in size as active. This is the /boot partition from where the kernel boot image resides. 

Test networking of nodes



On the server, open another terminal window and issue:

/root/oscar-1.2.2/scripts/ping_clients

If there is no problem with the networking, you will be shown ‘All clients responded”. Else check whether all nodes are powered on, defects in network cables, hub/ switch ports etc. From now on, ideally, you don’t need to work physically on the node machines. Hence you can plug off the monitor, keyboard, mouse, etc from the node machines. If the node machines need to be accessed and worked upon, you should use SSH (Secure Shell), similar to telnet but secure, to access them from the server. 

All done



Click on ‘Complete Cluster Setup’ and then on ‘Test cluster Setup’. This will pop up a terminal window and prompt you to enter a non-root username. Enter ‘shekhar’ (say). If the user account does not exist on the server machine, it will be created. In the latter case, you will be prompted for a password for the new account. Click on the ‘Quit’ button on the OSCAR window. Reboot the server machine. 

Test the cluster



To test the cluster, log in as the user that you created above (shekhar in our case) and issue:

cd OSCAR_test



./text_cluster

Enter the number of nodes when prompted (two in our case). For the number of processors on each client enter 1 (assuming uniprocessor machines). The test verifies the running of PBS and runs example programs coded using LAM, MPICH, PVM libraries by dispatching them through PBS to the nodes. You can see pbs_mom (see Understanding Clustering, page 42) running on the nodes by issuing the command ‘ps —e | grep pbs_mom’ on the nodes. 

If there are no error messages in the output, congratulations, you have your supercomputer up and running. Our cluster setup qualifies to be called a Beowulf cluster because it has been built using easily available hardware, free and open-source software, the /home directory on the server is exported to all the nodes via NFS (you can check this by issuing the command ‘mount’ on the nodes), and finally the server and nodes can execute command and scripts remotely on each other via SSH. 



Using the libraries installed on the cluster, you can start developing or executing cluster-aware applications on the server. The compilers for them (like, gcc, g++) are same as with PCQLinux. 

Shekhar Govindarajan