Build a Windows Supercomputer

Before we actually get down to setting up a Windows cluster, let us first look at the architecture for the same.

A cluster is composed of a number of standard off-the-shelf PCs (these are called nodes, in clustering terminology). These nodes are connected to each other with a network interconnect. A server node allocates processing tasks to the client (computing) nodes and also manages them. The client nodes are invisible to the outside world as far as the cluster as a whole is concerned. Only the server node is connected to peripheral input-output devices.

The server node has the necessary parallel processing libraries/environment for the purpose. The client nodes will be loaded with their own copy of the OS. You will also need software to manage the clients remotely, which again resides on the server node. It is preferable to have the client nodes of the same hardware configuration, in a supercomputing cluster. This ensures faster rollout times (in terms of deploying OS and software images across nodes) and also easier management.

The parallel processing software (or middleware, as it is called) consists of libraries. The software has a scheduler, which does the task of dividing the computational task into smaller tasks that will be executed in parallel. These smaller tasks are placed in a queue. A dispatcher monitors the queue and dispatches the computational tasks to individual client nodes for processing. The client nodes communicate with the server node and even between themselves with bi-directional messaging (communication software). This messaging is achieved by programming libraries like MPI (Message Programming Interface), MPI-FM (FM stands for Fast Messages which is a low-level messaging layer) or any other means. Once allocated a task, each of the clients processes its share of the workload and then delivers back the result to the server node for viewing.

We will assume that the machines you will be using for a Windows cluster have Intel processors, that is, regular machines capable of functioning as stand-alone desktops. You can also opt for multi-processor server class machines. Also, these nodes have their own memory and hard disks.

What you need



Microsoft has put together a Computational Clustering Technical Preview Toolkit (CCTPT) that helps you deploy a Windows-based, production-ready, computational cluster. This is a four CD and one DVD kit that also help you evaluate the performance of such a cluster. We received the version of the toolkit, which includes the following:

CD 1: Windows XP Professional (120 day evaluation)



CD 2: Windows 2000 Advanced Server (120 day evaluation)


CD 3: Interix 2.2 (Full version)


CD 4: Third party software and white papers


- MPI/Pro 1.6.4 (MPI Soft ware Technology Inc.)


- ClusterController 1.5.1 (MPI Software Technol ogy Inc.)


- Visual Fortran 6.6.A Stan dard Edition (Compaq)


- Math Kernel Libraries 5.1 (Intel)


- Computational Cluster Monitor (Cornell Theory Center)


- PLAPACK (Open Source)


- White papers from Cor nell Theory Center (CTC)


DVD: Visual Studio .NET Professional (60 day evaluation)

The toolkit limits your (free) cluster size to 32 nodes because of MPI/ Pro license. Additional licenses can be purchased if needed. And, of course, in a production setup, you will have to use the full versions of all the software and not the trial versions.

Installation



We follow the architecture that we looked at in the beginning. Our cluster has three types of resources: a Domain Controller (server node), cluster nodes (clients), and workstations, plus networking for the nodes.

We used the following hardware for setting up our cluster. The domain controller was an IBM Netfinity PIII Xeon 550 MHz with 256 MB RAM. The clients used were an assembled P-II 233 with 128 MB RAM, a P II 300 Mhz with 196 MB RAM, and a HP Brio 400 (Celeron 400 with 128 MB RAM). All the three client machines are used for regular desktop work. The network inter- connect was 100 Mbps Ethernet running over a 3Com LinkBuilder switch.

Domain Controller



This machine, apart from being an authentication mechanism for cluster nodes and workstations, also runs the Scheduler and the Cluster Controller.

Install Windows 2000 Advanced Server on this machine. Once through with installing the OS, configure this machine to act as a domain controller for your cluster domain (say, CLUSTER). This would require the additional installation of MS DNS and Active Directory Services for the Server. Once installed, it is preferable to create a login account for the cluster and workstation nodes in your cluster domain, say, ‘clustermem’ with a password.

All workstation nodes and client nodes are to be made members of this domain later, so that there is a single authentication mechanism for resources for all of them. Please note clearly that a Domain configuration is required. Workgroups are not supported by the message passing mechanism we will be using.

Workstations



These are one or multiple machines (depending on the size of your cluster and the number of users. You wouldn’t want to run a large capacity cluster with only one user machine) that would be used for building programs to be run on the cluster. So, apart from the OS, they will have all the necessary development libraries and environment to build the programs.

Install Windows 2000 Professional (with Service pack 2) or Windows XP on the workstations. Let’s call them WS1, WS2 and so on. Once the installation of the OS is through, make them a member of the ‘CLUSTER’ domain that you set up in the previous step. Reboot and login with the cluster domain member account you created on the domain controller (clustermem). From your Domain Controller, go the Active Directory Users and Computers sub-menu under Administrative Tools menu. Check if the workstation shows up as logged in.

We will now install the necessary software for development. From the toolkit, install the following in this order:

MS Visual Studio .NET Professional



Visual Fortran (needed for building some sample programs and benchmarks)


MPI/ Pro (for message passing across the machines over the interconnect)


Intel Math Kernel Library


PLAPACK (Parallel Linear Algebra Package)

Client nodes



These are the nodes whose processing power will be utilized by the parallel processing jobs on your cluster. They just need to have an OS and some software to get a ‘job’ and return it.

Client nodes are machines installed with Windows 2000 Professional (Service Pack 2), Windows 2000 Server or Windows XP. These machines are also made a member of the CLUSTER domain. Let’s call them NODE1, NODE2 etc. The only other thing needed to be installed on them is MPI/ Pro. You will also need to install the .NET Framework on the client. This comes bundled with MPI/ Pro 1.6.4 and automatically prompts you to install it before

MPI/ Pro.

Cluster Controller



After making sure that all the machines in the domain are communicating with the controller and also with each other, we will install the Cluster Controller. This is the job scheduler for parallel jobs that will run on your cluster nodes.

The Cluster Controller is installed on the Domain Controller server. The workstations also need to be installed with the Cluster Controller commands to directly submit jobs for execution. The other way could be to copy over the builds to the Domain Controller and submit them from there. For the former, the Cluster Controller installation also makes a common shared NTFS volume on the Domain Controller. This volume needs to be mapped with the same drive name on all the machines in the domain. This may need to be manually checked in case some nodes have more local drives.

A piece of software that allows you to graphically monitor the performance and status of your cluster nodes is the Computational Cluster Monitor (CCM). This communicates with the Cluster Controller and can be installed on any machine where CC is installed (the Domain Controller being an obvious choice).

Running jobs on your on your cluster



Before running the jobs on parallel client nodes, you will need to specify and check the environment variables for the compilers. You can build MPI programs in Visual Studio .NET. For this, go to Tools, options and in the directory settings, add the path of your MPI/ Pro’s ‘include’ and ‘lib’ paths. The ‘examples’ directory in MPI/ Pro home directory has sample Visual Studio projects. Try compiling them and if errors are encountered, you may have to re-check the environment variable paths.

Details of how to specify environment variables for command-line operation are also available in the accompanying documentation. MPI/ Pro comes with extensive documentation and it is recommended to get a printout to understand the syntax.

Because MPI/ Pro is installed on all the client nodes as a service, it can be invoked by the command-line utility ‘mpirun’ from any Workstation. This command can also remotely handle logins for the client nodes once the users’ domain account password is registered. This is done using the ‘mpipasswd’ program with the ‘-r’ option. This stores the password in encrypted form and uses it to authenticate the process on the Cluster Controller and the client nodes.

‘mpirun’ can accept two descriptor files: a machines file and a process group file. The syntax for running the command is:

With ‘machines’ file - mpirun <-np #> <-mach_file | -mf > {program}

With proc group file —mpirun —pg_file | -pg

The essential difference between the two files is the information they contain. While the ‘machines’ file only has the names of the client nodes (ranked in Round Robin order), the ‘proc group’ file has the names, number of processes to run on each machine, name and path of the executable and the ranks each machine should be assigned. The list of command-line arguments that can be passed to ‘mpirun’ can be seen using the ‘-help’ option.

Another method of executing parallel jobs is using the Cluster Controller. As already mentioned, this is the resource manager and the job scheduler. MPI/ Pro can interact with the Cluster Controller on the domain controller server using the ‘-ccs’ option. If your domain password is registered with MPI/ Pro, using this option with ‘mpirun’ will dynamically obtain the list of machines from the Cluster Controller. There is no need to specify a ‘machines’ or ‘proc group’ file in such a case.

Monitoring your cluster



The Cluster Controller has lots of user commands to monitor the status of jobs running on the clients. Some of them are:

ccq — to show the running and queued jobs



cctypes — to list the cluster resources and availability


ccusage — to show the nodes status and jobs


(more examples and commands listed in the documentation for Cluster Controller)


The Cornell Computational Cluster Monitor (CCCM) is the graphical utility that does real-time monitoring of the cluster. This consists of two programs — CTCDmon and CCCM. The CTCDmon talks to the Cluster Controller and sends the output to the CCCM GUI. Start the CTCDmon server by executing the following at command prompt (from the appropriate install location, say
C:\CCCM)

CTCDmon /RegServer



Execute the CCCM application after that. This gives the status of all the cluster nodes (down, busy or free) and also monitors the memory and CPU usage of the nodes.

Ashish Sharma