Issue Editor for this month: The High Performance Reality

If you have been following PCQuest for the past one year, you must have

noticed that we are regularly talking about High Performance Computing or HPCs.

We have covered the different technologies, implementations and case studies on

this topic. You may ask why we are putting so much emphasis on it. This is

because we see a tremendous scope for HPC across all IT and non-IT enterprises.

Today, pioneers have already adopted this technology in some form or the other

across various industry verticals. With the help of this technology, they are

able to do a lot more, that was earlier considered almost impossible. BFSI/stock

exchanges and rendering/animation are the two key verticals where HPC is being

used significantly in India. But one more trend, which is quite prominent today,

is that most of these deployments are based on commodity hardware and software.

Anindya Roy Issue Editor for this month

And not just in India, but also across the globe, more than half of the

biggest HPC deployments are based on commodity hardware and software. More than

70% of the fastest supercomputers are working on cluster architecture (which is

mostly done with commodity hardware and interconnects), which was not the case

five years back when clustering was less than 10% and the bigger chunk of the

pie was dominated by MPP (massively parallel processing). Today, MPP doesn't

enjoy even 20% of the share. So, there is a major change in HPC trend globally

as well. Today building a supercomputer is not a big deal (At least, if you are

not planning to build one to show-off at top500.org). And any enterprise, with a

requirement for more processing power and speed, can build one with very less

effort using commodity hardware, software and interconnects. And to my

conviction within a few years, each and every network (as simple as a two-node

network or as complex as a 20,000-node network, from home to corporate networks)

will be working as an HPC cluster, utilizing every bit of free processing power,

memory and storage resources across the network.

But, to make this a reality, there are some pre-requisites. The first thing

lacking today is a cross-platform clustering middleware that can work on

heterogeneous networks. While there are some hacks and tricks to run clustering

middleware to work on both Linux and Windows, there is none that is truly cross

platform. There are tricks to run a co-hosted Linux with fewer overheads on top

of Windows and then run the clustering middleware inside it.

But there are some nitty gritties. One, they are very difficult to configure;

two, they throw a good amount of overhead on the processor eating up resources;

and three, they don't support heterogeneous applications to be migrated over

the cluster. The next requirement is a good clustering middleware for Windows

platform, which can talk with existing applications as well (in Linux, we have

something called SSI or Single System Image). Microsoft is now working on its

clustering project, MS-HPC, and we have seen the beta version. Right now what

they have can run only on 64-bit versions of Windows 2003 Server. So, what is

needed for the MS platform is a clustering framework that can run on all



Windows-based desktop as well as server operating systems and that too on all
architectures including x86. At the same time we require MS applications such as

Web Servers, Exchange Server and SQL Server to be aware of and be compatible

with the clustering middleware.

I am not very sure whether MS is doing something about it because few months

back in an interview, their senior director, Windows Server Product



Management, told us that at that point they did not have plans to develop a
solution that would make their existing applications cluster friendly and take

advantage of underutilized/free resources over the network.

However, my gut feeling is that these requirements will be fulfilled pretty

soon. And we will see some heterogeneous SSI-based clustering middleware that

would actually migrate and process apps from different OS, and build a truly

heterogeneous cluster. How soon this translates into reality, only time will

tell.