Nine Servers Stretched

This time, when we were arriving at the specs for doing the server shootout,
a large public sector organization approached us for help in deciding which
servers to purchase. This was the Centre for Railways Information Systems. There
could be nothing better than taking a spec from a real live usage scenario. So
instead of arriving at our own specs, we asked CRIS to provide us with one that
they were looking at. After multiple rounds of discussions, they arrived at
specs that are broadly mentioned in the side box on this page.

We sent this spec to all the vendors and requested them to participate with
appropriate models. We received 10 servers from all major vendors in the
country. Unfortunately, two of the vendors didn't adhere to the spec, and sent
us servers with two quad core processors instead of dual core. These were from
HCL and an Intel GID brand from Bangalore called SkyRunner. These were not
included in the rankings, but reviewed separately. One server from Dell arrived
late, and on top of that gave some technical problems. Therefore, we were not
able to include it this time. Hopefully, if we're able to resolve the problem,
we would be carrying it in our next issue.

The test setup

We realized from the specs that servers have become more powerful and we
would need a decent set of client configurations to be able to load them. Our
existing test setup, which had twenty machines with P4 2.4 GHz processors and
256 MB of RAM, was not capable of stressing these servers. So, we decided to
upgrade our setup. We replaced all the P4 machines with 1.8 GHz Dual Core
machines with 64-bit support and 512 MB of RAM. And, we used a Dual Xeon
processor based server, as the controller. All these machines were connected
over a Gigabit link. Before starting to benchmark the servers, we benchmarked
these clients and established that each of these clients is capable of
generating a load of around 170 Mbps. So that our total setup was able to yield
a stress-load of 3400 Mbps. That's sufficient to stress any Workgroup-class
server.

The benchmarks

Next, we had to decide which benchmarks to run on the servers. For this, we
once again considered the CRIS requirement. They were planning to run an
application that was more I/O intensive than compute intensive. Plus, the
maximum load they were planning to put on their server was 50 clients. We
therefore considered these specific requirements for arriving at our benchmarks
and for doing the calculations later to arrive at the results. There were four
key benchmarks that we ran on the servers. Three of these stressed the I/O
performance of the servers, while the fourth one stressed the compute power.
Here are their details:

NetBench

NetBench uses a number of client machines to generate file I/O requests to a
shared location on the server. It loads a server gradually. Starting with the
load generated by one client, it continues to add more load, as per the
predefined scheme, until it reaches the maximum load. You can also specify the
delay time, think time, the number of engines per client and many other such
things. As a result, it gives you the graphs of I/O throughputs and average
response time, at various load points. If you've loaded the server sufficiently,
then there will be one peak throughput point, beyond which the throughput will
start declining. For our tests in NetBench, we used 5 engines per client, so a
total of 100 engines, with all the clients running on WinXP with SP2. We used
CentOS 5 on the server being tested. One of the interesting points that came
out, while we were running NetBench on the servers was the number of clients for maximum throughput. Some servers gave maximum
throughout with lesser number of clients, while some gave the maximum throughput
with higher number of clients. The servers which gave maximum throughput with
larger number of clients have the potential to maintain a healthy performance,
even with sufficiently large number of clients.

To get the final overall score we took the maximum throughput at 50 clients
for all the servers. We selected this because this time we had a specific
requirement in our mind, and that was the application which CRIS was going to
use. Now to check for the available head room in all the servers over 50
clients, we also had taken into consideration the number of clients at which the
server attains peak throughput.

WebBench

WebBench is meant for testing a server's suitability as a Web server. It
provides the throughput of the server in bytes and requests-per-second answered
by the Web server. It requires Web server software to be running on the
server-hardware under test. You also need to copy the workload tree of WebBench
into the Web server directory of the server being tested. The workload tree
simulates usage of a website. It consists of various HTML and GIF files that act
as a website on the server being tested. WebBench provides both static and
dynamic test suites, which execute applications that actually run on the server.
Not only this, you can easily create your own test suites. Just like other
tests, we used WinXP with SP2 for the webBench clients and on the server we ran
CentOS 5 with Apache for the Web server. For deriving the final scores we did
exactly the same thing which we did with NetBench but the only difference was
that, instead of 50 we took 55 as the cut-off number of clients.

IOmeter

IOmeter stresses a device for I/O operations and its measurement tool records
the I/O performance of the device and their impact on the system. To do the
testing with IOmeter, we ran one instance of dynamo (IOmeter workload generator)
with one worker on every machine of the cluster. To stress the server, we tested
it with 64K transfer request size (the number of bytes read or written in each
I/O request) with 100 percent read and write, both randomly and sequentially,
for 2, 5 and 10 mins. From the exhaustive results, which IOmeter gave, we only
considered the 'Total I/Os per second'. It gave us the average number of I/O
operations per second, across the length of the complete test.

BZip2

If you are a Linux user, it is likely that you know what Bzip2 is, but you
must be wondering as how come it has become a benchmark. Well, BZip2 is not a
benchmark, rather it is one of the most powerful command line zipping
applications in Linux. We used this application to create a processor stressing
test, but kept the name intact.

As multi-cores are becoming common, the choice of benchmark is simultaneously
becoming trickier. You need something which is more of a batch processing job,
rather than a single number crunching process. First we decided to go with the
standard benchmarks, like Linpack or in fact HPL (High Performance Linpack). HPL
is undoubtedly one of the best benchmarks to stress the processors and give an
output in the form of GFlops. But there was a small problem that we faced in
this. Linpack being an Open Source benchmark, needs you to customize it for the
architecture that you run it on. So, if you compile a Linpack on Intel and run
it on AMD, it might give you erroneous results. We came to this conclusion when
we ran an Intel customized Linpack first on an Intel Dual-core Xeon machine,
which gave us around 35 GFlops, and when we ran the same benchmark on a
Dual-core Opteron, we only got 4 GFlops. This clearly meant that Linpack was
optimized for the Intel platform.

We then requested AMD to create a Linpack customized for Opterons. But, the
time was so less that they were not able to provide us with an Opteron-customized
version of Linpack. So, we decided to drop this test and go with something which
is more generic and will not change the result based on the architecture of the
processor. It was then, we created BZip2 benchmark.

It is essentially a set of batch files, which takes a standard 100 MB file
and creates 60 instances of it, then in one-go it starts compressing all of them
using BZip2. At the end, it gives us the time duration taken by the server to
finish the task. We really saw a huge difference of performance in quad-cores
and dual-cores, while running this test. For instance, a standard dual-core took
around 550 seconds to perform the job whereas a quad-core took just 120 seconds.
That's impressive!

Features considered

In features, we considered three broad areas, and then further divided them
into specific features. These include the following:

1.Manageability

a. Management port

b. Management software

c. LEDs and Alerts

d. Tool-Free Access

2.Expandability

a. Maximum Upgradable RAM

b. Processor Upgrade

c. Number of hard drive bays

d. Free PCI-X slots

3.Fault-tolerance: Hot-swappable fans, SMPS and hard drive

Out of these, most of the servers already had hot-swappable SMPS and hard
drive. Some had hot-swappable fans as well. Under expandability, most of the
servers supported at least 16 GB RAM. Processors could be upgraded in all of
them, and all had at least four drive bays. All also has some free PCI-X slots.
In manageability however, most of the specs were important. Therefore, we gave
higher weightage to manageability over expandability and fault-tolerance.

Price and warranty

Here, most of the servers, except one came with 3 years warranty. We divided
the weightage into a 60:40 ratio for price and warranty.

Finally, we used the Brown-Gibson model to arrive at all the weightages. The
total score was calculated out of 100. Here, performance and feature scores were
out of 40 each, and price/warranty was out of 20. All the results were evaluated
keeping the specfic requirements of CRIS in mind. We hope you find our
evaluations useful, and do send us your feedback.

Consolidated Netbench scores: From here you can figure out the file-serving performance of all the servers at a glance	Webbench score: From here you can figure out the Web Server performance of all the servers at a glance.

Anindya Ray, Saurangshu Kanunjna, Swapnil Arora with help from Jasmine,
jayanta Ghosh, Vijay Chauhan

IBM x3650

The x3650 emerged as the overall best in our shootout. It gained majorly in
price and features, and at the same time, also showed decent performance. The
server had two Intel Woodcrest 1.86 GHz Dual-core Xeon processors in a 2U
rack-mountable form factor. The chassis design of IBM is a piece of art. We
found it to have the best tool-free-access, which lets you pull out any
component of the server, with bare hands. The server came with two 10K rpm SAS
drives, which was probably why it lagged in disk I/O tests against servers that
had 15.5K rpm drives. The server came with 4 GB RAM, but one thing we noticed
about it was that the RAM was distributed across 8 modules of 512 MB each. This
makes sure that in case a RAM chip or a RAM slot fails, then you only lose 512
MB of RAM and the rest remains working. So, this gives you a kind of redundancy
in memory. The server supports hot swappable HDDs, power supply and fans. Here,
hot swappable fans was something that we found only in this and the Sun server.
The interesting thing was that the server had a large fleet of fans inside, and
yet it was one of the quietest machines of the lot.

Performance

Though the server's performance wasn't the best of the lot, its scores were
above average. In the file I/O test (NetBench), the server gave a throughput of
932 Mbps with 50 clients. However, these are not peak values. The server peaked
at 70 clients with a throughput of 1038 Mbps. We even suspect that this
performance would improve further if its 10K SAS drives are replaced with 15K.

Next, we checked the server for raw I/O performance using IOMeter. Here, the
server gave the second best throughput of the lot. In our Web server tests, the
server delivered a maximum of 9763 requests per second and peaked at 65 clients.

The server came with a Dual-core Xeon, having one of the lowest clock speeds.
Possibly that's the reason why it took around 562 seconds to complete our BZip2
test, which is again an average score.

Thankfully, the server's motherboard supports an upgrade to quad-core
processors, which can be done if you need more compute power.

Bottomline: With its price and features the server is an ideal buy for
workgroup environments. While its performance is sufficient for the requirement
we had set forth for these tests, it can be improved further, as it has
sufficient upgradability options.

Sun Fire X4200 M2

This server was the second best overall and it gave good performance too. It
managed to gain the runner up position in the shootout primarily due to its
performance and features. The server is innovatively designed, which doesn't
require you to open the main chassis to replace any of its hot-swappable
components. While this is true for most servers in case of hard drives and SMPS,
this server is a little different in that it has a separate enclosure for
replacing the hot-swappable fans as well. In fact, if you open the main chassis,
the server shuts down. This was the only server that came with two management
ports. One of them is a 10/100 Ethernet port. You just have to connect it to
network, install the accompanying management software from Sun, and manage this
server through this port. The other is a regular serial management TIA/EIA-232-F
asynchronous RJ45 port. The server is rack-mountable, and comes in 2U form
factor. It came to us with two dual-core AMD Opteron 2.8 GHz processors with 8
GB RAM. The RAM is expandable to 32 GB (16 GB per CPU). The server has ample 16
DIMM slots. As we had specified 4 GB RAM in our specs, we reduced the server's
RAM by 4 GB for our tests. We also reduced the price of 4 GB RAM from the
server's price. Despite that, the server was the costliest of the lot.
Interestingly, the server was also fairly noisy. The server's power button is
also tactically placed so that nobody accidentally switches it off.

Performance

In our File I/O benchmark,it gave the second best performance with a maximum
throughput of 1128 Mbps. The server peaked at a whopping 95 clients. With 50
clients, the server gave 814 Mbps throughput. In WebBench, it could handle up to
10,951 requests per second, but strangely, it peaked at only 35 clients for this
test. With 50 clients, it dropped to 10,418 requests per second, which was still
the best of the lot. It didn't perform very well in raw I/O tests, which we
suspect is because of its 10K rpm disks. Interestingly, this was the only server
that came with compact hot-swappable 2.5-inch SAS disks. Lastly, the server took
568 seconds to finish the Bzip2 benchmark, which is a bit on the lower side.

Bottomline: Given the rich set of features and excellent performance,
the server is a great buy. The price however, is on the higher side.

Wipro Netpower Z2108

This was the only true rack server from Wipro with 1U form factor. In fact
this was also the slimmest model in the shootout. The other two Wipro servers
were tower models, which could also be mounted in a rack, but would onsume 5U.

The server performance was one of the best among the Dual Cores, which was
nearly similar to the IBM server. Where it lost points was features. It could
only accomodate up to 4 drives, and didn't have any management ports. Plus, it
didn't suport hot swappable fans.

The server's price is also average, being the fourth cheapest amongst all
dual core servers we reviewed. The server ships with two Intel Dual Core
processors with 1.86 GHz frequency and 4 MB L2 cache. It came with 15K rpm SAS
hard drives, and two 1 Gbps full duplex Ethernet cards.

Performance

In our file I/O (NetBench) benchmark, the server's performance was average.
It managed to support only 45 clients, which was the lowest of the lot. With
this number of clients, it managed to deliver a peak throughput of 898 Mbps.
This makes it a suitable file server only for small workgroups.

Next, we checked the server for raw I/O performance using the IOMeter
benchmark. To our surprise, here the server gave the highest throughput. We were
initially a little surprised considering that this was stressing the same thing
as the File I/O benchmark. But when we got the results of our next benchmark,
viz WebBench, things became a little clearer.

In WebBench, the server managed to support up to 10072 requests per second,
and peaked at a whopping 85 clients. This was again one of the best scores. The
reason why this server performed well in both IOMeter and WebBenc was that both
were sending small packet sized requests, which the server was able to process
very fast. Whereas, the NetBench benchmark threw a mix of packet sizes, which
resulted in lower throughput.

Lastly, we ran the processor stress test, where again it did remarkably well,
giving the second best score. It took around 555 seconds to finish the BZip2
benchmark.

Bottomline: This is a good server for applications requiring small packet
sized I/O requests, such as a web server. As a file server, it's good for a
workgroup with 40 to 50 users.

Wipro NetPower O2501

Amongst the three Wipro servers that we received for the shootout, this was
the only one with AMD's Opteron processors. It has two Opteron 2212 processors
and is based on NVIDIA's chipset. It's a tower cum rack mountable server, which
comes with a form factor of 5U when put inside a rack.

The expandability options in this server are suffficient for most
requirements, but the other servers, including the two other Wipro servers,
fared better in it. This one supports up to 16 GB RAM, has 8 DIMM slots, and
supports upto 4 hard drives. These are slightly lower than many of the other
participants in the shootout. In fault tolerance, the server supports hot
swappable drives and power supply, but not the fans. Coming to manageability,
the server has a tool free access. However, it lacked management ports, which
both IBM and Sun boasted of.

Performance

This one performed better than Wipro Intel tower server, NetPower Z2503, but
it was not good enough to give the other servers a run for their money. In
NetBench, it gave a maximum throughput of 822 Mbps, which was the second lowest
in the shootout.

It gave this throughput with 55 clients. At 50 clients, it gave a maximum
throughput of 805 Mbps, which is much better than that of Tyan and comparable to
that of Sun, which gave 814 Mbps with the same number of clients. In Web Bench,
its performance was the lowest of the lot at 9052 requests per second, but it
did manage to support up to 85 clients. At 50 clients, it gave a delivered 7,935
requests per second which is again the lowest amongst all the servers. It took a
long time-632 secs-to finish the BZip2 test, which was the longest. In IOMeter
however, its performance was better than that of Sun. According to its
performance scores, it would do better as a web server than a file server.

It's the cheapest amongst the three Wipro servers, and third cheapest
overall. The server comes with a standard three years warranty.

The overall score of the server is equivalent to Wipro Intel Rack server (NetPower
Z2108), mainly due to its cheap price and almost similar features.

Bottomline: Overall, this is a plain vanilla server with average
performance and basic features.

Wipro NetPower Z2503

This was another tower cum rack mountable server that we received, which
consumes 5U rack space when mounted. It was shipped with two 1.86 GHz Intel Dual
Core Xeon 5120 processors. Like the O2501, this one also has tool free access.
Compared to the other Wipro servers, it mainly scored in price and performance,
Plus, it shipped with two 15K rpm, SAS HDDs, and hence its performance in raw
I/O test was among the top.

The server has 8 DIMM slots, which can take up to 32 GB RAM, which is higher as
compared to its Opteron based counterpart. In terms of redundancy both HDD and
power supply are hot swappable whereas Fans are not. One advantage it had over
other servers was that along with HCL it was the only server with four free PCI-X
slots, allowing you to add additional RAID cards or HBAs.

Performance

In NetBench it gave a maximum throughput of 709 Mbps at 70 clients which is
eventually the no. of client where it peaked.This was the lowest of the lot.
With 50 clients, it was able to achieve a throughput of 632 Mbps. In WebBench,
it delivered a maximum of 9087 requests per second, which was the third lowest.

In raw I/O test using IOMeter, its performance was among the top, third best
to be precise. This was largely because of higher rpm hard drives.

Coming to the processor performance test, among the dual cores it scored very
well as it completed the BZip2 test in 556 seconds.

As far as price is concerned it's che-aper than the Wipro Z2108 server and
comes with 3 years of standard warranty. Among the three Wipros this is the best
in terms of price, performance and features.

Bottomline: While the ser-vers's performance was average, its overall
score compirising of performance, price, and features was better than the Tyan
and Acer servers. In fact, this is in the top 4 when you consider its overall
score. It' s a good option for use as a Web server, but not as a file server.

Acer Altos G540

This server is one of the best performers of the lot. It gave better overall
performance than IBM and Wipro Rack. However rounding the figures gave us a tie
between all the three. This is a tower server which can also be fitted into
racks, in which case it consumes 5U of rack space. This server ships with two
2.0 GHz dual core Intel Xeon 5130 processors and 4 GB DDR2 RAM. It has a total
of 8 DIMM slots and its RAM can be upgraded to 32 GB. The server came with two
Seagate 73 GB SAS Cheetah drives with 15K RPM and its drive capacity is
expandable to 4 drives. The server shipped to us doesn't have a management port
which is one of the factors that takes Acer behind SUN. However an optional Acer
Remote Management card can be added for remote management, for an extra cost.
The server comes with redundant HDDs, power supplies and fans. However, the fans
are not hot swappable while the drives and power supplies are. The server has
tool free chassis but doesn't provide complete tool free access as you will need
a screwdriver to remove the power supply. It has 3 PCI Express and two 64-bit
PCI-X slots.

Performance

The server gave the best  performance in our processor benchmarks as it
took only 515 seconds to complete our Bzip2 test, which is the least time taken
by any dual core server. Most likely this is because it has a better Xeon
processor with 2.0 GHz frequency and 2 Mb of L2 cache. It was second best in our
raw I/O tests and in Web server tests. When we stressed the server as a Web
server at 50 clients, it gave the throughput of 9549 requests per second and
maximum throughput of 9920 requests per second and peaked at 45 clients. In
NetBench, with 50 clients it gave a throughput of 982 Mbps which is best among
all servers. The maximum throughput it gave was 1150 Mbps which again is best
amongst the lot, and it peaked at 95 clients.

Bottomline: Considering its performance and price of 1.5 lakhs with
warranty of 3 years, the server is a good buy.

Tyan Power-X AMD-OPT2216

Like Sun this one too is based on NVIDIA's chipset and has two 2.4 GHz AMD
Opteron processors. However, it didn't really create a big impression with
respect to others because it lags in performance and features. Even though it's
amongst the most expandable servers, it has some crucial elements missing. It
doesn't have hot-swappable fans, doesn't provide you with full fledged tool-free
access or management port. It offers you the maximum expandability with eight
HDD slots, 16 free DIMM slots, 3 free PCI-X slots and a max of 64GB RAM
upgradability. Since we were testing the server based on a fixed criteria, such
level of expandability was not a major requirement. So, it didn't score heavily
in features.

Performance

It performed decently in NetBench with a maximum throughput of 1,039 Mbps
with 85 clients. In fact it comprehensively beat the three Wipro servers. But
when repeating the same test with 50 clients, it couldn't repeat the magic and
its performance dipped to a maximum of 667 Mbps which is second lowest amongst
all servers. In WebBench, it handled a maximum of 8798 requests per second which
was amongst the lowest. This proves that it is a better file server than a Web
server. With 50 clients, it delivered a maximum of 8,798 requests per second
which is again amongst the lower scores that we obtained. Even in I/O meter
test, it couldn't revive itself, as this is the only dual core server with a
SATA HDD. Even though the vendor sent us a SAS disk, it didn't suffice as we
required two SAS HDDs for the tests. Unfortunately, a second SAS could not be
arranged in the stipulated time, instead we had to make do with the two SATAs.
It took 587 secs to complete the BZip2 test, which is on the higher side as
compared to other servers. So, even though it gave a decent performance in
NetBench, it didn't give good results in other tests.

Bottomline: This is the cheapest amongst all AMD CPU based servers.
Though it didn't finish amongst the top performers, you could still go for it if
expandability is high on your agenda.

Connoiseur SkyRunner SRXQCDP-2500

This was one of the two quad core servers we received, and as it didn't match
our required spec of dual core dual CPU servers, we could not include it in the
rankings. If you are looking for a rack server that can give you great
processing power, then this is the one for you. Its processor proved to be the
fastest in our shootout. The server came with two 2.6 GHz Quad-core Xeon
processors.

While running Intelcustomized HPL (High Performance Linpack) we got a
whopping 53 GFlops. This is exactly half the power of the renowned Indian
supercomputer Param 10000.

The server also comes with 15K rpm SAS drives, which makes it a good
performer in IO/Disk access tests. For fault tolerance, it has hot-swappable
HDDs and SMPS, but fans in the box are not hot-swappable. This server can
accommodate six hard disks and supports all types of RAID sets. But it is too
noisy and creates a sound of around 100 db with rear fans. This server comes
with only one PCI-X slot leaving less scope for upgradations.

Performance

As mentioned, this server performed better in benchmark tests for processing
power than all servers we had tested in past. The server also performed well in
other I/O and disk access tests. It took just 121 seconds to finish our BZip2
benchmark, where we zip 60 files of 100 MB size at one go. This means 60 threads
are generated at the same time and thrown on the processor. While we were
running this test, we noticed that all cores of the processor were saturated.
This means that there was no head space left for the processor to perform even
better and the score that we got was the best possible.

In NetBench, the server didn't give great performance and peaked at 50
clients with a throughput of 944 Mbps. In raw I/O using IOMeter, it came third.
However, in Web server performance, the server handled a whopping 10, 362
requests per second and peaked at 65 clients.

Bottomline: If you are looking for a server for huge multithreaded
data processing, such as batch processing tasks, then this is the best option.
Plus, the price tag of less than Rs 2.2 lakh is good value for money.

HCL GL 2700 OW

Like the SkyRunner, this one also could not be included in the rankings as it
had quad core processors. But surprisingly, this was the cheapest server of the
lot, despite having two quad core processors. It shipped with 1.66 GHz Quad-core
Xeon processors and came with 4 GB of RAM. With 8 DIMM slots, you can expand up
to 16 GB of RAM.

This server provides plenty of options for expandability, as it has four PCI-X
slots and can support up to 6 hard drives. The server was shipped with two 250
GB SATA hard drives. The server doesn't provide tool-free-access, nor does it
have any management ports. For fault tolerance, it provides hot-swappable HDDs
and SMPS, but not fans.

Performance

In our processor performance test, HCL performed well and took only 268
seconds to complete our BZip2 test. Though this is not as good as the
SkyRunner's results, it is better than the dual-core servers.

When we stressed the server as a Web server using WebBench, it performed very
well. At 50 clients, it gave a throughput of 10647 requests per second and
maximum throughput of 10937 requests per second (which is higher than the
SkyRunner. It peaked at 65 clients. In NetBench however, the HCL server didn't
do very well. At 50 clients, it gave a throughput of 811 Mbps and maximum
throughput of 886 Mbps. Both of these scores are lower than SkyRunner.

In our IO test, HCL gave average performance, but managed to perform better
than SkyRunner in the sequential-write test.

Bottomline: If you are looking for a good Web server, then this will
work perfect for you. In fact, at 97K, this server offers great value for money.
Warranty however needs some improvement. This was the only server to ship with
one year warranty. All other servers gave a standard 3 years warranty. In HCL,
you have to pay extra for warranty beyond one year.