What we're about to tell you is something nobody has ever told you before.
Till now whenever any magazine (including us) has carried any shootouts, what's
covered are performance results of products, how the products compared against
each other, their pros, cons, etc. There's also a broad mention of the test
procedures that were followed by the various features that were compared. More
stress is given on the actual product reviews and the test results and less on
the actual tests and how they were run. But what we're going to tell you this
time is completely different. We'll let you in on the actual test procedures
and all the excitement and heartaches we've experienced; and the learning that
followed while doing this shootout. Instead of focusing on product evaluations
alone and declaring a winner, we'll focus on uncovering how we actually ran
the benchmarks on those products. And then, we'll tell you how to interpret
their results, and finally we'll reveal the results we got on the NAS boxes.
Why are we doing this you might ask?
For one, you'll be able to better understand the results we received for
the NAS boxes. Two, if you ever need to, you can run those tests on your own
storage boxes to find out their performance. And three, if you're planning to
purchase a new NAS box, then you can even ask the vendors to conduct similar
tests on their boxes and show you the results. In other words, we're
empowering you with information to help you make a much better purchase decision
now and in the future.
We've spent more than three months in testing the NAS boxes we received for
evaluation. During this phase, we learnt a lot of new things, not only about
performance, but also about features, vendor support issues, pricing models, and
much more. So presented in this story therefore, is a complete buying guide to
NAS boxes, with our personal learning. With this, you'll not only be able to
decide whether or not to buy one of the four NAS boxes we tested, but any NAS
box that's out there.
Choosing benchmarks
This is easier said than done. On the face of it, we ran just two benchmarks-Netbench
and IOmeter. But when we actually got down to doing it, we ended up creating
twelve different test mixes from this duo for testing various aspects of the NAS
boxes. NetBench at one time was given away free of cost by Ziff Davis Labs of
US. You just had to bear the shipping and handling charges for the CD. Though
they've discontinued doing this, you'll still find it available for download
at various websites.
The benchmark provides a lot of flexibility and control over the test setup
and let's you create a real live scenario for testing the throughput and
response time of NAS boxes and servers.
The other benchmark, IOmeter is also free and can be downloaded from http://iometer.org.
There are dozens of different metrics to adjust while running this benchmark and
it can reveal the true potential of any storage device. It measures the number
of I/Os per second of the storage device and also gives the throughput in Mbps.
You can also easily run these benchmarks in your setup, besides doing the
regular file transfer tests.
The test environment
Just like any other network device testing, this one also requires an isolated
network. Many organizations already have a separate test network that they use
for testing anything new before incorporating it into the production network. If
you don't, then maybe it's time you also created at least a basic setup.
What you will need though is a Gigabit Ethernet switch and a few PCs that will
act as the client load. The more PCs you have the better. We used 19 PCs in our
test setup, and a 20th machine that acted as the controller for the remaining
machines. On top of this of course, you could use a KVM over IP switch to manage
all the nodes remotely.
Our observations |
1. NetBench throws a lot of I/O requests to the NAS box through multiple clients. It records how quickly is the NAS box able to respond to these requests. At the end, it just maps them on two graphs. One shows the maximum throughput for each client set, and the other shows the associated response time. So you can easily tell not only the peak throughput delivered by a NAS box, but with how many clients did it deliver the same. 2. We tested the NAS boxes with network load balancing 3. The I/O performance of a NAS box changes based on the 4. Never run performance tests on a production system. 5. If your NAS's performance suddenly deteriorates, 6. Rebuilding a RAID with a new disk takes 2 to 3 hours 7. Perform network backups from the NAS after office |
Understanding NetBench
NetBench uses a number of PCs to generate file I/O requests to a shared location
on the NAS. It loads the NAS slowly, starting with one PC and gradually
continues to add more as per your test definition until it reaches the maximum
defined load. You can also specify how long it should run and many other things
(See screenshot). It gives you graphs of the I/O throughput and average response
time at various load points. If you've loaded the NAS sufficiently, then there
will be one peak throughput point, beyond which the throughput will start
declining.
Setting up NetBench
The first thing you need to do is create a shared directory on your NAS box with
full read/write access to everyone. Connect it to the Gigabit Ethernet switch.
Then install the NetBench Controller program on a PC running Windows 2000/2003
or XP. You will define all benchmarks and control the entire test process from
here. Next, you need the PCs that will send requests to the NAS box. These
should ideally be of the same spec and preferably P4 machines running Windows
XP. Install the NetBench client software on them. You can add up to 60 PCs to
this load. Each PC can emulate several PCs, so you can simulate a much bigger
load. Note that all machines have to be assigned an IP address manually. No DHCP
please! Before starting the test, you need to configure the computer running the
controller program. First, change the computer name to “controller”. Then
open and modify the client ID file 'client.cdb' from CONTROLLER_DIR>\CLIENTIDS.
Here you'll find an entry for each client in your test bed with an associate
ID number. Add each client's IP address and a unique ID number and save the
file. On all clients, you need to map the shared folder on your NAS to the drive
letter f:.
Running the tests
First run NetBench client on all the PCs. Then go to the controller machine and
launch the controller program. Here, from the menu, select Clients>Start Log
in option to connect all client machines to the controller. Then add the tests
you want to run from Suit>Add. Here, open the DM.tst. Next, from the tool bar
select the “Go” button to start the test. The default test takes about an
hour to execute, but you can modify it as per your requirement. Let's see how
to do that.
Modify NetBench tests
From the controller window choose Suites>Create/Edit Test Suites. Open DM.tst.
You will see many test mixes (see screenshot on p.103). Each client represents
one PC and each engine will run a test mix. So two engines on a client means it's
going to run the same test mix twice, i.e. simulating the workload of two PCs.
Note that there's no ideal mix. You have to define the right one as per the
number of clients in your test setup and the device under test. We, used 7 test
mixes, which gradually increased the load on the NAS from one client to 18
clients running 4 engines each, meaning 72 clients. Each mix took about 3
minutes to complete.
Analyzing Results
NetBench automatically creates a full test report with graphs showing throughput
and response time for each test mix. From this, you can easily tell the maximum
throughput and the number of clients and engines used to achieve it. You can
compare this with reports for other NAS boxes to determine which one delivers
higher throughput and with how many clients. All test results are converted into
an Excel file by NetBench for easier analysis.
NetBench & file compression
Many NAS boxes support compressed folders, wherein any data stored on the NAS is
compressed on the fly to give you more storage space. The compression does of
course take a toll on the NAS's CPU cycles. What has to be checked is whether
this affects the overall performance.
Using IOmeter
IOmeter is a widely used I/O performance analysis benchmark, specifically meant
to test storage devices. IOmeter consists of a workload generator called Dynamo
to do I/O operations and IOmeter itself to measure the I/O operations .The good
thing about this benchmark is that there's a whole range of test mixes you can
create. IOmeter is available on a variety of Operating Systems, including
Windows, Linux, and even NetWare. You can even test a NAS box by running IOmeter
from a single client. Installing IOmeter is fairly simple. You can install it on
multiple client machines as well if you want, and in fact you should do that if
you really want to stress the storage device. The only problem is that there's
no central controller from where you can control all the IOmeter installations.
You have to manually go to each machine to run it. But, like in NetBench, you
need to create a shared folder on the NAS box and map it to all clients running
IOmeter.
When you start IOmeter on both clients, Dynamo is automatically started.
Here, look for the topology window and click on all the managers to expand the
menu. Here you will see a default manager having with the same name as your
client machine. By default it has two workers. This is similar to NetBench's
client instances. In our tests we used four workers on every client. You can
have up to 100 workers under one manager. Next look for the Disk Target tab.
This shows you all available drives, including the mapped drive on the NAS.
Select it for all workers. Here, you can also specify the maximum disk size
to be used by IOmeter. By default this value is set to zero meaning
the entire disk will be used by IOmeter.
Simple file copy benchmark
Even if you can't these benchmarks, the easiest thing to do is a simple file
transfer. We took about 100 GB of documents, spreadsheets, presentations, PDFs,
and even MP3s and graphics files. We copied them to a shared folder on the NAS
and measured how long it took. We even created an ISO image of this data and
copied it to the NAS. This gave the NAS a continuous stream of data to store,
and told us the maximum throughput that the NAS box is capable of delivering to
a single machine.
Besides performance, you also need to look into the features offered by the
NAS before buying. If you think a NAS is nothing but a set of disks connected
together to offer storage space on the network, then think again. You're in
for a surprise. We found lots of small but crucial points that you need to look
into when scouting for a NAS.
Expandability
Two key elements that can be expanded are hard drives and RAM. Not all vendors
allow you to upgrade RAM in their boxes. Their claim is that their NAS boxes are
optimized and tuned to work with the RAM they ship with. If you need a RAM
upgrade, then the only option is to upgrade to a new box with more RAM. So
besides spending time configuring the new NAS box to work on your network, you'll
also be busy migrating all data from the old box to new one. Do check with your
vendor on RAM upgradability before buying.
Hard drive upgrades are another story altogether. Many vendors don't allow
you to just pick up a hard drive from the market and plug it into a free bay in
your NAS. The reason for this is that every hard drive has its own firmware
version, and the NAS vendor certifies hard drive models to work with its NAS. If
you plug in a hard drive with a different firmware version into your NAS box,
even if it's the same HDD brand, then the NAS box's warranty gets void. So
what happens if one of the hard drives in your NAS box crashes? You can't plug
in any hard drive you want in it. So either you buy a spare hard drive at the
time of purchasing the NAS, or shell out extra money to pay the NAS vendor for a
replacement. Needless to say that the amount you have to pay is directly
proportional to how quickly you need the replacement!
This also opens up another interesting aspect. If you want to add a new hard
drive to increase the capacity of your NAS box, then many vendors will charge
you extra for it. The licensing models of some vendors are based on the capacity
they provide you.
Available storage
Here's another gem to worry about. Did you know that a 1 TB NAS box will never
give you 1 TB of storage space, because a part of it is used up by the RAID and
the OS? Even if you did, then you may not know that there can be major
variations in how much storage space is used up by them. So a 1 TB NAS box could
have anywhere from 600 to 700 GB of available storage space. That's a whopping
100 GB difference in available storage space! To give you an idea, consider a
NAS box that has 250 GB x 4 disks, which amounts to 1 TB storage capacity. When
you configure it in RAID 5, then one disk is gone, reducing the total available
storage capacity to 750 GB. Further, the vendor may use more space to host the
NAS OS, take snapshots for OS recovery, etc. This could easily take anywhere
from 50 to 100 GB, leaving you with 650 to 700 GB of data storage space. Moral
of the story: If you need 1 TB of storage capacity, then go for a 1.5 TB NAS.
Data compression
Check whether a NAS box supports compression, as it will help you utilize the
storage capacity more effectively. Data in every organization increases by leaps
and bounds. So you might feel that you've bought a whopping 1 TB of storage
capacity for your network now, but one year later, you'll be wondering where
all that storage space went. At that point, you'll wish your NAS box supported
compression. Some NAS boxes we received for evaluation were running propriety
NAS OS and didn't support data compression. However NAS boxes running Windows
Storage server had this feature. Performance wise also it gives you negligible
difference over file storing without compression.
Monitoring and mgmt
Remote management and monitoring is what everybody wants in a NAS. A few
features a NAS device should therefore have are-
Web-based interface, remote desktop connectivity, remote login through telnet or
SSH, e-mail and SNMP notifications, disk quota management, file screening and
storage reports. You could also look out for a LCD screen on the NAS itself.
This can be pretty useful in giving you lots of critical information about the
NAS device and letting you control it.
Some more questions to ask |
Q> Can I upgrade the same box into a SAN? Some vendors provide iSCSI support on their NAS boxes. They add a module of disk arrays and make the old box as a controller. The Controller further connects the disk array module via Ethernet and communication happens over IP. Q> Can I upgrade my NAS capacity? |
File Systems Supported
A typical network will have different types of clients: Microsoft, Novell, Apple
and UNIX/Linux. A NAS device should appear on the network as a native file
server to each of its clients with files saved and retrieved in their native
file formats.
Data backup
Most NAS devices support backing up data to a local tape drive or network backup
server. Apart from this, also look for the following:
Snapshots: Incremental point-in-time copies of stored data are created
on local devices' disks only. They allow easy data restoration without having to
access the tape.
Disk Backup: Some NAS boxes come with a disk backup option, where
backup is done on NAS storage only. For this you have to configure the RAID in
such a way that at least one disk should remain free. For example if you have
four drives, then create RAID with three drives and leave one for disk
basedbackup.
Data replication: To keep an updated copy of critical data in a
separate location, online and ready for use at any time, thereby enabling faster
DR.
Device to device replication: Two NAS box are connected in parallel and synced
for high storage availability. Check whether your NAS also supports automatic
failover to the stand by NAS.
Backup to DVD: Another type of backup that some vendors provide. It's
useful when you need to send out some very critical data.
Security
A NAS device should easily be able to integrate with your existing directory
service for authentication. Anti virus: How does your NAS vendor support
anti-virus software? Some NAS devices allow you to install anti-virus software
directly on them, while others let you do remote scanning from an anti-virus
server on the network.
Access Control Lists: Allow file and share-level access for authenticated
users and groups to files and folders stored on the NAS. Check what kind of ACL
is supported by the NAS.
Logging: A good NAS box should audit and log most tasks done like log on, log
off, security modification, password changes, user creation, etc
E-mail and SNMP alerts: These are a given, and a concern if some vendor doesn't
provide them.
Storage Control
It's criticla to prevents a user from filling up the entire storage space.
Check for file screening support to control which types of files are allowed to
be saved on the NAS. How comprehensive are a NAS's storage reports?
High availability
Today even an entry level NAS comes with dual NICs. Check whether it's able to
do network load balancing, automatic failover, and teaming. The last option
aggregates the two physical links, thereby doubling link speed.
Other features to check
A NAS box that can support databases is different from one that supports basic
file storage, and their cost differential is also
significant. You can also look for a redundant power supply in the NAS.
Price
Last but not the least is the price you pay. There's a huge variation in the
prices between various NAS boxes.
It all depends upon how many features is the vendor bundling into the NAS.
For instance, the HP NAS we received cost Rs. 3.15 Lakhs. It gave only 629.9 GB
storage space, but also had tons of useful features. The Level One NAS on the
other hand, was priced at only Rs. 98,500, gave storage space of 1.3 TB, but its
features were limited.
This NAS was the best performer of the lot. The device runs Linux based
Guardian OS, and has a simple and easy to understand Web interface for
management. It also has a LCD display. Interestingly, its OS takes 10 GB space
on every hard disk, which ensures that the server will keep running even if two
or even three of the hard drives fail. It ships with 4 x 250 GB 7200 RPM
hot-swappable SATA drives and 512 MB RAM, which can even be upgraded to a good 4
GB. The device also ships with a Manager application, which is useful only if
you're running multiple Snap Servers in your data center.
The SnapServer can also work as an FTP or a Web server, and supports CIFS,
SMB, NFS and AppleTalk file systems. It also has plenty of monitoring
capabilities from regular system monitoring, users monitoring, open files
monitoring to iSCSI device monitoring. The box shipped to us with CA Antivirus
pre-installed, and even BakBone NetVault software for taking backups to tape
drives. The box also supports taking snapshots and there's even an option to
create a single image of its configuration and all its data. This can be great
for DR purposes, as you can simply copy the image to another storage device or
tape drive. After configuring RAID 5, the available storage capacity on the
server comes down to 691 GB.
NetBench |
^ IOMeter: I/Os per second |
On the performance front, like we said, it gave excellent results. It scored
a whopping peak throughput of 608 Mbps in NetBench, and that too with 15 clients
(whereas others peaked out at 5 clients). Even beyond 15 clients, its throughput
remained higher than the other NAS boxes. In IOmeter, the I/Os per second it
recorded for both 64 K and 128 K request sizes were the highest in sequential
reading and writing of data. In transferring 100 GB of data, it only took 73,
which was the fastest among all NAS boxes reviewed. Likewise, transferring a
compressed ISO image of the same 100 GB of data was also the fastest at 44
minutes.
Price: Rs 2,85,000 (3 yrs warranty on hardware, 90-day software support) Contact: Adaptec India, Bangalore, Tel: 41339000 E-mail: Deepak_naik@adaptec.com SMS Buy 130220 to 6677 |
Cheapest of the lot, the LevelOne GNS 8000B was the only NAS box in the
shootout to ship with a redundant power supply. It was also the NAS with the
slowest processor, an Intel Celeron M 1.3 GHz. Plus, it had 512 MB RAM and a
total storage capacity of 2 TB. However, its usable storage capacity comes down
to around 1.3 TB, with the rest being used up for data protection. The NAS has
an LCD panel, which apart from displaying information, also lets you configure
its IP address. Then, of course, it has Web based management, which is pretty
easy to use. It has hot swappable hard drives, and unlike many other vendors,
who insist that you need to plug in a hard drive from them only, in this one,
you can plug in one on your own. So if (God forbid), one of the hard drives
crashes, you're not at the vendor's mercy to get yourself a replacement. The
NAS supports file systems like NFS, FTP, AppleTalk and provides DHCP and event
logs. Authentication is very simple and can be done in just 3 steps. This NAS
supports ADS and LDAP based authentication, as also RAID levels 5, 0, 1, and 10,
and comes pre-configured with level 5. You can't install an anti virus on it,
so it has to rely on an external anti-virus server for protection. On the down
side, it does not support file screening, so you can't control unwanted file
extensions from being stored on it. It can back up data to tape, DVD and even
supports snapshots.
NetBench |
^ IOMeter: I/Os per second |
The NAS gave consistent performance in all the tests. Being an entry-level
NAS, it's not meant to handle a very large number of clients. So the maximum
throughput it gave in NetBench was 239 Mbps, and that too with only five
clients. Beyond five clients, its performance dipped a little at 10 clients, but
interestingly it picked up again at 15 clients, resulting in 238 Mbps
throughput. It wasn't able to handle the load beyond 15 clients and its
throughput dropped continuously. In IOMeter, its performance is better with
smaller performance requests. So whether it was a random or sequential read or
write request we sent to the NAS, the maximum IOs per second for 64k requests
were always higher than 128k. Its performance in both sequential reading and
writing is much better than in random. Among other NAS boxes also, this one gave
better performance in random reads and writes for 128k requests.
Being an entry-level NAS, don't expect to do lots of file transfers to it.
It took the longest to copy a 100 GB mix of various files, at 101 minutes. The
same time reduced to half when we compressed these 100 GB files into a single
ISO image. Overall, this is a good choice for very small setups.
Price: Rs 98,500 (5 yrs warranty) Contact: Minds (India), Delhi, Tel: 09818299663 E-mail: jerry.albert@level1.co.in SMS Buy 130221 to 6677 |
This is the latest NAS from HP, and is also the most feature rich of the lot.
The first thing we noticed about it was its ease of use. You don't need a
system administrator to configure it. And if its OS crashes, you can restore it
by popping in the accompanying DVD and booting from it. The software does the
rest. It automatically configures RAID 5 for you. The box has Windows Storage
Server 2003, and also has data protection cum backup software for backing up
data to tape, a network share, DVD or another NAS box directory. Other features
in data protection include protection of shared folders and taking snapshots.
This box supports iSCSI, allowing the NAS box to be integrated into a SAN
environment. Plus, it also has tools for Exchange and SQL Server that let you
host Exchange or SQL databases on the NAS box. While the total storage capacity
of the NAS is 1 TB, the actual available capacity is 692.9 GB. Remaining space
is used up by OS and data protection. On the upgradability front, you can't
add any more RAM to it. The only way to get more RAM is to upgrade the entire
box. The NAS box has a Web interface, which opens a remote desktop of the NAS
box on the Web interface and lets you create and manage shares.
NetBench |
^ IOMeter: I/Os per second |
On the performance front, the maximum throughput we got using NetBench was
203 Mbps and that too with only 5 clients. The throughput started dipping
immediately after that, clearly indicating that the product is meant for small
workgroups. We repeated the same test on a compressed folder on the NAS, and the
maximum throughput rose slightly to 208 Mbps. Basically, the Windows Server
caches data on the fly before compressing and saving it in the compressed
folder. As the whole process happens in memory, it improves the performance a
bit. Our IOMeter test results indicated that the HP NAS is excellent in doing
sequential reading of data as compared to sequential writing and random
reading/writing. As compared to the other NAS boxes, this one was also a tad
better in doing random data writes of 64K data request size. Overall, we noticed
that its random reading and writing operations are not too affected by the data
request size.
But sequential reads and writes are immensely affected. When transfer
requests are increased from 64K to 128K, the sequential reads and writes are
reduced by half.
The AiO took 88 minutes to copy 100 GB of data to it, and the same data
compressed into an ISO image took only 51 minutes. While working with the NAS,
we observed that access to its management interface from the remote desktop was
a little slow. We further investigated and found lots of management services
running on the box. This was the reason for the slow-responding interface.
We stopped a few of these services and found that access to the management
interface became faster. We're not sure whether it also improves the overall
performance of the NAS as we didn't get sufficient time to check it. Perhaps
HP should look at this more closely. This was also the costliest NAS in the
shootout.
Price: Rs 315,000 (1 yr warranty) Contact: Hewlett Packard India, Bangalore Tel: 25051692 E-mail: neeraj.matiyani@hp.com SMS Buy 130222 to 6677 |
While sourcing NAS boxes for review, we approached Tandberg for an evaluation
unit. So we tested the NAS we use at Cybermedia Labs. It was a Viking FS-412
with four 400 GB hard disks. The available storage capacity in this was 1.02 TB,
and the rest was occupied by OS and RAID 5 configurations. It also supports RAID
0, 1 and 3, and shipped with Windows Storage Server 2003, 1 GB RAM which can be
expanded to 4 GB. It also supports various file systems such as DFS, NFS, HTTP,
FTP and SMB. One advantage it has over Snap and LevelOne is that it supports
file screening. It also provides various types of log reports like application
log, Web administration, FTP, NFS, security and HTTP share. It has support for
anti virus and also comes with a recovery CD. On the down side, it doesn't
back up to DVD.
NetBench |
^ IOMeter: I/Os per second |
We couldn't run the performance tests in the same way as we did on the
other NAS boxes since it is our production system. So its results can't be
compared against the other NAS boxes. We're including its results because we
observed something very interesting while testing it. While in the other NAS
boxes, we ran our tests on the entire available capacity, in the Tandberg NAS,
we created a separate 100 GB partition for the tests. The tests were, of course,
done after isolating the NAS box into a separate network so that nobody else was
accessing it. In NetBench, it gave a peak throughput of 218 Mbps with 5 clients.
After that, the throughput fell as we added more clients to it. The interesting
part came after we ran IOmeter on it. All scores were higher than all other NAS
boxes.
This could very much be, because we only used a 100 GB data partition to test
it. While copying 100 GB of data, it took 81 mins, which was second fastest, and
a compressed ISO image of the same file took 57 mins, which was interestingly
the slowest. So, while the NetBench and regular file transfer tests were
comparable to others, IOmeter was exceptionally higher, possibly due to the
smaller partition used.
Price: Rs 2,50,000 (1 yr warranty) Contact: Tandberg Data, Delhi Tel: 9810502321, E-mail: manish.mehta@tandberg.com.sg SMS Buy 130223 to 6677 |
Sanjay Majumder, Saurangshu Kanunjna, Swapnil Arora with help from Vijay
Chauhan