What we're about to tell you is something nobody has ever told you before. Till now whenever any magazine (including us) has carried any shootouts, what's covered are performance results of products, how the products compared against each other, their pros, cons, etc. There's also a broad mention of the test procedures that were followed by the various features that were compared. More stress is given on the actual product reviews and the test results and less on the actual tests and how they were run. But what we're going to tell you this time is completely different. We'll let you in on the actual test procedures and all the excitement and heartaches we've experienced; and the learning that followed while doing this shootout. Instead of focusing on product evaluations alone and declaring a winner, we'll focus on uncovering how we actually ran the benchmarks on those products. And then, we'll tell you how to interpret their results, and finally we'll reveal the results we got on the NAS boxes. Why are we doing this you might ask?
For one, you'll be able to better understand the results we received for the NAS boxes. Two, if you ever need to, you can run those tests on your own storage boxes to find out their performance. And three, if you're planning to purchase a new NAS box, then you can even ask the vendors to conduct similar tests on their boxes and show you the results. In other words, we're empowering you with information to help you make a much better purchase decision now and in the future.
We've spent more than three months in testing the NAS boxes we received for evaluation. During this phase, we learnt a lot of new things, not only about performance, but also about features, vendor support issues, pricing models, and much more. So presented in this story therefore, is a complete buying guide to NAS boxes, with our personal learning. With this, you'll not only be able to decide whether or not to buy one of the four NAS boxes we tested, but any NAS box that's out there.
This is easier said than done. On the face of it, we ran just two benchmarks-Netbench and IOmeter. But when we actually got down to doing it, we ended up creating twelve different test mixes from this duo for testing various aspects of the NAS boxes. NetBench at one time was given away free of cost by Ziff Davis Labs of US. You just had to bear the shipping and handling charges for the CD. Though they've discontinued doing this, you'll still find it available for download at various websites.
The benchmark provides a lot of flexibility and control over the test setup and let's you create a real live scenario for testing the throughput and response time of NAS boxes and servers.
The other benchmark, IOmeter is also free and can be downloaded from http://iometer.org. There are dozens of different metrics to adjust while running this benchmark and it can reveal the true potential of any storage device. It measures the number of I/Os per second of the storage device and also gives the throughput in Mbps. You can also easily run these benchmarks in your setup, besides doing the regular file transfer tests.
The test environment
Just like any other network device testing, this one also requires an isolated network. Many organizations already have a separate test network that they use for testing anything new before incorporating it into the production network. If you don't, then maybe it's time you also created at least a basic setup. What you will need though is a Gigabit Ethernet switch and a few PCs that will act as the client load. The more PCs you have the better. We used 19 PCs in our test setup, and a 20th machine that acted as the controller for the remaining machines. On top of this of course, you could use a KVM over IP switch to manage all the nodes remotely.
throws a lot of I/O requests to the NAS box through multiple clients. It
records how quickly is the NAS box able to respond to these requests. At
the end, it just maps them on two graphs. One shows the maximum throughput
for each client set, and the other shows the associated response time. So
you can easily tell not only the peak throughput delivered by a NAS box,
but with how many clients did it deliver the same.
2. We tested the NAS boxes with network load balancing enabled and found that the performance actually went down instead of improving. This could possibly be because of the RRDNS process, which is balancing the load.
3. The I/O performance of a NAS box changes based on the volume size you use. As volume size increases, I/O performance reduces.
4. Never run performance tests on a production system. If you must do it, then you must first put it on an isolated network and ensure that there are no users accessing it.
5. If your NAS's performance suddenly deteriorates, check all HDDs. In RAID 5, even if a drive fails, the NAS continues to run, but with slower performance.
6. Rebuilding a RAID with a new disk takes 2 to 3 hours depending upon the RAID array size.
7. Perform network backups from the NAS after office hours, or when the network is free.
NetBench uses a number of PCs to generate file I/O requests to a shared location on the NAS. It loads the NAS slowly, starting with one PC and gradually continues to add more as per your test definition until it reaches the maximum defined load. You can also specify how long it should run and many other things (See screenshot). It gives you graphs of the I/O throughput and average response time at various load points. If you've loaded the NAS sufficiently, then there will be one peak throughput point, beyond which the throughput will start declining.
Setting up NetBench
The first thing you need to do is create a shared directory on your NAS box with full read/write access to everyone. Connect it to the Gigabit Ethernet switch. Then install the NetBench Controller program on a PC running Windows 2000/2003 or XP. You will define all benchmarks and control the entire test process from here. Next, you need the PCs that will send requests to the NAS box. These should ideally be of the same spec and preferably P4 machines running Windows XP. Install the NetBench client software on them. You can add up to 60 PCs to this load. Each PC can emulate several PCs, so you can simulate a much bigger load. Note that all machines have to be assigned an IP address manually. No DHCP please! Before starting the test, you need to configure the computer running the controller program. First, change the computer name to “controller”. Then open and modify the client ID file 'client.cdb' from CONTROLLER_DIR>\\CLIENTIDS. Here you'll find an entry for each client in your test bed with an associate ID number. Add each client's IP address and a unique ID number and save the file. On all clients, you need to map the shared folder on your NAS to the drive letter f:.
Running the tests
First run NetBench client on all the PCs. Then go to the controller machine and launch the controller program. Here, from the menu, select Clients>Start Log in option to connect all client machines to the controller. Then add the tests you want to run from Suit>Add. Here, open the DM.tst. Next, from the tool bar select the “Go” button to start the test. The default test takes about an hour to execute, but you can modify it as per your requirement. Let's see how to do that.
Modify NetBench tests
From the controller window choose Suites>Create/Edit Test Suites. Open DM.tst. You will see many test mixes (see screenshot on p.103). Each client represents one PC and each engine will run a test mix. So two engines on a client means it's going to run the same test mix twice, i.e. simulating the workload of two PCs. Note that there's no ideal mix. You have to define the right one as per the number of clients in your test setup and the device under test. We, used 7 test mixes, which gradually increased the load on the NAS from one client to 18 clients running 4 engines each, meaning 72 clients. Each mix took about 3 minutes to complete.
NetBench automatically creates a full test report with graphs showing throughput and response time for each test mix. From this, you can easily tell the maximum throughput and the number of clients and engines used to achieve it. You can compare this with reports for other NAS boxes to determine which one delivers higher throughput and with how many clients. All test results are converted into an Excel file by NetBench for easier analysis.
NetBench & file compression
Many NAS boxes support compressed folders, wherein any data stored on the NAS is compressed on the fly to give you more storage space. The compression does of course take a toll on the NAS's CPU cycles. What has to be checked is whether this affects the overall performance.
IOmeter is a widely used I/O performance analysis benchmark, specifically meant to test storage devices. IOmeter consists of a workload generator called Dynamo to do I/O operations and IOmeter itself to measure the I/O operations .The good thing about this benchmark is that there's a whole range of test mixes you can create. IOmeter is available on a variety of Operating Systems, including Windows, Linux, and even NetWare. You can even test a NAS box by running IOmeter from a single client. Installing IOmeter is fairly simple. You can install it on multiple client machines as well if you want, and in fact you should do that if you really want to stress the storage device. The only problem is that there's no central controller from where you can control all the IOmeter installations. You have to manually go to each machine to run it. But, like in NetBench, you need to create a shared folder on the NAS box and map it to all clients running IOmeter.
When you start IOmeter on both clients, Dynamo is automatically started. Here, look for the topology window and click on all the managers to expand the menu. Here you will see a default manager having with the same name as your client machine. By default it has two workers. This is similar to NetBench's client instances. In our tests we used four workers on every client. You can have up to 100 workers under one manager. Next look for the Disk Target tab. This shows you all available drives, including the mapped drive on the NAS.
Select it for all workers. Here, you can also specify the maximum disk size
to be used by IOmeter. By default this value is set to zero meaning
the entire disk will be used by IOmeter.
Simple file copy benchmark
Even if you can't these benchmarks, the easiest thing to do is a simple file transfer. We took about 100 GB of documents, spreadsheets, presentations, PDFs, and even MP3s and graphics files. We copied them to a shared folder on the NAS and measured how long it took. We even created an ISO image of this data and copied it to the NAS. This gave the NAS a continuous stream of data to store, and told us the maximum throughput that the NAS box is capable of delivering to a single machine.
Besides performance, you also need to look into the features offered by the NAS before buying. If you think a NAS is nothing but a set of disks connected together to offer storage space on the network, then think again. You're in for a surprise. We found lots of small but crucial points that you need to look into when scouting for a NAS.
Two key elements that can be expanded are hard drives and RAM. Not all vendors allow you to upgrade RAM in their boxes. Their claim is that their NAS boxes are optimized and tuned to work with the RAM they ship with. If you need a RAM upgrade, then the only option is to upgrade to a new box with more RAM. So besides spending time configuring the new NAS box to work on your network, you'll also be busy migrating all data from the old box to new one. Do check with your vendor on RAM upgradability before buying.
Hard drive upgrades are another story altogether. Many vendors don't allow you to just pick up a hard drive from the market and plug it into a free bay in your NAS. The reason for this is that every hard drive has its own firmware version, and the NAS vendor certifies hard drive models to work with its NAS. If you plug in a hard drive with a different firmware version into your NAS box, even if it's the same HDD brand, then the NAS box's warranty gets void. So what happens if one of the hard drives in your NAS box crashes? You can't plug in any hard drive you want in it. So either you buy a spare hard drive at the time of purchasing the NAS, or shell out extra money to pay the NAS vendor for a replacement. Needless to say that the amount you have to pay is directly proportional to how quickly you need the replacement!
This also opens up another interesting aspect. If you want to add a new hard drive to increase the capacity of your NAS box, then many vendors will charge you extra for it. The licensing models of some vendors are based on the capacity they provide you.
Here's another gem to worry about. Did you know that a 1 TB NAS box will never give you 1 TB of storage space, because a part of it is used up by the RAID and the OS? Even if you did, then you may not know that there can be major variations in how much storage space is used up by them. So a 1 TB NAS box could have anywhere from 600 to 700 GB of available storage space. That's a whopping 100 GB difference in available storage space! To give you an idea, consider a NAS box that has 250 GB x 4 disks, which amounts to 1 TB storage capacity. When you configure it in RAID 5, then one disk is gone, reducing the total available storage capacity to 750 GB. Further, the vendor may use more space to host the NAS OS, take snapshots for OS recovery, etc. This could easily take anywhere from 50 to 100 GB, leaving you with 650 to 700 GB of data storage space. Moral of the story: If you need 1 TB of storage capacity, then go for a 1.5 TB NAS.
Check whether a NAS box supports compression, as it will help you utilize the storage capacity more effectively. Data in every organization increases by leaps and bounds. So you might feel that you've bought a whopping 1 TB of storage capacity for your network now, but one year later, you'll be wondering where all that storage space went. At that point, you'll wish your NAS box supported compression. Some NAS boxes we received for evaluation were running propriety NAS OS and didn't support data compression. However NAS boxes running Windows Storage server had this feature. Performance wise also it gives you negligible difference over file storing without compression.
Monitoring and mgmt
Remote management and monitoring is what everybody wants in a NAS. A few features a NAS device should therefore have are-
Web-based interface, remote desktop connectivity, remote login through telnet or SSH, e-mail and SNMP notifications, disk quota management, file screening and storage reports. You could also look out for a LCD screen on the NAS itself. This can be pretty useful in giving you lots of critical information about the NAS device and letting you control it.
|Some more questions to ask|
|Q> Can I
upgrade the same box into a SAN?
Some vendors provide iSCSI support on their NAS boxes. They add a module of disk arrays and make the old box as a controller. The Controller further connects the disk array module via Ethernet and communication happens over IP.
Q> Can I upgrade my NAS capacity?
File Systems Supported
A typical network will have different types of clients: Microsoft, Novell, Apple and UNIX/Linux. A NAS device should appear on the network as a native file server to each of its clients with files saved and retrieved in their native file formats.
Most NAS devices support backing up data to a local tape drive or network backup server. Apart from this, also look for the following:
Snapshots: Incremental point-in-time copies of stored data are created on local devices' disks only. They allow easy data restoration without having to access the tape.
Disk Backup: Some NAS boxes come with a disk backup option, where backup is done on NAS storage only. For this you have to configure the RAID in such a way that at least one disk should remain free. For example if you have four drives, then create RAID with three drives and leave one for disk basedbackup.
Data replication: To keep an updated copy of critical data in a
separate location, online and ready for use at any time, thereby enabling faster
Device to device replication: Two NAS box are connected in parallel and synced for high storage availability. Check whether your NAS also supports automatic failover to the stand by NAS.
Backup to DVD: Another type of backup that some vendors provide. It's useful when you need to send out some very critical data.
A NAS device should easily be able to integrate with your existing directory service for authentication. Anti virus: How does your NAS vendor support anti-virus software? Some NAS devices allow you to install anti-virus software directly on them, while others let you do remote scanning from an anti-virus server on the network.
Access Control Lists: Allow file and share-level access for authenticated users and groups to files and folders stored on the NAS. Check what kind of ACL is supported by the NAS.
Logging: A good NAS box should audit and log most tasks done like log on, log
off, security modification, password changes, user creation, etc
E-mail and SNMP alerts: These are a given, and a concern if some vendor doesn't provide them.
It's criticla to prevents a user from filling up the entire storage space. Check for file screening support to control which types of files are allowed to be saved on the NAS. How comprehensive are a NAS's storage reports?
Today even an entry level NAS comes with dual NICs. Check whether it's able to do network load balancing, automatic failover, and teaming. The last option aggregates the two physical links, thereby doubling link speed.
Other features to check
A NAS box that can support databases is different from one that supports basic file storage, and their cost differential is also
significant. You can also look for a redundant power supply in the NAS.
Last but not the least is the price you pay. There's a huge variation in the prices between various NAS boxes.
It all depends upon how many features is the vendor bundling into the NAS. For instance, the HP NAS we received cost Rs. 3.15 Lakhs. It gave only 629.9 GB storage space, but also had tons of useful features. The Level One NAS on the other hand, was priced at only Rs. 98,500, gave storage space of 1.3 TB, but its features were limited.
This NAS was the best performer of the lot. The device runs Linux based Guardian OS, and has a simple and easy to understand Web interface for management. It also has a LCD display. Interestingly, its OS takes 10 GB space on every hard disk, which ensures that the server will keep running even if two or even three of the hard drives fail. It ships with 4 x 250 GB 7200 RPM hot-swappable SATA drives and 512 MB RAM, which can even be upgraded to a good 4 GB. The device also ships with a Manager application, which is useful only if you're running multiple Snap Servers in your data center.
The SnapServer can also work as an FTP or a Web server, and supports CIFS, SMB, NFS and AppleTalk file systems. It also has plenty of monitoring capabilities from regular system monitoring, users monitoring, open files monitoring to iSCSI device monitoring. The box shipped to us with CA Antivirus pre-installed, and even BakBone NetVault software for taking backups to tape drives. The box also supports taking snapshots and there's even an option to create a single image of its configuration and all its data. This can be great for DR purposes, as you can simply copy the image to another storage device or tape drive. After configuring RAID 5, the available storage capacity on the server comes down to 691 GB.
|^ IOMeter: I/Os per second|
On the performance front, like we said, it gave excellent results. It scored a whopping peak throughput of 608 Mbps in NetBench, and that too with 15 clients (whereas others peaked out at 5 clients). Even beyond 15 clients, its throughput remained higher than the other NAS boxes. In IOmeter, the I/Os per second it recorded for both 64 K and 128 K request sizes were the highest in sequential reading and writing of data. In transferring 100 GB of data, it only took 73, which was the fastest among all NAS boxes reviewed. Likewise, transferring a compressed ISO image of the same 100 GB of data was also the fastest at 44 minutes.
Rs 2,85,000 (3 yrs warranty on hardware, 90-day
Contact: Adaptec India, Bangalore,
SMS Buy 130220 to 6677
Cheapest of the lot, the LevelOne GNS 8000B was the only NAS box in the shootout to ship with a redundant power supply. It was also the NAS with the slowest processor, an Intel Celeron M 1.3 GHz. Plus, it had 512 MB RAM and a total storage capacity of 2 TB. However, its usable storage capacity comes down to around 1.3 TB, with the rest being used up for data protection. The NAS has an LCD panel, which apart from displaying information, also lets you configure its IP address. Then, of course, it has Web based management, which is pretty easy to use. It has hot swappable hard drives, and unlike many other vendors, who insist that you need to plug in a hard drive from them only, in this one, you can plug in one on your own. So if (God forbid), one of the hard drives crashes, you're not at the vendor's mercy to get yourself a replacement. The NAS supports file systems like NFS, FTP, AppleTalk and provides DHCP and event logs. Authentication is very simple and can be done in just 3 steps. This NAS supports ADS and LDAP based authentication, as also RAID levels 5, 0, 1, and 10, and comes pre-configured with level 5. You can't install an anti virus on it, so it has to rely on an external anti-virus server for protection. On the down side, it does not support file screening, so you can't control unwanted file extensions from being stored on it. It can back up data to tape, DVD and even supports snapshots.
|^ IOMeter: I/Os per second|
The NAS gave consistent performance in all the tests. Being an entry-level NAS, it's not meant to handle a very large number of clients. So the maximum throughput it gave in NetBench was 239 Mbps, and that too with only five clients. Beyond five clients, its performance dipped a little at 10 clients, but interestingly it picked up again at 15 clients, resulting in 238 Mbps throughput. It wasn't able to handle the load beyond 15 clients and its throughput dropped continuously. In IOMeter, its performance is better with smaller performance requests. So whether it was a random or sequential read or write request we sent to the NAS, the maximum IOs per second for 64k requests were always higher than 128k. Its performance in both sequential reading and writing is much better than in random. Among other NAS boxes also, this one gave better performance in random reads and writes for 128k requests.
Being an entry-level NAS, don't expect to do lots of file transfers to it. It took the longest to copy a 100 GB mix of various files, at 101 minutes. The same time reduced to half when we compressed these 100 GB files into a single ISO image. Overall, this is a good choice for very small setups.
|Price: Rs 98,500 (5
Contact: Minds (India), Delhi, Tel: 09818299663
SMS Buy 130221 to 6677
This is the latest NAS from HP, and is also the most feature rich of the lot. The first thing we noticed about it was its ease of use. You don't need a system administrator to configure it. And if its OS crashes, you can restore it by popping in the accompanying DVD and booting from it. The software does the rest. It automatically configures RAID 5 for you. The box has Windows Storage Server 2003, and also has data protection cum backup software for backing up data to tape, a network share, DVD or another NAS box directory. Other features in data protection include protection of shared folders and taking snapshots. This box supports iSCSI, allowing the NAS box to be integrated into a SAN environment. Plus, it also has tools for Exchange and SQL Server that let you host Exchange or SQL databases on the NAS box. While the total storage capacity of the NAS is 1 TB, the actual available capacity is 692.9 GB. Remaining space is used up by OS and data protection. On the upgradability front, you can't add any more RAM to it. The only way to get more RAM is to upgrade the entire box. The NAS box has a Web interface, which opens a remote desktop of the NAS box on the Web interface and lets you create and manage shares.
|^ IOMeter: I/Os per second|
On the performance front, the maximum throughput we got using NetBench was 203 Mbps and that too with only 5 clients. The throughput started dipping immediately after that, clearly indicating that the product is meant for small workgroups. We repeated the same test on a compressed folder on the NAS, and the maximum throughput rose slightly to 208 Mbps. Basically, the Windows Server caches data on the fly before compressing and saving it in the compressed folder. As the whole process happens in memory, it improves the performance a bit. Our IOMeter test results indicated that the HP NAS is excellent in doing sequential reading of data as compared to sequential writing and random reading/writing. As compared to the other NAS boxes, this one was also a tad better in doing random data writes of 64K data request size. Overall, we noticed that its random reading and writing operations are not too affected by the data request size.
But sequential reads and writes are immensely affected. When transfer requests are increased from 64K to 128K, the sequential reads and writes are reduced by half.
The AiO took 88 minutes to copy 100 GB of data to it, and the same data compressed into an ISO image took only 51 minutes. While working with the NAS, we observed that access to its management interface from the remote desktop was a little slow. We further investigated and found lots of management services running on the box. This was the reason for the slow-responding interface.
We stopped a few of these services and found that access to the management interface became faster. We're not sure whether it also improves the overall performance of the NAS as we didn't get sufficient time to check it. Perhaps HP should look at this more closely. This was also the costliest NAS in the shootout.
|Price: Rs 315,000 (1
Contact: Hewlett Packard India, Bangalore Tel: 25051692
SMS Buy 130222 to 6677
While sourcing NAS boxes for review, we approached Tandberg for an evaluation unit. So we tested the NAS we use at Cybermedia Labs. It was a Viking FS-412 with four 400 GB hard disks. The available storage capacity in this was 1.02 TB, and the rest was occupied by OS and RAID 5 configurations. It also supports RAID 0, 1 and 3, and shipped with Windows Storage Server 2003, 1 GB RAM which can be expanded to 4 GB. It also supports various file systems such as DFS, NFS, HTTP, FTP and SMB. One advantage it has over Snap and LevelOne is that it supports file screening. It also provides various types of log reports like application log, Web administration, FTP, NFS, security and HTTP share. It has support for anti virus and also comes with a recovery CD. On the down side, it doesn't back up to DVD.
|^ IOMeter: I/Os per second|
We couldn't run the performance tests in the same way as we did on the other NAS boxes since it is our production system. So its results can't be compared against the other NAS boxes. We're including its results because we observed something very interesting while testing it. While in the other NAS boxes, we ran our tests on the entire available capacity, in the Tandberg NAS, we created a separate 100 GB partition for the tests. The tests were, of course, done after isolating the NAS box into a separate network so that nobody else was accessing it. In NetBench, it gave a peak throughput of 218 Mbps with 5 clients. After that, the throughput fell as we added more clients to it. The interesting part came after we ran IOmeter on it. All scores were higher than all other NAS boxes.
This could very much be, because we only used a 100 GB data partition to test it. While copying 100 GB of data, it took 81 mins, which was second fastest, and a compressed ISO image of the same file took 57 mins, which was interestingly the slowest. So, while the NetBench and regular file transfer tests were comparable to others, IOmeter was exceptionally higher, possibly due to the smaller partition used.
|Price: Rs 2,50,000 (1
Contact: Tandberg Data, Delhi Tel: 9810502321,
SMS Buy 130223 to 6677
Sanjay Majumder, Saurangshu Kanunjna, Swapnil Arora with help from Vijay Chauhan