How to Choose and Benchmark a NAS

What we're about to tell you is something nobody has ever told you before.

Till now whenever any magazine (including us) has carried any shootouts, what's

covered are performance results of products, how the products compared against

each other, their pros, cons, etc. There's also a broad mention of the test

procedures that were followed by the various features that were compared. More

stress is given on the actual product reviews and the test results and less on

the actual tests and how they were run. But what we're going to tell you this

time is completely different. We'll let you in on the actual test procedures

and all the excitement and heartaches we've experienced; and the learning that

followed while doing this shootout. Instead of focusing on product evaluations

alone and declaring a winner, we'll focus on uncovering how we actually ran

the benchmarks on those products. And then, we'll tell you how to interpret

their results, and finally we'll reveal the results we got on the NAS boxes.

Why are we doing this you might ask?

For one, you'll be able to better understand the results we received for

the NAS boxes. Two, if you ever need to, you can run those tests on your own

storage boxes to find out their performance. And three, if you're planning to

purchase a new NAS box, then you can even ask the vendors to conduct similar

tests on their boxes and show you the results. In other words, we're

empowering you with information to help you make a much better purchase decision

now and in the future.

We've spent more than three months in testing the NAS boxes we received for

evaluation. During this phase, we learnt a lot of new things, not only about

performance, but also about features, vendor support issues, pricing models, and

much more. So presented in this story therefore, is a complete buying guide to

NAS boxes, with our personal learning. With this, you'll not only be able to

decide whether or not to buy one of the four NAS boxes we tested, but any NAS

box that's out there.

Choosing benchmarks



This is easier said than done. On the face of it, we ran just two benchmarks-Netbench
and IOmeter. But when we actually got down to doing it, we ended up creating

twelve different test mixes from this duo for testing various aspects of the NAS

boxes. NetBench at one time was given away free of cost by Ziff Davis Labs of

US. You just had to bear the shipping and handling charges for the CD. Though

they've discontinued doing this, you'll still find it available for download

at various websites.

The benchmark provides a lot of flexibility and control over the test setup

and let's you create a real live scenario for testing the throughput and

response time of NAS boxes and servers.

The other benchmark, IOmeter is also free and can be downloaded from http://iometer.org.

There are dozens of different metrics to adjust while running this benchmark and

it can reveal the true potential of any storage device. It measures the number

of I/Os per second of the storage device and also gives the throughput in Mbps.

You can also easily run these benchmarks in your setup, besides doing the

regular file transfer tests.

The test environment



Just like any other network device testing, this one also requires an isolated
network. Many organizations already have a separate test network that they use

for testing anything new before incorporating it into the production network. If

you don't, then maybe it's time you also created at least a basic setup.

What you will need though is a Gigabit Ethernet switch and a few PCs that will

act as the client load. The more PCs you have the better. We used 19 PCs in our

test setup, and a 20th machine that acted as the controller for the remaining

machines. On top of this of course, you could use a KVM over IP switch to manage

all the nodes remotely.

Understanding NetBench



NetBench uses a number of PCs to generate file I/O requests to a shared location
on the NAS. It loads the NAS slowly, starting with one PC and gradually

continues to add more as per your test definition until it reaches the maximum

defined load. You can also specify how long it should run and many other things

(See screenshot). It gives you graphs of the I/O throughput and average response

time at various load points. If you've loaded the NAS sufficiently, then there

will be one peak throughput point, beyond which the throughput will start

declining.

Setting up NetBench



The first thing you need to do is create a shared directory on your NAS box with
full read/write access to everyone. Connect it to the Gigabit Ethernet switch.

Then install the NetBench Controller program on a PC running Windows 2000/2003

or XP. You will define all benchmarks and control the entire test process from

here. Next, you need the PCs that will send requests to the NAS box. These

should ideally be of the same spec and preferably P4 machines running Windows

XP. Install the NetBench client software on them. You can add up to 60 PCs to

this load. Each PC can emulate several PCs, so you can simulate a much bigger

load. Note that all machines have to be assigned an IP address manually. No DHCP

please! Before starting the test, you need to configure the computer running the

controller program. First, change the computer name to “controller”. Then

open and modify the client ID file 'client.cdb' from CONTROLLER_DIR>\CLIENTIDS.

Here you'll find an entry for each client in your test bed with an associate

ID number. Add each client's IP address and a unique ID number and save the

file. On all clients, you need to map the shared folder on your NAS to the drive

letter f:.

Running the tests



First run NetBench client on all the PCs. Then go to the controller machine and
launch the controller program. Here, from the menu, select Clients>Start Log

in option to connect all client machines to the controller. Then add the tests

you want to run from Suit>Add. Here, open the DM.tst. Next, from the tool bar

select the “Go” button to start the test. The default test takes about an

hour to execute, but you can modify it as per your requirement. Let's see how

to do that.

Modify NetBench tests



From the controller window choose Suites>Create/Edit Test Suites. Open DM.tst.
You will see many test mixes (see screenshot on p.103). Each client represents

one PC and each engine will run a test mix. So two engines on a client means it's

going to run the same test mix twice, i.e. simulating the workload of two PCs.

Note that there's no ideal mix. You have to define the right one as per the

number of clients in your test setup and the device under test. We, used 7 test

mixes, which gradually increased the load on the NAS from one client to 18

clients running 4 engines each, meaning 72 clients. Each mix took about 3

minutes to complete.

Analyzing Results



NetBench automatically creates a full test report with graphs showing throughput
and response time for each test mix. From this, you can easily tell the maximum

throughput and the number of clients and engines used to achieve it. You can

compare this with reports for other NAS boxes to determine which one delivers

higher throughput and with how many clients. All test results are converted into

an Excel file by NetBench for easier analysis.

NetBench & file compression



Many NAS boxes support compressed folders, wherein any data stored on the NAS is
compressed on the fly to give you more storage space. The compression does of

course take a toll on the NAS's CPU cycles. What has to be checked is whether

this affects the overall performance.

Using IOmeter



IOmeter is a widely used I/O performance analysis benchmark, specifically meant
to test storage devices. IOmeter consists of a workload generator called Dynamo

to do I/O operations and IOmeter itself to measure the I/O operations .The good

thing about this benchmark is that there's a whole range of test mixes you can

create. IOmeter is available on a variety of Operating Systems, including

Windows, Linux, and even NetWare. You can even test a NAS box by running IOmeter

from a single client. Installing IOmeter is fairly simple. You can install it on

multiple client machines as well if you want, and in fact you should do that if

you really want to stress the storage device. The only problem is that there's

no central controller from where you can control all the IOmeter installations.

You have to manually go to each machine to run it. But, like in NetBench, you

need to create a shared folder on the NAS box and map it to all clients running

IOmeter.

When you start IOmeter on both clients, Dynamo is automatically started.

Here, look for the topology window and click on all the managers to expand the

menu. Here you will see a default manager having with the same name as your

client machine. By default it has two workers. This is similar to NetBench's

client instances. In our tests we used four workers on every client. You can

have up to 100 workers under one manager. Next look for the Disk Target tab.

This shows you all available drives, including the mapped drive on the NAS.

Select it for all workers. Here, you can also specify the maximum disk size

to be used by IOmeter. By default this value is set to zero meaning



the entire disk will be used by IOmeter.

Simple file copy benchmark



Even if you can't these benchmarks, the easiest thing to do is a simple file
transfer. We took about 100 GB of documents, spreadsheets, presentations, PDFs,

and even MP3s and graphics files. We copied them to a shared folder on the NAS

and measured how long it took. We even created an ISO image of this data and

copied it to the NAS. This gave the NAS a continuous stream of data to store,

and told us the maximum throughput that the NAS box is capable of delivering to

a single machine.

Key factors when buying!!!

Besides performance, you also need to look into the features offered by the

NAS before buying. If you think a NAS is nothing but a set of disks connected

together to offer storage space on the network, then think again. You're in

for a surprise. We found lots of small but crucial points that you need to look

into when scouting for a NAS.

Expandability



Two key elements that can be expanded are hard drives and RAM. Not all vendors
allow you to upgrade RAM in their boxes. Their claim is that their NAS boxes are

optimized and tuned to work with the RAM they ship with. If you need a RAM

upgrade, then the only option is to upgrade to a new box with more RAM. So

besides spending time configuring the new NAS box to work on your network, you'll

also be busy migrating all data from the old box to new one. Do check with your

vendor on RAM upgradability before buying.

Hard drive upgrades are another story altogether. Many vendors don't allow

you to just pick up a hard drive from the market and plug it into a free bay in

your NAS. The reason for this is that every hard drive has its own firmware

version, and the NAS vendor certifies hard drive models to work with its NAS. If

you plug in a hard drive with a different firmware version into your NAS box,

even if it's the same HDD brand, then the NAS box's warranty gets void. So

what happens if one of the hard drives in your NAS box crashes? You can't plug

in any hard drive you want in it. So either you buy a spare hard drive at the

time of purchasing the NAS, or shell out extra money to pay the NAS vendor for a

replacement. Needless to say that the amount you have to pay is directly

proportional to how quickly you need the replacement!

This also opens up another interesting aspect. If you want to add a new hard

drive to increase the capacity of your NAS box, then many vendors will charge

you extra for it. The licensing models of some vendors are based on the capacity

they provide you.

Available storage



Here's another gem to worry about. Did you know that a 1 TB NAS box will never
give you 1 TB of storage space, because a part of it is used up by the RAID and

the OS? Even if you did, then you may not know that there can be major

variations in how much storage space is used up by them. So a 1 TB NAS box could

have anywhere from 600 to 700 GB of available storage space. That's a whopping

100 GB difference in available storage space! To give you an idea, consider a

NAS box that has 250 GB x 4 disks, which amounts to 1 TB storage capacity. When

you configure it in RAID 5, then one disk is gone, reducing the total available

storage capacity to 750 GB. Further, the vendor may use more space to host the

NAS OS, take snapshots for OS recovery, etc. This could easily take anywhere

from 50 to 100 GB, leaving you with 650 to 700 GB of data storage space. Moral

of the story: If you need 1 TB of storage capacity, then go for a 1.5 TB NAS.

Data compression



Check whether a NAS box supports compression, as it will help you utilize the
storage capacity more effectively. Data in every organization increases by leaps

and bounds. So you might feel that you've bought a whopping 1 TB of storage

capacity for your network now, but one year later, you'll be wondering where

all that storage space went. At that point, you'll wish your NAS box supported

compression. Some NAS boxes we received for evaluation were running propriety

NAS OS and didn't support data compression. However NAS boxes running Windows

Storage server had this feature. Performance wise also it gives you negligible

difference over file storing without compression.

Monitoring and mgmt



Remote management and monitoring is what everybody wants in a NAS. A few
features a NAS device should therefore have are-



Web-based interface, remote desktop connectivity, remote login through telnet or
SSH, e-mail and SNMP notifications, disk quota management, file screening and

storage reports. You could also look out for a LCD screen on the NAS itself.

This can be pretty useful in giving you lots of critical information about the

NAS device and letting you control it.

File Systems Supported



A typical network will have different types of clients: Microsoft, Novell, Apple
and UNIX/Linux. A NAS device should appear on the network as a native file

server to each of its clients with files saved and retrieved in their native

file formats.

Data backup



Most NAS devices support backing up data to a local tape drive or network backup
server. Apart from this, also look for the following:

Snapshots: Incremental point-in-time copies of stored data are created

on local devices' disks only. They allow easy data restoration without having to

access the tape. 

Disk Backup: Some NAS boxes come with a disk backup option, where

backup is done on NAS storage only. For this you have to configure the RAID in

such a way that at least one disk should remain free. For example if you have

four drives, then create RAID with three drives and leave one for disk

basedbackup.

Data replication: To keep an updated copy of critical data in a

separate location, online and ready for use at any time, thereby enabling faster

DR.



Device to device replication: Two NAS box are connected in parallel and synced
for high storage availability. Check whether your NAS also supports automatic

failover to the stand by NAS. 

Backup to DVD: Another type of backup that some vendors provide. It's

useful when you need to send out some very critical data.

Security



A NAS device should easily be able to integrate with your existing directory
service for authentication. Anti virus: How does your NAS vendor support

anti-virus software? Some NAS devices allow you to install anti-virus software

directly on them, while others let you do remote scanning from an anti-virus

server on the network.

Access Control Lists: Allow file and share-level access for authenticated

users and groups to files and folders stored on the NAS. Check what kind of ACL

is supported by the NAS.

Logging: A good NAS box should audit and log most tasks done like log on, log

off, security modification, password changes, user creation, etc



E-mail and SNMP alerts: These are a given, and a concern if some vendor doesn't
provide them.

Storage Control



It's criticla to prevents a user from filling up the entire storage space.
Check for file screening support to control which types of files are allowed to

be saved on the NAS. How comprehensive are a NAS's storage reports?

High availability



Today even an entry level NAS comes with dual NICs. Check whether it's able to
do network load balancing, automatic failover, and teaming. The last option

aggregates the two physical links, thereby doubling link speed.

Other features to check



A NAS box that can support databases is different from one that supports basic
file storage, and their cost differential is also



significant. You can also look for a redundant power supply in the NAS.

Price



Last but not the least is the price you pay. There's a huge variation in the
prices between various NAS boxes.

It all depends upon how many features is the vendor bundling into the NAS.

For instance, the HP NAS we received cost Rs. 3.15 Lakhs. It gave only 629.9 GB

storage space, but also had tons of useful features. The Level One NAS on the

other hand, was priced at only Rs. 98,500, gave storage space of 1.3 TB, but its

features were limited.

Review: Snap Server 520 

This NAS was the best performer of the lot. The device runs Linux based

Guardian OS, and has a simple and easy to understand Web interface for

management. It also has a LCD display. Interestingly, its OS takes 10 GB space

on every hard disk, which ensures that the server will keep running even if two

or even three of the hard drives fail. It ships with 4 x 250 GB 7200 RPM

hot-swappable SATA drives and 512 MB RAM, which can even be upgraded to a good 4

GB. The device also ships with a Manager application, which is useful only if

you're running multiple Snap Servers in your data center.

The SnapServer can also work as an FTP or a Web server, and supports CIFS,

SMB, NFS and AppleTalk file systems. It also has plenty of monitoring

capabilities from regular system monitoring, users monitoring, open files

monitoring to iSCSI device monitoring. The box shipped to us with CA Antivirus

pre-installed, and even BakBone NetVault software for taking backups to tape

drives. The box also supports taking snapshots and there's even an option to

create a single image of its configuration and all its data. This can be great

for DR purposes, as you can simply copy the image to another storage device or

tape drive. After configuring RAID 5, the available storage capacity on the

server comes down to 691 GB.

NetBench

^ IOMeter: I/Os per second

On the performance front, like we said, it gave excellent results. It scored

a whopping peak throughput of 608 Mbps in NetBench, and that too with 15 clients

(whereas others peaked out at 5 clients). Even beyond 15 clients, its throughput

remained higher than the other NAS boxes. In IOmeter, the I/Os per second it

recorded for both 64 K and 128 K request sizes were the highest in sequential

reading and writing of data. In transferring 100 GB of data, it only took 73,

which was the fastest among all NAS boxes reviewed. Likewise, transferring a

compressed ISO image of the same 100 GB of data was also the fastest at 44

minutes.

Price: Rs 2,85,000 (3 yrs warranty on hardware, 90-day software support)  Contact: Adaptec India, Bangalore, Tel: 41339000 E-mail: Deepak_naik@adaptec.com  SMS Buy 130220 to 6677

Review: LevelOne GNS 8000B

Cheapest of the lot, the LevelOne GNS 8000B was the only NAS box in the

shootout to ship with a redundant power supply. It was also the NAS with the

slowest processor, an Intel Celeron M 1.3 GHz. Plus, it had 512 MB RAM and a

total storage capacity of 2 TB. However, its usable storage capacity comes down

to around 1.3 TB, with the rest being used up for data protection. The NAS has

an LCD panel, which apart from displaying information, also lets you configure

its IP address. Then, of course, it has Web based management, which is pretty

easy to use. It has hot swappable hard drives, and unlike many other vendors,

who insist that you need to plug in a hard drive from them only, in this one,

you can plug in one on your own. So if (God forbid), one of the hard drives

crashes, you're not at the vendor's mercy to get yourself a replacement. The

NAS supports file systems like NFS, FTP, AppleTalk and provides DHCP and event

logs. Authentication is very simple and can be done in just 3 steps. This NAS

supports ADS and LDAP based authentication, as also RAID levels 5, 0, 1, and 10,

and comes pre-configured with level 5. You can't install an anti virus on it,

so it has to rely on an external anti-virus server for protection. On the down

side, it does not support file screening, so you can't control unwanted file

extensions from being stored on it. It can back up data to tape, DVD and even

supports snapshots.

NetBench

^ IOMeter: I/Os per second

The NAS gave consistent performance in all the tests. Being an entry-level

NAS, it's not meant to handle a very large number of clients. So the maximum

throughput it gave in NetBench was 239 Mbps, and that too with only five

clients. Beyond five clients, its performance dipped a little at 10 clients, but

interestingly it picked up again at 15 clients, resulting in 238 Mbps

throughput. It wasn't able to handle the load beyond 15 clients and its

throughput dropped continuously. In IOMeter, its performance is better with

smaller performance requests. So whether it was a random or sequential read or

write request we sent to the NAS, the maximum IOs per second for 64k requests

were always higher than 128k. Its performance in both sequential reading and

writing is much better than in random. Among other NAS boxes also, this one gave

better performance in random reads and writes for 128k requests.

Being an entry-level NAS, don't expect to do lots of file transfers to it.

It took the longest to copy a 100 GB mix of various files, at 101 minutes. The

same time reduced to half when we compressed these 100 GB files into a single

ISO image. Overall, this is a good choice for very small setups.

Price: Rs 98,500 (5 yrs warranty) Contact: Minds (India), Delhi, Tel: 09818299663 E-mail: jerry.albert@level1.co.in  SMS Buy 130221 to 6677

Review: HP AIO 400

This is the latest NAS from HP, and is also the most feature rich of the lot.

The first thing we noticed about it was its ease of use. You don't need a

system administrator to configure it. And if its OS crashes, you can restore it

by popping in the accompanying DVD and booting from it. The software does the

rest. It automatically configures RAID 5 for you. The box has Windows Storage

Server 2003, and also has data protection cum backup software for backing up

data to tape, a network share, DVD or another NAS box directory. Other features

in data protection include protection of shared folders and taking snapshots.

This box supports iSCSI, allowing the NAS box to be integrated into a SAN

environment. Plus, it also has tools for Exchange and SQL Server that let you

host Exchange or SQL databases on the NAS box. While the total storage capacity

of the NAS is 1 TB, the actual available capacity is 692.9 GB. Remaining space

is used up by OS and data protection. On the upgradability front, you can't

add any more RAM to it. The only way to get more RAM is to upgrade the entire

box. The NAS box has a Web interface, which opens a remote desktop of the NAS

box on the Web interface and lets you create and manage shares.

NetBench

^ IOMeter: I/Os per second

On the performance front, the maximum throughput we got using NetBench was

203 Mbps and that too with only 5 clients. The throughput started dipping

immediately after that, clearly indicating that the product is meant for small

workgroups. We repeated the same test on a compressed folder on the NAS, and the

maximum throughput rose slightly to 208 Mbps. Basically, the Windows Server

caches data on the fly before compressing and saving it in the compressed

folder. As the whole process happens in memory, it improves the performance a

bit. Our IOMeter test results indicated that the HP NAS is excellent in doing

sequential reading of data as compared to sequential writing and random

reading/writing. As compared to the other NAS boxes, this one was also a tad

better in doing random data writes of 64K data request size. Overall, we noticed

that its random reading and writing operations are not too affected by the data

request size.

But sequential reads and writes are immensely affected. When transfer

requests are increased from 64K to 128K, the sequential reads and writes are

reduced by half.

The AiO took 88 minutes to copy 100 GB of data to it, and the same data

compressed into an ISO image took only 51 minutes. While working with the NAS,

we observed that access to its management interface from the remote desktop was

a little slow. We further investigated and found lots of management services

running on the box. This was the reason for the slow-responding interface.

We stopped a few of these services and found that access to the management

interface became faster. We're not sure whether it also improves the overall

performance of the NAS as we didn't get sufficient time to check it. Perhaps

HP should look at this more closely. This was also the costliest NAS in the

shootout.

Price: Rs 315,000 (1 yr warranty)  Contact: Hewlett Packard India, Bangalore Tel: 25051692 E-mail: neeraj.matiyani@hp.com  SMS Buy 130222 to 6677

Review: Tandberg Viking FS-412

While sourcing NAS boxes for review, we approached Tandberg for an evaluation

unit. So we tested the NAS we use at Cybermedia Labs. It was a Viking FS-412

with four 400 GB hard disks. The available storage capacity in this was 1.02 TB,

and the rest was occupied by OS and RAID 5 configurations. It also supports RAID

0, 1 and 3, and shipped with Windows Storage Server 2003, 1 GB RAM which can be

expanded to 4 GB. It also supports various file systems such as DFS, NFS, HTTP,

FTP and SMB. One advantage it has over Snap and LevelOne is that it supports

file screening. It also provides various types of log reports like application

log, Web administration, FTP, NFS, security and HTTP share. It has support for

anti virus and also comes with a recovery CD. On the down side, it doesn't

back up to DVD.

NetBench

^ IOMeter: I/Os per second

We couldn't run the performance tests in the same way as we did on the

other NAS boxes since it is our production system. So its results can't be

compared against the other NAS boxes. We're including its results because we

observed something very interesting while testing it. While in the other NAS

boxes, we ran our tests on the entire available capacity, in the Tandberg NAS,

we created a separate 100 GB partition for the tests. The tests were, of course,

done after isolating the NAS box into a separate network so that nobody else was

accessing it. In NetBench, it gave a peak throughput of 218 Mbps with 5 clients.

After that, the throughput fell as we added more clients to it. The interesting

part came after we ran IOmeter on it. All scores were higher than all other NAS

boxes.

This could very much be, because we only used a 100 GB data partition to test

it. While copying 100 GB of data, it took 81 mins, which was second fastest, and

a compressed ISO image of the same file took 57 mins, which was interestingly

the slowest. So, while the NetBench and regular file transfer tests were

comparable to others, IOmeter was exceptionally higher, possibly due to the

smaller partition used.

Price: Rs 2,50,000 (1 yr warranty) Contact: Tandberg Data, Delhi Tel: 9810502321, E-mail: manish.mehta@tandberg.com.sg SMS Buy 130223 to 6677

Sanjay Majumder, Saurangshu Kanunjna, Swapnil Arora with help from Vijay

Chauhan