In Search of a Perfect Test

There's no easy answer. I've bought a cell phone which topped our reviews,

and then performed so poorly in the long run that I had to trash it. I've bought

a Ford sedan that topped comparisons, but has been plagued by a chronic problem.

I've bought a snazzy BlackBerry Pearl that topped review parameters and

features, but makes a terrible phone-and guzzles battery charge like there's no

tomorrow. And take the king of gadgets: I'm surrounded by owners trashing their

iPods, because of a dying battery or other glitches that are unfixable.

It's

tough for a reviewer to get beyond features and design, and to really get into a

buyer's, or an experienced user's, shoes. The problem becomes more acute with

enterprise-class products, like servers, where the cost of a wrong selection is

multiplied by the numbers and the mission-critical apps. But “more” is relative.

For a small business user, who's spent a hard-earned Rs 1 lakh on a laptop, the

cost of a bad selection is not just troubling, it's personal: it's his own

money.

A reviewer faced with a server to test for a thousand readers can get

overwhelmed by the gigabytes and gigahertz. So can a buyer. We have to do better

than that buyer, and recognize that over a five-year period the server may be

used. Raw performance and cutting-edge specs would bring far less value to the

table than reliability and predictability.

Prasanto K Roy,Chief Editor

One step we took as part of this

month's server shootout was to work with a user. A public sector organization

approached us for help with its server selection, ahead of a tender. We decided

to adopt the very detailed (but not cutting-edge) specs that this user needed,

to help us keep a focus on reality. So every spec was compared against what was

needed for this (fairly representative) application.

If a test unit is

expandable to 64 GB memory or has 16 hard disk slots, we compare that against

the headroom this user would actually need in the three to five year life,

demanded of the server. And so, top specs do not always get top brownie points.

Instead, fault tolerance, hot-swappability, overall disk performance, etc, get

higher weights.

What we could not do was really test the servers for long-term

reliability. We considered accelerated failure tests, such as heat, humidity and

vibration, in a suitably-equipped lab. We finally skipped those. Partly because

of the time, trouble and cost of these, and the fact that these are potentially

destructive-and we hadn't quite got the vendors to agree to destroying their

precious servers. And partly because enterprise-class rack-mounted servers,

including for this user, tend to be used, in fairly controlled environments.

So,

as a buyer, what you get from our reviews is a report on what's the best product

based on performance tests, features, and price/warranty, tempered with some

real-world experience. You're still stuck with figuring out long-term

reliability and support. For that, most buyers fall back on brand, reputation

and relationship.

Meanwhile, we're still looking for the perfect test. How we do

we factor in things that obviously matter a great deal to buyers? Should we add

softer factors like support and market reputation? (And, then we need to figure

out how to test these, other than long market surveys.) Tell me what you think!