No Benchmark Business Here

Sunil Kapoor is the CIO of an upcoming IT-savvy hospital near

Chandigarh. His mandate–the creation of a paperless environment. As Kapoor

goes about selecting hardware, he is subjected to marketing spiel from almost

all vendors operating in this segment. He has heard enough about benchmarks and

figured that all vendors, somehow, top the list in their respective categories

with respect to some parameter or the other. Kapoor has found a way to cut

through the hard sell. He has devised two simple rules: for systems with a

smaller applications budget, price is the key determinant while for bigger and

core applications, he tells the vendors, "Forget the benchmarks, lets run

the application in my environment." And the decision is based on these

results. No bench mark business here, declares Kapoor. However, it is difficult

for vendors to operate without benchmarks. More on that later.

India is one of the fastest-growing markets for servers and

all the major vendors want to have the biggest share of the cake. In FY 2000-01,

servers saw highest growth in value terms and revenues shot up from Rs1023 crore

to Rs1805 crore growing at 76% (DQ estimates). According to the DQ CIO

(September 15, 2001) CIOs say investment will continue in security, database and

network applications. To that add the discounts. The market remains hot for

servers and benchmarks are here to stay.

Benchmarks and vendors

CIOs need to differentiate between fact and the marketing

brouhaha being created by various vendors. While vendors need to address the

core issue of creating the necessary mind share by advertising, this is usually

at a broader level like IBM’s eServer campaign or Sun’s earlier ‘dot in

the dotcom’ campaigns. However, to demonstrate the product superiority over

others given the same processors and similar operating systems (OS) and

applications, the only option available is benchmarking.

Comments M Ganesh, Country Manager, Enterprise Server Group,

"Benchmarks ensure that the marketplace has a fair and useful set of

metrics to differentiate candidate systems. Also, benchmarking is important for

mid-range and high-end systems like Unix servers and storage systems, since they

are high value products with sophisticated technology and made for specific

computing applications." Adds Kamal Dutta, country sales manager, Unix

systems, HP, "Benchmarks provide us with the key to differentiate our

products from others and place before CIOs."Multiple benchmarks indicate

how a particular vendor’s system is likely to perform compared to another. The

most common benchmarks are- Processors like the SPEC (Standard Performance

Evaluation Corporation), generic ones like TPC (Transaction Performance

processing Council) or application specific performance. eg SAP.

Benchmarks and the CIO

So, how important are benchmarks in the CIO’s purchase

decision? The unanimous answer is ‘not very.’ According to the CIO’s we

spoke to, the key problem with benchmarks is that ‘somehow all the vendor’s

products top the results.’ It is true that most of the benchmarks are endorsed

by independent third parties and hence have a high degree of credibility.

Theoretically, it should help the CIO’s’ in comparing the performance of

competing products. But in reality, the CIO’s problem is a number of

benchmarks and vendors keen to talk about only those benchmarks that fit in with

their marketing compulsions.

For example, the SPEC standard has about seven to eight broad

benchmarks each with sub divisions. SPECint is subdivided into SPECint95 and

SPECint2000. This makes it easier for the vendors to publish only categories

where its products are superior. Adds Venky Mahadevan, VP-IS, Coca Cola India,

"While benchmarks are important , we did not use formal benchmarking tools

to evaluate server performance."

Rajeev Arya, Director — IT and Business Systems, Amway

points out that benchmarks are being done in a ‘controlled environment".

Comments Sunil Kapoor, Head-Information Technology, Fortis Healthcare, "We

don’t go with the results of benchmarking as we want to know the system

performance in our environment." In this respect, MNC’s with globally

defined systems and platforms are more fortunate. Ravi Parasuram, GM MIS,

Planning and Control Division, Sony India says, "Globally, Sony is on the

AS400 platform. So there is no discussing benchmarks with vendors".

Says Ganesh, "To effectively use a benchmark, it is

imperative to carefully evaluate its characteristics against those of the target

workloads." CIOs need to take the flaws of any testing methodology into

account. Testing in a simulated environment is likely to be done by users who

are familiar with the working of the system as against new users in the actual

environment.Results may be skewed due to this. In house testing can be the best

tool available or alternatively define your required metrics and make the vendor

accountable to deliver them. Where does that leave benchmarking? Millions are

spent on getting top rating benchmarks and flashing them in marketing campaigns.

Well, CIOs could use them to narrow down the field and shortlist technologies

that meet their needs best.

Yograj Varma In New

Delhi