Testing Times

Case I

In May 1995, 823 customers of a major bank in the US suddenly found $900
million credited into their accounts. They were not sure whom to thank till it
came to light that a programming error was responsible for their swelling credit–perhaps
the most costly error in banking history.

Case II

Nike’s shortfall of $100 million in the third quarter of 2001 was due to
flaws in its supply chain management system. 45% of all system downtime in the
US in 2000 was due to defective systems.

The situations described above are a nightmare for the software industry–for
enterprises get paralyzed due to faulty software. Defective software results in
huge losses. In the US alone it runs into billions of dollars and it is
estimated that losses represent 1% of the country’s GDP.

Why do software programs or products fail in the first place? The answer
probably lies in the multi-pronged issues that affect software quality itself.
For instance, the software sold in the market comes out with disclaimers, end
user license agreements (EULA) and limited liability clauses from the vendor.
Software solutions also fail because the vendor and the user companies find
great difficulty in measuring quality. While vendors base their products on
common quality attributes like reliability, efficiency, and functionality, in
reality there are many glitches that surface in user conditions. Oversight of
these defects by the enterprises often leads to catastrophic consequences.

This view only gets reinforced, once we look at what Gary Chapman, director
of the 20th century project conducted by the University of Texas noted,
"Repeated experiences with software glitches narrows down the use of
computers to familiar and routine tasks. Our studies have shown that most of the
users rely on less than 10% of the features of common programs like word
processors and browsers." Users remain unaware of the fact that they are
using defective software.

As Sashi Reddi, CEO, National Software Testing laboratories (NSTL) India puts
it, "Software has been historically plagued by defects. In other
industries, where defects are considered as a serious flaw, a user of software
is comfortable if the solution merely works. Some of this lax approach to
software quality is changing as a result of growing recognition of the costs
incurred due to defects in the software."

The evolution

Software testing for a long time was centered on fixing bugs in the program
only when the user encountered it. This was more on a demand basis than
pre-empting the problem at the project completion stage. As Watts Humphrey, the
software quality guru claimed, "bugs are like landmines that can explode
any moment." This observation sums up the consequences of defective
software processes. According to KR Jayakumar, CEO, Amity Soft Technologies,
"The concept has considerably evolved in the recent past. Testing which
used to be an after thought has now evolved into a major activity. Moreover
software tools have changed the landscape, and as a result testing has become a
specialized discipline." "Independent third party testing is the peak
of this maturity, it signals a new breed of software professionals and an
industry by itself," adds Jayakumar. One of the biggest drivers that will
infuse more quality into software will come when enterprises become more
conscious of the deliverables promised by the vendor.

A recently published report of the National Institute of Standards and
Technology (NIST) says, " Software testing is the process of applying
metrics to determine the product quality. It is the dynamic execution of the
software and the comparison of the results of that execution against a set of
predetermined criteria." Analysts firmly believe that software testing is
an infrastructure technology that provides three main quality attributes–conformance,
performance, and interoperability–to a solution.

Comments A V Asvini Kumar, CEO, Thinksoft Global, "Testing is always a
part of SLA’s and typically takes up 30% of all efforts spent on a software
project. But in the last two decades it has assumed significance because,
software now dominates our lives in myriad ways. Also software now has become
mission critical, life critical, and we are dependent on complex systems for our
survival and hence delivering quality systems has become a mandatory
requirement."

The market scenario

Testing the solutions thoroughly before it reaches the end-user is a major
mandate every software vendor faces today. The one way of measuring quality is
through a wide range of testing tools available in the market. However, the
market is still fragmented. Hence the testing procedures adopted by companies
vary. This leads to issues of standardization and testing methodologies.
Industry estimates put the market for software testing tools at $ 931 million
and say that it will grow to $ 2.6 billion by 2004.

The market for testing tools can be divided into two broad streams–application
and web testing tools. An application test tool evaluates the software’s
source code, functionality etc. A web testing tool addresses requirements like
HTML testing, load /traffic management among others. In the testing tool space,
companies like Rational Software have carved a niche for themselves. Rational
for instance has got a slew of testing solutions (Pure Coverage, Purify, Visual
Test, Rational Robot etc) that improves the speed, quality, and the
predictability of the software projects. Says Eric Shurr, chief marketing
officer, Rational Software, "TestStudio and other Rational testing tools
liberate development teams from the time consuming manual tasks by automating
the testing processes" Other major players in the testing firmament include
names like Segues’ SilkTest, WinRunner from Mercury and Compuware’s QA Run
and AutoTester.

Closer home, the market for software testing is still nascent. The handfuls
of companies doing this kind of work are off shore in nature. Comments Asvini
Kumar, "In India we have not evolved to a stage where information is a
valued as a commodity. Hence the low index of deployment, usage of IT
applications puts quality issues on the back burner as resources, and
investments become top priority. "But this situation is gradually changing
and five years down the line we will see a lot of action in this space,"
adds Kumar.

Meanwhile Sashi Reddi of NSTL has a slightly different view. He believes that
Indian enterprises are clearly aware of this concept and have already started
demanding quality solutions. He goes on to say, "India has the largest
number of CMM level 4 and 5 developer companies. These companies have clearly
recognized quality benchmarks as a distinct competitive edge."

The economic impact

Let us look at couple of events that sums up the far-reaching consequences
of defective software. In April 1999 a bug played truant with the $1.2-billion
US defense satellite launch. This is one of the costliest unmanned accidents in
recent times. Close on its heels, in October 1999, the $125 million NASA Mars
Climate Orbiter was lost in space due to a small data conversion error.
Investigations into the incident revealed that the Orbiter frayed out of the set
path because the software used some data in English units that should have been
in metric units. The third incident occurred in December 1999 when the Mars
Polar Lander mission failed for unknown reasons. But critics dubbed that one
also as due to defective software processes. The failed missions exposed the
loopholes in the US space program, prompting the government to order a review of
space launch programs including the software used and the testing processes. The
result: the US government lost more than billion dollars due to software
problems overlooked at the testing stage.

Analysts estimate that in the US alone inadequate software testing
infrastructure leads to losses to the tune of $22 to $60 billion. Interestingly
half of the costs are borne by the users who ultimately suffer due to errors in
the software. Given the cost implications, both to the vendor as well as the
user companies, they are now putting in place or seriously considering a
software testing architecture as part of their MIS policy. Says Jayakumar,
"Enterprises will no longer be able to absorb costs due to defective
software due to the highly competitive and low margin economic situations. On
the developer side we see a growing degree of realization in testing their
solution before it reaches the end user. For instance we are undertaking three
major testing projects this month alone, from major companies. This reflects how
quality-conscious developers are becoming in recent times."

The Challenges

Despite growing awareness, software testing even today becomes an activity
towards the end of the product development lifecycle. This leads to testing that
is not comprehensive, and hence results in the release of sub-quality software.
Reflecting on this Sashi Reddi says, " The biggest challenge is to convince
developers and enterprises to have a QA department working with the IT teams
right from the moment of conception to implementation. Another main challenge is
to enforce a clear separation between the development team and the testing team
that will truly foster independent testing."

Latha Ramesh, a testing specialist with Chennai-based HiQ Consulting says
that the dearth of qualified testing professionals is a major challenge
confronting the testing industry. She says that in India software testing is not
looked upon as a career choice since many consider it as a mundane exercise.
"This perception has to change, only then can we kickstart a quality
revolution," she adds.

Other issues like lack of standardized testing procedures and competitive
pressures in delivering a software solution in the shortest possible time are
potential threats to software quality. As Asvini Kumar puts it, " The
pressure to reduce both costs and the time to market, unwittingly results in
shortsighted budgeting approaches and as a consequence, systems are delivered
with invisible defects."

Also, software project managers and development teams are quite attached to
the product and hence it takes a lot of courage to spell out the problems with
the developers to correct them. "Test project management is the toughest
job," says Jayakumar. In the end, what emerges is that there’re still a
lot of inadequacies in the realm of software and testing. But across the world,
vendors and users are becoming increasingly conscious about the solutions they
are developing and implementing. The emergence of quality benchmarks like ISO
9001 and SEI-CMM defined levels have ushered in a quality culture. But still,
invisible errors unknown to the developer create chaos after implementation.

Shrikanth G in Chennai