Removing Defects In Software Quality

During the last
three decades, software engineering practitioners have produced many prediction
models. However, it is difficult to predict the behavior of the software before
it is actually being used. A good defect removal model must be able to provide
early signals of warning or of improvement so that timely actions can be planned
and implemented. It should be possible to manage and monitor the quality of the
software when it is under development.

Defect removal is
one of the top expanse elements in any software project. Effective defect
removal leads to:

• Reduced development cycle
  time

• Reduced product cost

• Improved product quality

• Increased customer
  satisfaction

Background

Literature speaks
about the phase-based defect removal model used for predicting defects removed
at the end of each phase. Defects injected, removed and escaped in each phase
are expressed generally in terms of defects per KLOC (kilolines of code). The
KLOC size measure may not be accurate when we are in the software requirements
specifications (SRS) or design phase. In the current model, function point (FP)
is used for size estimate, which can be computed more accurately and from user
needs itself. International Function Point User’s Group (IFPUG) has
standardized FP as the size metric and International Organization for Standards
(ISO) has made a step to standardize it, since there is wide-spread belief that
FPs are a better size metric than LOC. In addition, we have considered the
phase-wise defect injection percentage to estimate the defects for each phase.
This gets around the problems of lack of uniformity and language dependence.

The closed loop defect removal
model

This model
accounts for injected defects and escaped defects separately. The creation of
the work product is subject to defect injection during SRS, design and
implementation phases whereas bad fixes cause for defect injection during the
testing phase. Defects are removed generally through one or other form of
reviews and testing.

Generating initial estimates
and tolerance limits

Using statistical
process control, the size of the project is estimated in terms of FP. The total
defect density (DD) value for the organization is baselined. It is expressed as
defects per FP. The total DD can be set based on other data too, like the
project’s defined software process, similarity to an earlier project and
equivalent team structure. The percentage defect injection is baselined for each
phase.

The mean, upper tolerance value
and lower tolerance value for injected defects in each phase is obtained as
mean number of defects injected into the phase = FP × Total DD × percentage
injection for the phase.

Upper tolerance value for
number of injected defects in SRS = (upper natural process limit — center
line) DefectDensitySRS × FP.

Lower tolerance value for
number of injected defects in SRS = (center line — lower natural process
limit) DefectDensitySRS × FP.

Defects to be removed at a
phase = defect removal effectiveness × (defects existing on phase entry +
defects injected during development of the phase).

The phase wise DRE for each phase
is set based on the organizational goals for process improvement.

The
initial estimates for the five phases–SRS, design, implementation, testing and
acceptance testing–are generated as per an algorithm. The testing defects
consist of unit test, module test and integration test defects.

Feedback mechanism

At the end of
each phase, actual number of defects obtained is fed to the model. This data is
an input to the decision process and action strategy to modify the process or
products. Determine if the defect detection profile is within the tolerance
limits established. If actuals are lesser than the lower natural process limit,
it can be due to lower error injection or ineffective reviews. If actuals are
more than the upper natural process limit, it can be due to higher error
injection or effective reviews. The effectiveness of reviews is determined based
on the effort spent for the review and the experience of the review team. High
and low values can be defined for the organization. We have made the following
guidelines for interpretation of the values.

If LOC per person hour review is
less than 30 LOC per person hour–assuming three persons review 100 LOC at a
rate of 100 LOC per hour for code review. Or if pages per person hour review is
less than three pages per person hour–assuming three persons review 30 page
document at a rate of 10 pages per hour for SRS and design review, the review
effort is assumed to be high, otherwise low. Similarly, if defects per KLOC is
more than 50 for code review or defects per 10 pages is more than seven for SRS
and design review, the defect injection is considered high.

If total experience per person of
the review team is more than three years it is considered as high.

Re-estimation is not required
when the actuals are within the tolerance limits. However, if the actuals are
consistently above the middle of the tolerance range, it indicates that the
baselined process needs to adopt an improved methodology for defect removal.
Conversely, this could indicate that the process was injecting more defects.

Re-estimation

With proper input
on the review effectiveness and defect injection, the model can re-estimate the
defects for each phase. If high defect injection is indicated by the data, the
estimates for subsequent phases are also incremented proportionately, keeping
the phase wise percentage injection constant. If reviews are rated ineffective,
defect removal effectiveness for the phase is calculated based on the actual
output. If both these conditions do not apply, the percentage injection for the
phases is revised, based on actual output. While doing this, the sum of
injection percentages should be maintained at 100, and the ratio of injection
percentages of subsequent phases should be maintained constant.

This will enable taking earlier
action to rectify problems that will otherwise not be found until later in the
development process. This provides developers with data in a timely manner to
discover departure from the development process and to return to control. The
re-estimates are done after the SRS, design and coding phase. Data obtained
during testing is not used to modify the estimates. The test data–including
unit test, module test and integration test–are used along with the review
data to predict the latent defects using Rayleigh model.

Tools used

The closed loop
defect removal model is implemented using MS Excel. The user will give the size
of the project and actual defect data as input and will get subsequent estimates
based on these. The baselines can be changed if desired.

Limitations

The model heavily
depends on the accuracy of FP estimates. It will not be accurate in cases where
there is a combination of high defect injection and ineffective review, or low
defect injection and ineffective review. These errors will be reduced after
re-estimations.

Application with actual data

Results found
after applying this model to a project is encouraging. A 100 FP project began
with a low defect density estimate, based on a similar past project. But after
the SRS phase, the actual defects obtained were high and they decided it was due
to high defect injection. All estimates were revised subsequently. The actual
defect for the design phase was lesser in number and they assigned the cause as
poor review. Hence the model took into consideration all the escaped defects and
revised the estimates for subsequent phases. The code review defects were also
lesser than predicted and hence the testing defects were re-estimated. They
focused more on the testing phase, and it was found that the actual estimates
came well within the limits for the testing as well as acceptance test phase.
The data was found useful for taking decisions on the quality aspects of the
project.

Predicting reliability

Number of latent
defects of the software product when available to customers is an objective
statement of the quality of the product. The latent defects are estimated by
using the Rayleigh model. An eight step developmental process with step size
equal to one–software requirements specifications, DES, coding, unit test,
module test, integration test, acceptance test and post acceptance test–is
used. The model parameters can be derived from the actual defect data obtained
upto acceptance testing phase. The end product reliability can be achieved by
substituting the data.

The effective model

From the above,
it is evident that the closed loop defect removal model is extremely effective
for in-process quality management. The project leader can plan effectively for
the effort and time to be spent for reviews based on the output given by the
model. The corrective actions can be taken during the course of the project
itself. On the higher end, the defect prevention activities can be applied
during development with proper planning and consequently we will end up with a
better quality product and improved customer satisfaction.

Excerpted from
Closed Loop Defect Removal Model Using SPC

Achamma Jose, Anju NK, SK Pilai

Courtesy: Network
Systems and Technologies, Trivandrum

This paper won QAI’s ‘Best of
the Best 2000’ award for the best paper category at the SEPG Conference India
2000, Bangalore, February 2000.