Better Processes For Better Quality

Our Other

Sites

• PC
  
  Quest
• Voice&Data
• Computers@Home
• CIOL
• DQ
  
  Week
• DQ
  
  Channels India

Better Processes For Better Quality



High quality and productivity are the twin aims of any software development project. The selection of right processes are therefore crucial for both.

A software development project is one, in which a software product to fulfill some needs of a customer should be developed and delivered within a specified cost and time period. In other words, the three main characteristics of a project are its cost, schedule and quality, where ‘quality’ represents how well the product satisfies the customer. A project is generally initiated when some estimates for these parameters are established.

A project is successful if it meets or exceeds the

expectations on all three fronts–cost, schedule and quality. The software industry can cite many examples of projects that did not succeed. Although the situation has considerably improved over the years, many projects still fail to reach completion within budget, deliver within schedule, or fulfill quality expectations.

One analysis of project data shows that about one-third of projects have cost and schedule overruns of more than 125%. Examples of projects that are runaways–that is, out of control–have also been documented.

Possible reasons for project failures include improper estimation, loose

requirements management, weak project management, improper risk management and poorly engineered solutions, among others. Many of these reasons can be combined in one category called ‘process failure.’ That is, a software project often fails because the process followed in the project was not suitable. For example, the major reasons for runaways are unclear objectives, bad planning, new technology, no project management methodology and insufficient staff. At least three of these five reasons can be considered as ‘process failure.’ The other two–insufficient staff and new technology–can be considered as risks whose management is also a part of the process. For a project to succeed, a key success parameter is the set of processes followed in the project. If suitable process models are chosen for the important tasks in the project, and the chosen processes are executed properly, then the chances of a project succeeding become extremely high.

As having a high productivity can generally reduce cost and minimize the

schedule for a project, high quality and productivity (Q&P) can be viewed as the twin aims of a project for delivering a software product. Although processes are needed to satisfy the project goals, they are also essential for satisfying the objectives of an organization that is in the business of executing software projects. Of course, the organization will want all of its projects to succeed. It is larger than its projects, however, and has some desired objectives over and above the twin objectives of a project. First, an organization generally wants predictability. That is, it is not enough that a project has high Q&P–the organization also seeks a predictable Q&P. Without predictability, good estimation is not possible, and building reasonable estimates is essential to any project-oriented business. Second, an organization desires continuous improvement in Q&P.

Q&P of an organization depends on three factors–process, people, and

technology. This relationship is sometimes called the quality triangle. The quality triangle is similar to the process-technology-leadership triangle, also known as the iron triangle.

As the process has a major effect on the Q&P delivered by an organization, one way to improve Q&P is to improve the processes used by the organization. The personal software process proposed by Humphrey concentrates on improving the estimation and software development capability of individual software engineers. Whereas the People Capability Maturity Model focuses on improving the human resources in an organization. 

Technically, a process for a task comprises a sequence of steps that should be followed to execute that task. The software process consists of various processes that should be followed to perform different tasks in a project. For an organization, however, the processes are much more than a sequence of steps–they encapsulate the collective experience of that organization. That is, the processes capture past experience in executing projects, enabling the organization to leverage this experience in future projects. Essentially, processes incorporate what the organization has learned about successfully executing projects. By capturing the ‘recipes’ for success in the form of processes, and then following the processes in future projects, an organization ensures continued successes in its projects. The encapsulation of the experience of successful and effective engineers and managers and its dissemination to others, allow the benefits of experience to be conferred on even a newcomer in the organization. As a result, processes also play a key role in effectively managing the growth of an organization.

An established approach for improving the processes of an organization is to enhance the processes based on experience gained from successful and failed projects, such that the enhanced processes prevent failures in future and help emulate the successes. If an organization hopes to take software process improvement seriously, a considerable effort must be invested in capturing the experience with processes and plowing it back into the processes for leveraging the experience. Learning and leveraging experience with processes constitute an important aspect of CMM level 3.

Although learning and process improvement are always possible, identifying successes and failures and determining their causes become easier and less subjective if quantitative data are available about the performance of the process on projects. The organization’s ability to monitor a project’s process and apply proper control to ensure success is also considerably enhanced if the project goals are set in quantitative terms and quantitative information is available about the progress of the project. If quantitative information is available about the process capability, then improvements in the organization’s Q&P over time can also be determined unambiguously. This measurement can help quantify the return on the investments made in process improvement by the organization. In general, an organization’s ability to effectively plan and manage projects and improve its process is significantly enhanced if a quantitative approach is employed. Quantitatively managing the process is the focus of level 4 of the

CMM.

Capability Maturity Model for software



Once it is accepted that proper processes are essential for an organization to consistently deliver high quality and have high productivity, a question immediately arises–what are the desirable characteristics of an organization’s processes for executing software projects? The next issue is–how can the organization improve the processes for improving the Q&P, and what are the characteristics of the improved process? This area is where process frameworks play an important role. 

A process framework specifies some characteristics that the process must have to ‘qualify’ as a process of some maturity. The maturity of a process may be classified in some levels, and a framework may characterize a process in two or more levels. A framework specifies only the characteristics that processes at different levels should have; it does not prescribe any process. Thus different processes can fulfill the requirements of the framework. By specifying characteristics of processes for different levels of maturity, frameworks also provide guidance regarding the improvements needed to move from one maturity level to the other.

Many frameworks are available for software processes, including ISO 9001, CMM, Trillium, SPICE and BOOTSTRAP. Currently, the two most widely used and most influential models are ISO 9001 and the

CMM.

ISO 9001 is a general standard for providing service. It has been specifically interpreted for software in ISO 9000-3. TickIT provides further guidelines on how software organizations should use ISO 9001. ISO 9001 has 20 clauses that an organization must satisfy to qualify as ‘ISO certified.’ The ‘ISO certified’ category does not contain any additional distinctions. Further improvement is generally handled through the auditing process. In other words, the ISO 9001 framework includes only two levels. The model is general and considers the working of the entire organization–not just its software projects.

The CMM for software is a framework that focuses on processes for software development. It was developed by observing best practices in software organizations as well as non-software organizations. Hence, it reflects the collective process experience and expectations of many companies. It can be used both to evaluate the software process of an organization and to plan process improvements.

Maturity levels in CMM



One objective of the CMM is to distinguish mature processes from immature or ad hoc processes. In a software organization that has immature software processes, projects are executed without many guidelines, and the outcome of a project depends largely on the capability of the team and the project leader. Consequently, the result is not predictable. On the other hand, in an organization with mature processes, a project is executed by following various processes that the organization has set up for software projects. In this case, the outcome of the project is less dependent on the team capability and more controlled by the processes. It follows, then, that the more mature the processes, the better the control on the projects and the more predictable the results.

The range of results that can be expected in a project when it is executed using the software process of an organization is the software process capability. The actual result achieved in a project executed using the software process is the software process performance. Clearly, the process performance depends on the process capability. To improve process performance consistently on projects, the process capability must therefore be enhanced–the process must become more mature.

The path to higher maturity includes some well-defined plateaus that are viewed as maturity levels by the CMM. Each maturity level specifies certain

characteristics for processes, with higher maturity levels having more advanced characteristics that are found in more mature software processes. Hence, the CMM framework describes the key elements of software processes at different levels of maturity. Consequently, it also specifies the path that a software process follows in moving from immature and ad hoc process to highly mature process. This path includes five maturity levels.

In level 1, an organization executes a project in a manner that the team and project manager see fit. At the defined level (Level 3), the processes are used on an ad hoc basis. The repeatable level (level 2) applies to an organization in which project management practices are well established, although

organization-wide processes may not exist. At the defined level (level 3) the software processes for the organization have been precisely defined and regularly followed. With organization-wide processes, the organization may learn from different projects and subsequently improve the processes to benefit future projects. At the managed level (level 4) quantitative understanding of the process capability makes it possible to quantitatively predict and control the process performance on a project. Once the foundation of quantitative process management exists, then the process capability can be improved in a controlled manner and the improvement can be evaluated quantitatively. At the optimizing level (level 5), the process improves continuously, with level 4 providing the mechanisms to quantitatively evaluate the effectiveness of process enhancement initiatives.

Several reasons exist for selecting these levels. These are:

• They represent the phases observed in organizations as their processes evolve and mature.

• Each level represents some reasonable process improvement from the previous level.

• The levels provide guidance in defining a set of process improvement areas, once the current level of the organization is determined.

Each maturity level (except level 1) is characterized by some key process areas (KPAs), which specify the areas on which the organization should focus to elevate its processes to that maturity level. Most of the KPAs at level 2 focus on project management, whereas the KPAs in level 3 target institutionalization of processes and some additional processes for engineering of software. The KPAs at level 4 revolve around quantitatively managing the process and projects, and the KPAs at level 5 focus on process improvement through defect prevention, technology introduction and process enhancements.

Maintaining processes at higher levels of maturity is a challenging task requiring commitment from the organization and a proper work culture. Of the 700

assessments that were conducted between 1992 and 1997 and whose assessment results were provided to the SEI, approximately 165 organizations were assessed at level 2, 105 at level 3, 16 at level 4, and 4 at level 5. That is, of the 700 organizations, only about 20 were at level 4 or 5. However, the number of high-maturity organizations is growing rapidly.

KPAs in different levels



The KPAs for a particular level can be considered as the requirements for achieving that maturity level. For an organization to achieve a level, all of the KPAs at that maturity level as well as the KPAs at all lower maturity levels must be satisfied by the processes of that organization. Each KPA specifies some goals that the processes of the organization must meet to satisfy that
KPA.

In addition, each KPA specifies a group of activities, called key practices, that can collectively satisfy the goals of that

KPA. The key practices are organized into various groups called commitment to perform, ability to perform, activities performed, measurements and analysis, and verifying implementation. The activities under ‘commitment to perform’ describe the actions that the organization must take to support the particular

KPA. ‘Ability to perform’ activities focus on issues like training, resource requirements and control structures, all of which are important for developing the ability in specific personnel and the organization as a whole to perform activities for that

KPA. ‘Activities performed’ describes the actual process activities that are recommended. ‘Measurements and analysis’ targets the measurements that should be done for the activities of the

KPA, and ‘verifying implementation’ activities focus on ensuring that the implementation of the processes is verified through independent persons and senior management.

In many senses, the goals for each KPA capture the essence of that KPA. They specify the objectives that the CMM has set for the processes relating to the KPA. Although the goals can be satisfied by an organization performing the key practices, they can also be satisfied by alternative practices that differ from the key practices specified in the

CMM.

The goals of requirements management (RM) ensure that the requirements are properly documented and the requirement changes are properly managed in the project. Note that the process for analyzing and specifying requirements is not a goal of RM. Goals of software project planning (SPP) ensure that proper planning is done for a project, which includes an estimation and a listing of activities to be performed, and that the plan is documented. Goals of software project tracking and oversight (SPTO) ensure that, during the project execution, the actual performance of the project is evaluated against the plans. Actions are taken when the actual performance deviates from the plans significantly.

Software subcontract management (SQA) focuses on reviews and audits carried out to ensure that proper processes are followed. Its goals also make sure that quality assurance activities are planned and that proper actions are taken when the project fails to comply with the established standards and processes. Goals of SCM ensure that programs and documents that must be controlled in the project are identified, that changes to them are controlled, and that these activities are properly planned. The software subcontract management (SSM), KPA is applicable when the organization that is developing software

subcontracts some parts of the development effort to another organization. This KPA is usually not applicable to organizations that handle all activities relating to the project themselves.

The goals of organization process focus (OPF) ensure that process definition and improvement activities are executed in a planned manner in the

organization, which requires the formation of a group dedicated to process activities. Organization process definition (OPD) goals require that the processes are defined and documented and that information about the use of the process is collected and made available to other projects. That is, the KPA ensures that, in addition to having defined processes, the experience of other projects about how to make the best use of the processes becomes available, so a new project can leverage past experience.

The goals of training program

(TP) ensure that the organization has identified the training needs for the various roles and that project people receive necessary

training in a planned manner. The goals of integrated software management (ISM) require that the process used for a project be tailored from the standard process; the defined processes are then used for managing the project. The goals of software product engineering

(SPE) focus on the engineering tasks being performed properly in the project, and the different work products remaining consistent, even under the face of changes. The intergroup

coordination (IC) KPA becomes more of an issue when multiple engineering groups are involved–for example, when a system is being built that may require mechanical, electrical and software systems. In a general sense, however, it tries to ensure that the interfaces between the different groups that contribute to a project are properly defined and work smoothly. The goals of peer reviews (PR) ensure that the peer review activities are properly carried out in a project and that sufficient support for conducting peer reviews and follow up activities is provided.

Level 4 includes only two KPAs–QPM and

SQM. The goals of QPM ensure that the capability of the organization process is understood quantitatively and that the process capability is employed to set quantitative goals for a project. Data on actual performance of the current project are collected and then compared with data on past performance; if significant deviations are observed, proper corrective actions are applied to bring the project back in control. In

QPM, it is expected that monitoring and corrective actions will take place on an ongoing basis. The goals of SQM require that the project set quantitative quality goals and have suitable plans for achieving these goals. In this

KPA, the actual performance of the project is also measured and compared with the planned progress. Significant deviations ‘trigger’ corrective actions.

The three KPAs at level 5 focus on improving the capability of the process. The DP KPA requires that the defect prevention be done proactively–by systematically analyzing the causes of defects and then eliminating those causes. If defects can be prevented from entering the software, then the effort spent in removing them can be reduced, thereby improving quality and productivity. Similarly, TCM focuses on the proactive introduction of technology in the organization to improve quality and productivity. Goals of PCM require that process improvement take place continuously, in a planned manner, and with the involvement of a large cross section of the organization. Although not explicitly mentioned in the goals, the fact that level 5 comes after level 4 implies that effects of defect prevention, technology and process change be measured quantitatively at level 5.

Assessment method



One goal of standards or frameworks is to provide some consistency in evaluation so that the framework can be used as a basis for comparison with a different organization. Achieving this goal requires that the evaluation of processes with respect to the framework must be standardized such that the outcome of evaluation is independent of the evaluator. Detailed instructions for conducting an assessment appear in the SEI’s handbook for assessors. 

The approach that organizations use for their process assessment and improvement is called the

CMM-based appraisal for internal process improvement (CBA-IPI). Another form of assessment, called software capability evaluation, usually takes place at the request of someone–typically a customer–outside the organization being evaluated. The CBA-IPI is intended to help the organization being assessed improve its processes. It therefore evaluates the strengths and weaknesses of the software process in addition to evaluating the satisfaction of the different goals of the different

KPAs.

The assessment is performed by an assessment team, which is led by an SEI-authorized lead assessor, and consists of 6 to 10 experienced people from the organization under scrutiny. The team members must be familiar with CMM and processes-related issues and receive assessment training from the lead assessor. During the course of the assessment, the team members collect information about the software process of the organization. 

There are three main sources of information:

• Maturity questionnaires.

• Documentation.

• Interviews.

A maturity questionnaire is an instrument that is used to get some feedback regarding the process being used in the organization. It contains a set of questions for each

KPA, most of them based on the key practices under the KPA. The questions ask whether a practice is being followed with possible answers being ‘yes,’ ‘no,’ ‘don’t know’ and ‘does not apply’. As a first step in assessment, the questionnaire is given to project leaders, their supervisors and some project team members. The answers are then compiled. If the answer to a question is overwhelmingly ‘yes,’ then the statement in the question could be treated as an observation about that practice. The assessment process requires at least two independent observations from two different sources before confirming that a practice is being followed. Frequently, however, a maturity questionnaire is used as an aid for further exploration during interviews and document examination.

For the assessment, four to six projects are selected. These projects should be representative of the project profile of the organization. Documentation from these projects is made available to the assessment team, and their project leaders are interviewed individually to obtain more observations and seek any necessary clarifications. Besides documents of the selected projects, documents describing the processes are examined for the purpose of making observations.

In addition, groups of technical personnel representing different functions are interviewed. These groups are called functional area representative (FAR) groups. In an assessment, four to six FAR groups may be interviewed, with each FAR group having four to ten people. 

Possible FAR groups include the following:

• Project leaders.

• Middle managers to whom project leaders report.

• Configuration controllers.

• Software Engineering Process Group members.

• Training personnel. 

• Developers.

• Testers.

• Analysts.

The interviews attempt to obtain evidence regarding the usage of key practices of the different KPAs. Generally, the maturity questionnaire and the examination of the project documents and the documents regarding the processes and policies will yield some evidence for most key practices. Interviews therefore become a means for clarifying doubts and as additional sources of information.

Between the information-gathering sessions, the team consolidates whatever information it has gathered and reaches agreement on which key practices can be considered as satisfied–a key practice is considered as satisfied, if two independent information items support it. Agreement requires consensus of the entire team. 

To consolidate the information on a

KPA, a coverage sheet is used which lists all requirements for the KPA and provides space for making observations. For consolidation, the assessment team might be organized into multiple mini-teams, with each mini-team handling a few

KPAs. After consolidating the available information, the mini-team responsible for a KPA presents the results to the entire assessment team. As noted earlier, a key practice is considered as ‘satisfied’ only when all team members agree. If doubts persist, then these ‘gaps’ are identified and clarified during the interview sessions through appropriate questions and requests for documents. As we can see, the information-gathering steps in the process are interleaved with consolidation activity to allow proper assimilation of information. If all key practices are satisfied, then the goals for those KPAs are satisfied. Otherwise, goal satisfaction requires the team to determine whether an alternative practice exists that satisfies the goal. If all goals of a KPA are satisfied, then the KPA is satisfied. An organization is considered to have reached a level if it satisfies all KPAs for that level and for all levels below it.

Excerpted from  CMM in Practice  By Pankaj Jalote Published by Addison-Wesley For queries: jalote@cse.iitk.ac.in

During consolidation, the strengths and weaknesses for each KPA are noted. In the presentation of the final findings, the goal satisfaction for all KPAs is presented, along with the strengths and weaknesses information. In this manner, the final findings not only report the maturity level of an organization, but also delineate the areas in which improvement is possible. These areas can then be used to plan additional process improvement activities.

Dr Pankaj Jalote is Professor and Chairman of Dept of Computer Science and

Engineering at the Indian Institute of Technology, Kanpur.