Cut through that code

Software development suffers from self-inflicted complexities...a lack of

appreciation of the power of mathematics and the semantically rich symbolic

systems

Software development has at least two sources of complexity: complexity in

herent in the problem domain over which the     software developer has no control,

and self-inflicted complexity arising from the activities of the software

development team. It is the second, which is of concern here because it is so

widespread.

A software product is a knowledge artifact. Therefore intellectual control at

every stage in its production is essential. Curiously, while this fact is

universally acknowledged, it is seldom practiced. A programming language

represents a formal mathematical system but programmers are seldom aware of this

fact because they are rarely taught to view it as such. An important aspect of a

mathematician’s training is learning to avoid self-inflicted complexity in one’s

work and to focus on resolving domain complexity. The short-sightedness of the

software engineering community in not training its practitioners into comparable

standards of intellectual rigor and discipline lies at the root of the so called

software crisis, which is well-described in the following quote from a widely

referenced article by Gibbs: "Studies have shown that for every six new

large-scale software systems that are put into operation, two others are

cancelled. The average software development project over-shoots its schedule by

half; larger projects do worse. And some three quarters of all large systems are

‘operating failures’ that either do not function as intended or are not used

at all."

Software development suffers from self-inflicted complexities essentially

because the software engineering community does not demand professionalism from

its practitioners. For example, a formal software development approach pays a

lot of attention to development methodologies such as the Waterfall,

prototyping, or spiral approaches, and support and monitors them with a formal

quality management system. This works well in manufacturing industries where

volume production by replication of a product already developed must be achieved

flawlessly and consistently, and where incremental improvements are required to

sustain and open up newer markets. A focus on methodologies is misplaced if

replication costs are negligible but creativity in analysis and design is

paramount to success. Indeed some of the best software developed has not gone

through a formal methodology. The experience of Stroustrup, the famous designer

of C++, echoed in the following quote from his book on C++, is well-known among

top software developers: "Until recently, there never was a C++ paper

design; design, documentation, and implementation went on simultaneously. There

never was a ‘C++ project’ either, or a ‘C++ design committee."

Throughout, C++ evolved, to cope with problems encountered by users, and through

discussions between the author and his friends and colleagues.

Such outstanding people succeed because they work at a higher level of

abstraction, then see a detailed area, dive into it, investigate it, and return

once again to a higher level of abstraction. (Note that even the awesome project

of putting a man on the moon, three decades ago, succeeded without a predefined

methodology because intellectual control was never lost. The project also

included development of mission-critical software!) Institutions supporting such

analysts and designers also have an important role to play, as one may note from

Stroustrup’s comment regarding his employer: "AT&T Bell Laboratories

made a major contribution to this by allowing me to share drafts of revised

versions of the C++ reference manual with such implementers and users. Because

many of these people work for companies that could be seen as competing with

AT&T, the significance of this contribution should not be underestimated. A

less enlightened company could have caused major problems of language

fragmentation simply by doing nothing. As it happened, about a 100 individuals

from a couple of dozen organizations read and commented on what became the

current reference manual and the base document for the ANSI C++ standardization

effort."

The central activity in software development is the decomposition of the

problem into sub-problems, managing each sub-problem through detailed analysis

and design, and finally integrating their respective solutions into a software

product. To retain intellectual control, one naturally follows stepwise

refinement. This is an iterative process in which analysis, design, prototyping

are the important steps, juxtaposed according to the creative demands of the

situation. It is not a mechanical process. A good understanding of the overall

program and data structure is required before commencing any design.

Analysis



Analysis is the study of a problem prior to taking some action. In software

development, it presupposes the existence of a requirements document which

formalizes the needs of the client, and establishes a list of mandates.

Unfortunately, the framing of requirements produces unusual difficulties.

Clients are rarely domain experts. Often, they will need the vendor’s help in

putting together the requirements of the desired system in sufficient detail.

For a vendor, educating and advising the client about requirements that cannot

be met at all or cannot be met within the proposed budget and schedule

constraints, is tricky and open to a variety of (mis) interpretation, specially

under the pressure of changing and emerging technologies, government

regulations, competitors, etc. Just getting the client to freeze the

requirements is a major task because it puts an unusual premium on

person-to-person communication and the persuasive powers of the vendor in order

to bring about a mutual agreement on the requirements. Since the client may feel

vulnerable in the face of the vendor’s (obvious) superiority in technical

knowledge, mutual trust is critical for him/her to assess risks.

Success in analyzing the requirements will depend upon the ability to

understand and model the problem domain into an abstract model, which can be

converted into a specification of measurable requirements.

Design



Software architecture is conceived in the mind of the designer. It begins to

manifest itself gradually on a physical medium ¾ paper, white boards, video

screens, etc. ¾ through a system of notations. It must be well defined,

expressive, simple, and unambiguous. Commenting on the importance of notation,

the famous mathematician Whitehead had said, "By relieving the brain of all

unnecessary work, a good notation sets it free to concentrate on more advanced

problems." More recently, the Defense Science Board in the U S had this to

say, "The part of software development that will not go away is the

crafting of conceptual structures; the part that can go away is the labor of

expressing them."

Good notations eliminate much of the tedium of checking the consistency and

correctness of a design because such actions become amenable to automation.

Design is the realm of domain experts. Therefore prudence requires that they

work with the deeply familiar terminologies and notations of their domain not

only to maintain intellectual control over their work but also to make full use

of their intuitive and creative powers developed over many years of experience

and over many failures! Unfortunately, the practice in the industry is

otherwise. A software engineer is asked to become a ‘Jack of the domain trade’

and design using the half-baked terminologies and notations of software

engineering! Developing domain expertise takes years before it can be used

confidently, a timeframe generally not available to a software developer, so the

practice is truly illogical. Logically, domain experts should become software

developers and not vice-versa. Domain experts can easily learn software

development skills because they already understand the value of abstraction,

inheritance, encapsulation, etc.

Design provides the blueprint for the pathways and control of information

flow, and the structures in which information must be represented. Equally

importantly, it must provide safe means of handling error conditions and of

recovery from them. Design implies making clever (and pleasing) choices from the

many possible ways of reaching a goal in the given circumstances. There is

nothing like a best design. Design, above all, is a highly creative activity.

Thus there are no methodologies, which will guarantee success, but there are

people who can help achieve it.

Prototyping enables us to  n Test the design and discover missing requirements n Discover early where the hard parts of the system will be n Deliver functionality as early as possible n Gain insight n Define and develop low-level reusable elements n Deal with coding complexity through iterative refinements n Experiment with human-computer interfaces n Het meaningful feedback from the customer as the software develops n Handle the inevitable change requests from a wizened customer

The design phase must conclude with an annotated list of function templates

and data types, which can be used in building software prototypes from the

design. This cannot be overemphasized since the designers are best placed to

detect and suggest patterns and structures that are reusable with firm

conceptual roots. Code reusability will follow almost as a consequence.

Prototype



Prototyping is a powerful means of keeping complexity under control.

Prototypes can be disposable, incremental, or a combination of the two.

Prototypes are the analog of draft versions of a manuscript. In essence, we

begin with things we know how to do to enable us to develop a perspective of how

to do it better, and, more importantly, how we might tackle the remaining tasks.

We may do this by producing a draft version of the software, gain experience,

and discard it to produce a better draft, repeating the process till we are

satisfied with a version. Alternatively, we could continue with the same draft

version, incrementally modifying it as we go along till the software is

complete. In practice, one tends to use a mixture of these two basic tactics.

Prototyping is a means of exploring our way towards a goal when our

understanding of the problem is subjective and incomplete. Even though a

prototype may be disposable, it can often be used as a valuable test bed for

subsequent development activities related to the next prototype or a version of

the product, and test various hypotheses regarding the product and its

construction process.

Back to the beginning



The software crisis is man-made. In every established profession, the means

of avoiding analysis, design, and prototyping (although not necessarily known by

these names) are intuitively understood and practiced with diligence. Software

engineers have heard about these means, talk about them, but fall short of

practicing them with due care and diligence. This is because there is no

commonly (and intuitively) accepted understanding of the abstraction levels

involved in dealing with them and no universally agreed upon symbolic system

(much less, a semantically rich one) for seamlessly transitioning from one to

another. Compounding the crisis is the inept management of software projects.

Why software engineers have stayed aloof from mathematics appears inexplicable

and calls for a major sociological study. The remedy for the software crisis is

obvious to those who understand the power of mathematics and the power of

semantically rich symbolic systems. Software engineering needs mathematicians to

turn it around. Note that the software crisis does not exist in domains where

the domain experts themselves have become their own software developers because

they are able to put off the hindrances caused by programming languages till the

moment of coding.

In its absence, even mathematicians have to go through mental and coding

gymnastics to solve a problem on a computer. This is a sheer waste of effort

since their native symbolic system is so powerful that they can move seamlessly

from problem statement to problem solution, unlike available programming

languages which handle only implementation but not requirements, analysis, or

design aspects of software development. What software engineering desperately

needs is a programming language that will naturally induce people trained in

mathematical modeling to do their own programming by using their own symbolic

systems. That is the real key to software productivity. To bring this about,

programming language compiler designers will have to be far more bold than they

have been so far.

The author is the R&D head of the exports group of IBM Global Services

India, Bangalore



mail@dqindia.com