When the Heart Stops Beating...

The Availability Continuum

• Base-availability systems are ready for immediate use, but will experience
  
  both planned and unplanned outages.
• High-availability systems include technologies that sharply reduce the
  
  number and duration of unplanned outages. Planned outages still occur, but
  
  the servers include facilities that reduce their impact.
• Continuous-operations environments use special technologies to
  
  ensure that there are no planned outages for upgrades, backups, or other
  
  maintenance activities. Frequently, companies use high-availability servers
  
  in these environments to reduce unplanned outages.
• Continuous-availability environments go a step further to ensure
  
  that there are no planned or unplanned outages. To achieve this level of
  
  availability, companies must use dual servers or clusters of redundant
  
  servers in which one server automatically takes over if another server goes
  
  down.
• Disaster tolerance environments require remote systems to take over
  
  in the event of a site outage. The distance between systems is very
  
  important in order to ensure that no single catastrophic event affects both
  
  sites. However, the price for distance is loss of performance due to the
  
  latent period required for the signal to travel the distance.

Source: IBM

When the going gets tough, the tough get going.

While

this age-old adage may apply to the human heart and mind, the response mechanism

in man-made computer systems is not all that active and strong. Unlike the human

heart, which is 99.999 % available till the first heart attack when there is an

‘outage’ for a few seconds, computer systems are not that infallible.

Systems that run mission-critical applications need to be

designed in such a way that they never fail. The domain of computi ng

science that deals with the span of solutions that make systems fail-proof is

called as High Availability (HA) computing. Ranging from phenomenally expensive

fault tolerant machines and high-availability clustering to the simple ‘ready-to-plug’

inventory of spares, these solutions offer system availability. System

availability is measured in terms of the percent of the time the system is

available- ranging from 99.9 % to 99.999%.

Pumping blood for the body



But the story goes beyond mere system availability. That is, the challenge

of putting up a highly available information system does not only lie in the

procurement of a sophisticated HA server but in extending HA performance to the

rest of the technology and support components like OS, middleware, application

software, network, and power supplies to say the least. Says ICICI Infotech

joint president Manoj Kunkalienkar, "The procurement and design of an HA

system is a heavy SI (systems integration) activity". ICICI Infotech has

designed and runs the complete technology infrastructure for the entire ICICI

Group, comprising scores of servers running a variety of mission-critical

applications made available across the country and run from two data-centers in

Mumbai. The system also has disaster-recovery capabilities.

It is important to understand that not everything has to be

designed for 99.99 % availability. Says Manoj, "We define HA from the

perspective of the application being available to the user". The question

business owners need to ask is — how much of downtime can I afford? At ICICI,

each of the business owners sat down to discuss the downtime affordable in their

respective businesses. The availability features of various applications were

then designed according to the uptime required for the respective businesses.

Critical applications like stock exchanges and reservation systems, where public

transactions are involved should have the highest levels of availability.

Hardware today is already highly reliable. For a single CPU

system, hardware availability is already at 3 nines or better, and for

CompactPCI (a standard for form-factor of boards) systems with hot swap

capability, hardware availability of four nines is easily achievable at

reasonable cost. Adds Manoj K, "Most hardware today is extremely reliable

and most of the servers don’t fail". To reach 5 nines or more of hardware

availability generally requires redundant hardware with automatic ‘failover’

capability. Many CompactPCI systems which support hot swap already have

redundant power supplies and fans, and can be configured with redundant I/O and

even redundant system slot CPUs.

Operation patch-up



The most stringent requirements for availability demand continuous service

with no loss of records of current transactions and history. A trading

application at a stock exchange like NSE is indeed mission critical and such

systems need machines that can run despite faults occurring. Such systems from

vendors like Compaq/ Tandem and Stratus are known as fault-tolerant systems.

Such systems guarantee continuous availability, both during equipment failure

and during subsequent return to service of repaired equipment. So-called

fault-tolerant computers that only add redundant power supplies and harden

storage subsystems with various disk array (RAID- Redundant Array of Inexpensive

Disks) configurations are neither fault-tolerant nor high-availability

computers. Such high levels of continuous availability do not come without a

significant price penalty. Fault tolerant computers are not the norm for anyone

who wishes continuous availability because they can turn out to be phenomenally

expensive.

Hardware capabilities of fault managed systems can be

generally categorized into three sets: redundancy to allow continued processing

after failure, highly reliable and redundant communication between components,

and management of the components including fault detection, diagnosis,

isolation, recovery and repair. The OS capabilities include functions to

isolate, prevent the propagation of, or mask the impact of hardware and software

faults. These functions help prevent faulty applications or faulty hardware from

pulling the entire system down.

Cluster to cut costs



An option that is less expensive than a fault tolerant server is that of

building a HA cluster. High availability is achieved through redundancy of

components. This strategy, sometimes referred to as n+1 sparing, requires that

one spare be provisioned for each class of spared components. High-availability

(HA) computing, as commonly practiced, utilizes the redundant resources of

clustered (two or more) processors. Such solutions address redundancy for all

components of a system, processors, main and secondary memory, network

interfaces, and power/cooling. While hardware redundancy may be effectively

addressed by clustered (redundant) hardware resources, the class of errors

detected are less comprehensive and the time required to recover from errors is

much longer than that of true FT machines. Still, fault recovery in tens of

seconds, from most common equipment failures, can be achieved at less than half

the cost of traditional FT computer systems.

A common thread in all HA planning is redundancy, and the

elimination of single points of failure (SPOFs). For instance, the primary

design of Air India’s computer center was done way back in 80s to ensure high

availability. Says MSV Rao, Director- IT, Air India, "We are probably

reaching four nines of availability". AI uses systems with dual redundant

CPUs with each file getting duplexed. But bottlenecks have been found outside

the server or system hardware in the network and communication links. Adds MSV

Rao, "We follow the classical definition of having no single point of

failure and on that count I find the condition of network links to be somewhat

less reliable from a HA point of view". AI runs on four Unisys mainframes

and a collection of SunSparc, Sun E3000, IBM RS 6000s, and SGI servers.

Mainframes typically have some HA capabilities or at least some concepts that

should be borrowed from mainframes onto open systems. Says Manoj of ICICI

Infotech, "Mainframes had a nice concept–Disks could be partitioned and

could be made independently bootable". Redundancies have been built into

the Air India systems at various levels- alternate network paths through

different modes and service providers, power coming in from two different

substations and the like. Thoughtful planning of system configurations with an

eye on eliminating SPOFs can pay off in terms of achieved availability.

If the pump fails…



In planning for HA applications, you must take into account the possibility

of a catastrophe. If availability is required in these circumstances, one must

have arrangements with data centers in other parts of the country to provide

temporary equipment and services, and your backup and roll-forward plans must be

made accordingly. The key is to have a redundant facility that is far away and

is not susceptible to loss of service because of the same condition but close

enough for key people to get there quickly if necessary. NSE has shifted its

mirror site to Chennai from Pune in order to ensure that the two are in

different seismic zones.

But low-budget HA options are available too. Advises NSE IT

CEO Satish Naralkar, "Have an inventory of standby spare parts. This makes

it much simpler administratively. Many of the components are hot swappable, that

is, you can replace faulty hardware on the fly". Also note that having a

100 % inventory at all times also may work out costlier than having a full

system. The key is to know the failure history from experience and decide which

ones are to be kept. One needs to plan for HA only as much as the business

demands. Sometimes, a combination of approaches may be the solution. NSE could

be termed as a good case of having a line-up of availability solutions depending

on business needs. A good combination could include a Stratus FT machine for its

trading systems, clustered HA systems for the back-end, less critical

applications on cold standby, and critical spares for some applications. HA,

however, is not all about shelling money for redundant systems and recovery

centers. Cautions Naralkar, "The key is to allow no SPOF at all points of

failure. It is like carrying a spare tire that too is flat!" Incidents such

as 9.11 have created interest in HA. Still, for a large number of companies, HA

still remains on the wish list. As Naralkar points out, "In such a

situation people fall back on the belief that such an eventuality may never hit

them".

Easwardas Satyan in

Mumbai

The ‘Availability’ Jargon Buster

2N: A redundant system in which there is a hot spare for every

(critical) component.

3N: A redundant system in which there are three of every (critical)

component, and a voting mechanism to prevent a failure from causing errors (see

TMR).

Cluster: A collection of two or more systems which do useful work, but

in which the remaining system(s) can take over the load in case of the failure

of one of the systems.

Cold spare: A spare component that is available to replace a failed

component, but is not installed.

Cold restart: Restarting a device from power off or with no state

information.

Fault tolerance: The system attribute of being able to operate

correctly while faults are occurring.

Heartbeating: Sending a signal from one component to the other to show

that the sending unit is still functioning correctly.

Hot restart: restarting a component while the system is still

operational or partially operational.

Hot spare: A spare component that is constantly updated with the state

of the component that it will take over in case of a failure.

Hot swap: A term for removing a failed piece of hardware and replacing

it with a cold spare in a ‘hot’ (operational) system (without bringing it

down).

MTTF: Meant time to failure, which is the average time between one

failure and the next one as measured over a large number of failures.

MTTR: Mean time to repair which is the average time it takes to repair

a component or system as measured against a large number of repairs.

N+M redundancy: A system that has N components working in normal

operation and has M components in standby. This is frequently reduced to N+1,

where there is one standby component for every operational N operational

component.

RAID: Redundant Array of Inexpensive Disks. RAID may be used to

provide high availability, larger virtual disks, or both. RAID technology is

split into levels- Linear,0, 1, 2,3, 4, 5 which define various characteristics.

Reboot: To restart a computer. Cold reboot occurs when the system is

powered off to reboot. Warm reboot occurs when the hardware is left running, but

the OS is restarted.

Rollback: The process of repairing the damage to the state of a system

caused by hardware or software failure.

Rollforward: The process of bringing the state of a system or software

component to a level at which it can be placed into service.

SPOF: Single point of failure. If a component is a SPOF, it is

critical that it be as reliable as possible, since the reliability of the system

can be no better than that of the SPOF component.

Warm spare: A spare component which is powered up and ready to be

configured to take over in case of a failure.