Number Crunching On A Billion Records

Imagine

you’re a newspaper photographer, and your photo-shop takes three days to

develop your photos. You’d rarely see your pictures published.

Every 10 years sees a mega-snapshot of India. That’s the census, the most

important source of information on the life of India’s billion-strong

population. Partition 1947 was based on census data on religion.

That snapshop takes very long to process. Detailed data from the

second-largest survey in the world, the India 2001 census, is still not

available two years down. The registrar general’s office has been unable to

tabulate, analyze and distribute the data. Demographer Ashish Bose, chairman of

SARH (Society for Applied Research in Humanities) says there’s little support

or funding from the ministries of Home or Health and Family Welfare or the

Planning Commission.

This is not unlike when weather data was collected, but number-crunching it

took three days to get to tomorrow’s prediction–too late. All you could do

then was validate the model against what had really happened.

IT has begun to make an impact. The preliminary census findings were released

with impressive speed in 2001 itself, and the broad data is available at

www.censusindia.net, and on CDs at a nominal cost (but much trouble).

Next step for the Home Ministry, through the census department: the national

ID number pilot project.

I’ve written earlier about the huge duplication of citizen data collection.

By census department, election commission, income-tax office (ITO), customs and

excise, food and civil supplies, the RTO (transport offices)… Each body starts

from scratch, from a blank form, and duplicates some of the same data while

focusing on its own interest area. These databases never talk to each other,

thanks to policy and privacy norms (the Census Act) as well as data structures.

For instance, a simple comparison by the ITO against the list of "foreign

travelers last month" for a mailing to them, is mind-bogglingly complex, if

it ever happens. And forget about the obvious use of data such as Delhi’s top

phone or electricity consumers, for tax return demands.

The National ID is as yet merely an idea; even the pilot project hasn’t

begun. Like the US social security number model, it defines a single, unique

citizen number to be used for all transactions in India. (The ITO’s PAN is now

required for certain transactions, but a tiny fraction of Indians have a PAN).

The project is expected to go live in 2005, and someday, the Election Commission

and others will be able to build upon this database and avoid reinventing the

wheel.

As with the 20 e-gov projects covered in this issue, India’s taking the

first steps to using technology for better governance.

Prasanto K Roy