How data-crunching tech has evolved

Our Bureau | Updated on February 09, 2011


Accuracy with speed. This is a combination that is not easy to achieve in any operation that involves processing of voluminous data collected from 240 million-odd households spread across remote corners of the country.

But with technology, all these seem no longer insurmountable challenges. More so, when you have organisations willing to employ the latest knowhow and adapt it to the conditions on the ground.

In 1961, the Census authorities collected relevant information from each and every household of the country. But given the sheer inability of the unit record machines to process this entire data, only 5 per cent of it was randomly selected and captured for further processing. And, despite this, it still took 8-9 years for the complete results to be out — by which time the findings had ceased to have practical use, except for the economic historian.

In the 1971 Census, key-punching (in place of hand-punching) and use of an IBM-1401 computer with card reader enabled more data (15 per cent) to be processed.

The capture ratio improved further in the subsequent two Censuses, when data entry through key-to-disk machines (1981) and Unix-based dumb terminals (1991) were used alongside HP 1000 CD-Cyber 730 & NEC-1000 computers (1981) and the ‘Medha-930' Main Frame system (1991). But these made no major difference to the time taken.

The real breakthrough came in 2001, when India became the first large country to deploy high-speed duplex scanners for image capturing and extraction of data through automated forms processing software. Dispensing with data entry meant that 100 per cent of the collected information could be captured. Along with accuracy, it also brought down the processing time to within five years.

For the 2011 Census, the same ICR (Intelligence Character Recognition) software, incorporating more advanced features, is proposed to be used. And it is expected to enable release of the entire Census results in 1-2 years, which may be useful enough for non-academic users.

Published on February 08, 2011

Follow us on Telegram, Facebook, Twitter, Instagram, YouTube and Linkedin. You can also download our Android App or IOS App.

This article is closed for comments.
Please Email the Editor