Why opinion poll surveys are tricky business

M Ramesh | Updated on April 10, 2019

Questionnaires True opinion?   -  istock

Selection and size of sample are crucial. If these are compromised to cut costs, the predictions can go horribly wrong

Why do opinion polls go so horribly wrong so often? Conducting opinion polls is tricky business — ask any psephologist. It is scientific enough, but a lot of things can — and do — go wrong.

For starters, the devil lies in sample size and method of selection. The sample has to be true to the ‘universe’. A saying in Tamil goes that a morsel from the cooking pot is good enough to judge the entire pot’s contents, but that is because the morsel is a truly representative sample. In surveys, getting a representative sample right is a matter of, well, money.

First the size. Mathematician and psephologist Rajeeva Karandikar holds that the ideal sample size is 50,000 across the country, if the objective is a simple determination of who will win among the NDA, the UPA and others. A smaller sample is inadequate, though a larger one does not add value.

But then, which pollster has the luxury of getting a 50K sample? Costs work out typically to ₹600 for every person interviewed, or ₹3 crore for a 50K sample. Few media houses today are prepared to spend such a sum on an opinion poll — even if they could afford to, the temptation is to take the cheaper offer.

Error creeps in right at that stage. A low budget opinion poll cuts corners in terms of choosing interviewees. Ideally, once the sample size is fixed, selection of interviewees ought to be done by ‘systematic’ (or circular) sampling. In this method, if (say) 20 per cent of the Lok Sabha constituencies is to be randomly identified, then systematic sampling goes like this: take the list of constituencies, pick one randomly from among the first five, then choose every 5th from that number.

For example, if constituency No.3 is picked, then you would choose constituencies numbered 8, 13, 18, and so on.

Then, having chosen the constituencies, you select some polling booths within each constituency following a similar method, and finally select voters within the booths chosen from the voters’ list, and interview them. Caution: if a selected voter is not available, then don’t go to another next door — it skews the sample.

This method is well-known to pollsters, but not everyone of them adopts this, because of time and money constraints. Instead, they yield to the temptation of sending a team to, say, shopping malls or railway stations with an instruction to pick interviewees of varied profiles in terms of age, gender and social status. You hardly get a representative sample.

‘Non-response error’

Even if you get the sample right, there are still other issues. For instance, the ‘non-response error’, or the distortion due to people not wanting to answer a question. Many baulk from disclosing their choice of vote. A cheaper but inaccurate way of getting round this problem is by “attributing” the answer to them, judging by their replies to other questions. Some pollsters do so, and the results are not always good.

CSDS-Lokniti has found another way of overcoming the ‘non-response error’. The interviewees would be given a printed ‘ballot paper’ and asked to tick-mark their choice in private, fold the sheet and drop it in a box.

Generally, post-poll surveys (not exit polls) predict more accurately, but here too there are issues. For instance, interviewees who didn’t vote often end up giving answers, not wanting to admit they did not vote. To check against this, pollsters must check the ink mark, but it is embarrassing to ask the voter for proof that he voted. Some surveys verify this pretending to check the indelibility of the ink. Unless the fake answers are filtered out, you’d end up wayward. But even after all this, things could go horribly wrong. For instance, the CSDS-Karandikar partnership correctly predicted the results of the recent Rajasthan and Madhya Pradesh assembly elections but was horribly wrong — like other analysts — in the case of Chhattisgarh. They still haven’t figured out why.

Pre-poll surveys could go wrong because of the voter’s likes or dislikes of the candidates, which are very difficult to capture in a questionnaire. Surveys are like snapshots, they reveal the situation at that point in time. Things could change drastically in days.

Moreover, when media houses know their methodology is not good and the surveys are very different from the general predictions of all others, they lose faith in their data. In such cases, they tend to play it safe and give out a number which is more in line with the trend, rather than stick to what their data show — widening the prediction-result chasm.

Published on April 10, 2019

Follow us on Telegram, Facebook, Twitter, Instagram, YouTube and Linkedin. You can also download our Android App or IOS App.

This article is closed for comments.
Please Email the Editor

You May Also Like