The value and limitations of Randomised Control Trials in economics

Alok Ray | Updated on October 29, 2019

This year’s Nobel laureates in economics have no doubt done pioneering work, but there are pitfalls in mindlessly applying the results of RCTs

Now that the initial euphoria over the award of Nobel in Economics to Banerjee-Duflo-Kremer trio has subsided a bit, we may look at their contributions in a more objective manner.

Instead of looking at global poverty as a single huge problem with a grand solution, they have broken it down to a large number of small manageable problems where incremental progress can be made by judicious use of instruments at the grassroots level.

They believe that in order to make improvements in the lives of the poor, one first needs to understand how and why the poor people behave and make decisions in certain ways. For example, poor families have been found to spend on television, even when they do not have enough food, which makes sense as TV provides the only avenue of entertainment and an window to the outside world in their cooped up small living space.

The methodology

To illustrate their method of research, consider the following. Everyone advocates more spending on school education as it helps poor children acquire skills and rise above the generational poverty trap.

They show, by using data from field experiments (technically called Randomised Control Trial or RCT) which compares the performance of two otherwise identical groups — one with and the other without the intervention — that the quality of education of the poor kids (something more crucial than mere attendance in school for enhancing the earning potential of students in later life) can be improved much more by tutoring of laggard students (many of them come from first-generation learner families) than by providing mid-day meals, textbooks or school uniforms.

It also follows from their studies that the Right to Education with no pass-fail till Class VIII has improved school attendance but has worsened the learning achievements of the average poor students and has led to large scale drop-outs after Class VIII. They, along with local teams of researchers, have undertaken similar studies involving alternative policy interventions in the fields of education, health care, sanitation, micro credit, etc., in a large number of developing countries.

The shortcomings

Though RCT has been in use in medical drug trials for a long time, extending its use to development economics is a novel idea. But one should also be aware of its limitations.

Drugs tested by RCT and approved by regulators have been later found to produce effects not observed during the study period. This is because it is not exactly possible to keep the two groups identical in all other respects. Also, results of a small sample study over a few years may not reveal all the effects that a much larger sample over a longer period can capture. Cost and time constraints often prevent undertaking bigger sample study over a sufficiently long period.

Similar problems may also occur in economic studies. First, the RCT finding in education, health or sanitation interventions in a few villages in Bengal may not be applicable to villages in another State, say Bihar or Arunachal. Local conditions vary a lot. Further, the conclusion of an RCT may fail to scale up as the population size increases. Take again the earlier example of the impact of tutoring of lagging students. The finding of a controlled experiment in a specific village would be difficult to replicate if it is extended to an entire state or country.

Apart from the differences in populations of different areas within a State or a country, it may simply be impossible to find enough number of dedicated and competent tutors (unlike the few qualified or better paid ones used in a controlled and closely monitored experiment in a village), as non-motivation and absenteeism of teachers in village schools is a known problem (highlighted by the same authors in other studies).

The researchers themselves are usually aware of these limitations and may well spell out the underlying assumptions and possible problems in scaling up the experiment in their research papers. But policymakers, without appreciating the limitations and the nuances, may simply take out the conclusion and apply it to the State or national level. In other words, in the process, the objection of the authors to a one-size-fits-all approach may simply be disregarded.

The purpose of my cautionary tale is not to take any credit away from the pioneering work done by the three Nobel laureates but only to point out the possible pitfalls while mindlessly applying the results of RCTs.

Media and politics

Finally, the role of media and politicians. In our country, may be because of the rare event of winning Nobel (unlike in the US or Europe), Nobel laureates are elevated to cult status. They are regarded as repository of unique wisdom even in areas beyond their area of expertise or research. For example, the statement by the Nobel laureate that the economy is going through difficult times gets the headline, though the same observation has already been made by many other researchers.

The politicians, understandably, swear by the names of Nobel laureates in support of their pet projects, without mentioning the exact role of the scholars or the caution expressed by them.

The case of NYAY or a Minimum Income Guarantee (MIG) scheme for the poor, advanced by the Congress party, immediately comes to mind. The names of Abhijit Banerjee (and Raghuram Rajan) have been repeatedly mentioned as brains behind the idea. However, Banerjee has made it clear in interviews that though he is broadly in favour of some minimum social safety net in the form of MIG and provided some relevant data, he did not work out or suggest a specific figure of ₹6,000 per month as proposed in NYAY.

His preference is for a much smaller sum as a starter and then increase the amount over time as the fiscal constraint relaxes and other specific subsidies get gradually subsumed under the scheme. Banerjee has also recognised in his writings that UBI (Universal Basic Income) on a national scale has not been tried or evaluated anywhere in the world so far.

The writer is a former professor of economics at IIM Calcutta.

Published on October 28, 2019

Follow us on Telegram, Facebook, Twitter, Instagram, YouTube and Linkedin. You can also download our Android App or IOS App.

This article is closed for comments.
Please Email the Editor

You May Also Like