Exit polls 2004: What went wrong and why -- Part I

Part-1

B.S. Chandrasekhar

The 1999 Lok Sabha elections were described as the first television elections and the just concluded elections could be described as the first pollsters¿ elections. This time opinion and exit polls set the agenda to the media or the media allowed itself to be led by psephologists- some real and some pseudo. Almost every alternate evening television channels put out long-winding programmes giving the results of the opinion and exit polls they had commissioned and the newspapers next morning faithfully carried these results on their front pages often with screaming headlines.  

When polling began on April 20^th the voters had already had the ¿benefit¿ of twenty pre-poll surveys. Those who voted on the subsequent days had the additional ¿intelligence¿ from five exit polls each conducted in four phases. Channels built up larger than life images of their pollsters and presenters of poll programmes and leaders and spokespersons of political parties gladly allowed themselves to be bullied by these men.   

Within three hours of counting of the votes the bubble burst as each one of the polls was proved wrong and interestingly all of them erred in the same direction. Experienced psephologists, highly rated statisticians, media-savvy social scientists, renowned market research agencies, large samples and big budgets but not one poll was able to foresee the emergence of Congress; not one survey sensed the anti-incumbency wave against the central government. This is all the more surprising as this time the index of opposition unity was most favourable for election forecasting. Some of the pollsters can take some comfort that they were percipient in some areas, they noticed some under currents, which would have been missed by political analysts. But still all of them completely missed the big picture.  Some won some battles but all of them lost the war very comprehensively.  

Scope

This article in two parts analyses what different pollsters predicted and how right or wrong their predictions turned out to be and also tries to locate the areas of strength and weakness of research agencies.  The second part of this article tries to identify the various reasons for the failure of these predictions. This is an overall assessment of the polls in general and not of any single poll and only to illustrate some points forecasts of different channels have been quoted. However NDTV estimates have been used more extensively for the reason that these estimates are easily accessible in Bangalore through the local edition of The New Indian Express.  

This website had earlier carried an article of the same author titled "the Theory and Practice of Election Forecasting" and there are references to that article.

Five Exit Polls

Five television channels had sponsored exit polls, some in collaboration with popular Indian English and Hindi newspapers. For the first time a foreign newspaper was a co-sponsor of one of the exit polls. In the Elections 2004 the main fight was between two alliances one led by BJP and the other by Congress. However in some of the states there were other major players and in all national level opinion and exit polls forecasts are made under three categories- NDA (BJP+), Congress+ and others. One channel has further split the others to Left and others. Exit poll forecasts made by the five channels and the final results are shown in Table-1. In Bihar polling has been postponed in three constituencies but the forecasts have included these seats also. To facilitate proper comparison in the final results these three seats are included under Congress+.

Table-1

Exit Poll Forecasts 

 

 

 

 

 

 

 

 

 

 

 

 

 

As could be seen the exit polls on the average have given 68 more seats to NDA than it got and consequently 36 seats less to Congress and 32 seats less to Others. There is no doubt all the exit polls have been wrong, some less wrong and some others more wrong.  

Criterion for Comparison

 In sample surveys the standard error is the sole measure of efficiency. The concept of standard error has been explained in the earlier article where it has been explained that an estimate of the vote-share from a sample of 10000 will have a maximum error of 0.50 percent. Some of these exit polls claim that their sample size is more than 100 thousand in which case the standard error of vote-share should not have exceeded 0.15 percent.  

Election forecasting is not estimating the vote-shares but projecting the number of seats to the contesting parties. There has to be some criterion to judge how successfully this has been done.  In the following analysis the difference in the number of seats between the forecast and the final result as a percentage of the total number of seats in Lok sabha is taken as the criterion or standard for assessment.  For a house of 543 members a difference of 12 seats means about 2 percent variation. Many agencies have given their estimates of seats in the range of 12 seats, plus or minus one percent of the strength of the house. For an estimate from a state the difference in the number of seats between the forecast and the final result as a percentage to the total number of seats from that state could be considered as the standard for comparison. In Uttar Pradesh a difference of 8 seats means 10% variation.  

Performance of polls at National level  

At the national level the NDA alliance got 68 or13 percent less seats than the exit polls on the average had projected. The Congress alliance got 7 percent more seats than projected and others 6 percent more than projected. In most of the states the fight was between the Congress alliance and NDA and others had their presence in less than 200 constituencies. Left parties have a stronger case to protest against the exit polls than Congress.   

Among the different exit polls NDTV has been least wrong about both NDA (+9%) and Congress (-4%) but Zee TV has been least wrong for others (-5%). On the other side Sahara and Star News have been most liberal about NDA (+15%), Sahara has also given the least number of seats to Congress+ (-8%) and Star News has given the least number of seats to others (-8%). As said earlier all the exit polls have been wrong with their estimates well outside reasonable tolerance limits and no channel could take solace that it was less wrong than the others.  

Consistent Bias towards NDA  

In the earlier elections the forecasters made big mistakes in a few states but still got the total picture right. The universal law of errors used to operate and the error in one state in one direction used to cancel out the error in the opposite direction in another state; if Congress or BJP did not rise to the expectations in one state it excelled in other state. This time in all the exit polls in all the states where forecasts have gone wrong the errors have been unidirectional with NDA being projected to get more seats than it got and Congress+ or others less. It is very rare to see this type of bias in a single direction in large sample surveys.   

We have seen above that compared to other channels NDTV has been fairer to Congress+, it has given only 22 less seats than the alliance finally got. But this is misleading to an extent. In Kerala NDTV had projected 13 seats to Congress where it failed to open its account. This means that Congress where it was fighting NDA has been given a total of -35 seats and this has been made good in Kerala- an interesting way of one error canceling another.  

It is not being suggested that either the channels or the research agencies were biased; one possible source of bias has been identified in the second part.  

State Level Estimates 

The exit poll forecasts at the national level are the aggregates of the estimates for 20 states and as such it is necessary to see what the forecasts for different states were. The five different surveys have given different estimates for the contestants in different states.  In the following detailed analysis mainly forecasts of NDTV have been taken for the reason already stated. Forecasts other polls are referred where relevant.

Multi-cornered Contests 

In these elections there was straight contest between two alliances in most of the States. Uttar Pradesh has always been an exception to this pattern. In this state, which sends 80 members to the Lok Sabha, four major parties fight the elections. In a way the electoral fortunes of the parties are made or unmade in UP and this state is a nightmare to the pollsters. This time also the psephologists have failed in this state and have failed very badly. NDTV exit polls had forecast 28 seats to BJP but the party got only 11 (-21 %) seats in contrast to SP, which got 12 (+15%) more seats than what was predicted. The other exit polls had predicted even more seats to BJP- Aajtak 30 and Star news 36. Forecasting in UP is difficult but at the same time correct forecasting in UP is crucial for national forecasting. Psephologists have to find a way out, as otherwise election forecasting in India will become a futile exercise.  

In West Bengal the Left Front has been dominating all elections for nearly three decades. Traditionally Congress was getting some of the urban seats but in the last three elections WBTC an alliance partner of NDA has entered the fray resulting in triangular contests in some constituencies. NDTV exit poll had forecast 9 seats to NDA and 4 seats to Congress but in the final counting NDA got just one seat and Congress 6 and West Bengal became another state where forecasts went wrong. Star news forecasts were even worse- 10 for NDA and 2 for Congress whereas Aajtak faired better in West Bengal- it had forecast 6 seats to NDA and 8 to Congress. The details of NDTV exit poll forecasts and final results of UP and West Bengal are shown in Table-2.

Table-2

Multi-cornered Contests

Exit Poll Forecasts (NDTV) and Final Results

 

-->