Why opinion polls go wrong
Arvind Kalia and Shipra Mathur
Objective
This paper explores in depth the reasons because of which an opinion poll or any important marketing research project goes wrong. It is written against the backdrop of the last assembly elections held in Rajasthan, Madhya Pradesh, Chattisgarh and Delhi in the year 2003. Before the elections all the credible, leading magazines of India brought out their own version of opinion polls only to see that their scientifically estimated findings went wrong once the elections were over. This article found out that precautions to follow the basic tenets of any research project, were not adequately taken care of. Had it been the case their accuracy would have been beyond doubt. The article does not purport to declare or conclude that there were wilful intentions to commit mistakes on the part of the research companies or sponsors. This article prescribes right ways to avoid human errors in such type of research projects.
Before the assembly elections many opinion polls were conducted by different organisations. This article probes the accuracy of the following opinion polls conducted by "The India-Today-Aaj Tak-ORG-MARG" and "The Outlook-Zee News-AC Nielsen".
Details of the Survey
| India Today | Outlook | |
|
Duration |
Not specified |
Oct., 30th - Nov. 9th, 2003 Small sample on Nov. 18th in Chattisgarh to assess the impact of Dilip Singh Judeo bribery scandal in the state. |
|
Sample size in four states |
17,413 in 115 assembly constituencies |
21, 936 in 123 assembly constituencies |
|
Sampling Method |
Random Sampling at the parliamentary constituency level |
Non-Random judgmental sampling |
|
Sample Representation |
Adequate representation of urban and rural males and females and all age groups |
Adequate geographical spread; adequate spread among urban and rural areas; in urban area male-female ratio was 60 : 40; and only males in the rural areas. |
|
Research approach |
Personal interview |
Personal interview |
|
Research instruments |
Based on personal interviews No information about the nature of the instrument whether it was structured or non-structured schedule, however, looking at the reports, it can be guessed that structured schedule might have been used. |
Structured schedule |
|
Research organisations |
India-Today-Aaj Tak-ORG MARG |
Outlook-Zee News-AC Nielson |
|
Margin of error |
1.7 - 2 per cent |
3 per cent |
|
Whether Predictive model used |
No |
Yes |
Survey Results vis-a-vis Actual Results
India Today-Aaj Tak-ORG MARG - Survey
| State Party
Details |
seats-200 Rajasthan Assembly |
Seats-230M. P. Assembly |
Seats-90Chattisgarh Assembly |
Seats-70Delhi Assembly | ||||||||
|
Predi- |
Actual Seats |
% differ- |
Predi- |
Actual Seats |
% differ- |
Predic- |
Actual Seats |
% differ- |
Predi- |
Actual Seats |
% differ- | |
Cong- |
110-120 |
56 |
49.1 |
75-95 |
38 |
49.34 |
38-48 |
36 |
5.27 |
45-55 |
47 |
-4.44 |
|
BJP |
73-83 |
120 |
-64.38 |
115-135 |
173 |
-50.43 |
38-48 |
50 |
-31.57 |
10-20 |
20 |
-100 |
|
Others |
- |
24 |
- |
- |
19 |
- |
- |
03 |
- |
- |
03 |
- |
Outlook-Zee News-A.C. Neilson Survey | ||||||||||||
|
Cong- |
113 |
56 |
50.45 |
58 |
38 |
34.49 |
43 |
36 |
16.28 |
56 |
47 |
16.08 |
|
BJP |
67 |
120 |
-79 |
147 |
173 |
-17.68 |
43 |
50 |
-16.27 |
11 |
20 |
-81.81 |
|
Others |
20 |
24 |
-20 |
25 |
19 |
24 |
04 |
03 |
25 |
03 |
03 |
- |
Note: In India Today survey the error has been computed taking lower range into account.
If looked at the above table the following observations can be made -
-
The research methodology is scientifically designed.
-
Sample size is adequate.
-
Limitations in the sample representativeness in terms of gender, caste and communities.
-
Percentage variation in predicted and actual results is as high as 100% which is amazing.
Reasons for this Amazing Variation ?
Any scientifically designed survey can go wrong drastically if errors occurring in interpersonal communication and observations are not taken care of. Interpersonal communication and observation are activities in which each of us is highly experienced, but, it appears almost paradoxical that the asking of seemingly straight forward question receiving straight forward answers, the observing of people`s behaviour in a formalized information seeking context, could be attended by serious problems. Any formal effort to obtain information from a sample of respondents is subject to three major sources of errors:
-
Sampling Error
-
Non-Response Error
-
Response Error
Although the article will deal with all these sources of errors, the third i.e., the response error shall be dealt with at length.
1. Sampling Error
Sampling or experimental error arises in research projects because not everyone in the population of interest is included. In any sampling situation the usual result is that the sample selected is not completely representative with respect to the characteristics of the population from which it is drawn, for example in both the surveys not all the population characteristics have been taken care of. Important characteristics like communities, class, caste were completely ignored while selecting sample. Similarly, occupational categories such as businessman, government employee were also omitted.
In Rajasthan poll the government employee factor is said to have played a pivotal role in deciding destiny of the political parties. We suggest in future surveys these dominant groups were given appropriate consideration while carving out the sampling population. It is beyond our comprehension to understand the logic of these professional research organisations to omit these dominant groups. There should be proper justification for the ignorance of these groups, at the time when political scenario in Rajasthan was replete with caste congregations demanding vociferously reservations in all government and non-government openings. Taking the example of occupational dominant groups, government employees in Rajasthan, it is said that they went en mass against the then existing government on the plea that the previous government deducted their salaries when they went on strike a few years back. The government employee is the largest occupational group in Rajasthan, spreading its wing in all the 37,000 villages in smaller towns and villages. They act as opinion leaders and silently influenced word of mouth publicity against the regime. By having proper sampling selection on these undercurrents authenticity of such polls can be enhanced manifold.
2. Non-Response Error
Non-response error occurs when an individual is included in a sample to be taken but because of any of the myriad possible reasons, is not reached. In most opinion polls it is a source of potentially sizeable error. The seriousness of non-response error is magnified by the fact that the direction of the error is often unknown, and while the maximum error due to the non-response can be determined, it is difficult to estimate the actual magnitude of error.
For example, the Outlook survey did not include independent female data from rural areas. The magazine has given the reason for not including females in the sample-"In the urban areas, interviews conducted with men and women in the ratio of 60:40, but in rural areas, only males were interviewed. The latter practiced has evolved out of field experience in the rural north where eliciting reliable independent ¿female¿ data been found to be difficult."
In the changing time it is improper on the part of research organisations to assume that females from rural north do not give independent reliable opinion for the serious issue like election. Because of the outreach of the mass media-TV, Radio and Press and the major boost in literacy, these assumption do not hold good in the present context. The research organisations should have developed adequate filters to select independent minded women from these areas. The female opinion leaders are abound in rural areas if we look at the facts regarding number of female MLAs, Sarpanchs, Parshads, teachers, social activists etc.
It is true that the female respondents in rural areas are by and large hesitant in talking to unknown strange male investigators of the research organisations. If these research organisations hire female investigators and send them in rural areas, they can extract independent reliable data.
3. Response Error
This error occurs in the collection of information from an inDIVidual if the reported value differs from the actual value of the variable concerned. When we speak of this error, it should be understood that it is inclusive of errors arising through communication, observation or both.
What are the sources of response error? In answering this question it will be helpful to consider the stages involved in providing the information. The information must first be formulated, i.e. it must be assimilated and made accessible for transmission. Once this has been accomplished it must be transmitted. Errors can arise in either stage. We shall use the term "inaccuracy" to denote the errors, arising in the formulation stage. The term "ambiguity" will be understood to mean the errors arising in the transmission stage.
|
3.1 Inaccuracies | |
|
We have agreed that inaccuracy refer to errors that are made in formulation of information. Suppose, the respondent is asked "Do you intend to vote for Congress in the next election?" The answer will be limited to "Yes" or "No" or "uncertain". A brief examination of possible answers and subsequent actions indicate that there are two different kind of inaccuracies with the respondent`s answer "Yes" - but has really no intention of voting Congress party, we should say that there is concurrent inaccuracy in the statement.
What are the sources of inaccuracy?
Inaccurate information may result either from the inability or unwillingness of the respondents to provide the desired information. In those instances where observation is used this statement may also be applied to the observer. The observer may be unable or unwilling to provide the desired information.
We can readily understand the inability of people to provide information because of its being inaccessible. In other words, the information to the question asked is not accessible to the respondents. They are not able to recollect it properly. Most of the questions such as - "How did Gehlot government manage drought in the state?" and "What do you feel about Vasundhara Raje being a Maharani and a Chief Minister candidate?" - require higher degree of information retrieval and assimilation in the mind of a respondent. Hence, he/she might have judged himself incapable/unable to provide information in response to such questions.
The problem of unwillingness of respondents to provide accurate information deals with the motivations of people why they are not willing to formulate accurately the information desired. No fully accepted general theory of motivation has yet emerged from behavioural sciences to which we can turn to in explaining this type of behaviour. However, several reasons are suggested why people may not be willing to make accurate information accessible. There are always some costs (negative utilities) attached to respondents formulating information. We may postulate that, when its possible to do so, the respondent will tend to act in a manner that will reduce these costs. An expert researcher should be able to anticipate these costs and conduct the interview in a manner that avoid the consequences of these costs. Let us analyse these costs one by one .
1.1 Time cost : Perhaps the most common reason for respondent¿s unwillingness to provide accurate information is the result of the time required to make the information available. The respondent may simply be busy and wish to complete the interview as quickly as possible. In this circumstance, it is not unusual for the respondents to decide that abrupt answers are the easiest and quickest way of terminating the interview. Rather than reflecting on or verifying the information provided, he gives hasty, ill considered answers and resists probing if attempted. Hence, it results into complete inaccurate information.
These research projects lack in detail regarding how they overcame this hurdle. Many a time, research organisations offer some token gifts to respondents to generate interest in the survey. May be because of the budget constraints , this method of gaining attention was not resorted to by the research companies. The surveys of this magnitude and importance should have provision of some incentive to the respondents.
1.2 Conflict with opinions imputed to the investigator
A complex sum of inaccuracy stems from the respondent¿s appraisal of the investigators and their opinions and expectations that are imputed to them. Although, much remains to be learned about the nature of cues, from which respondents draw inferences about investigators` opinions there is sufficient evidence to conclude that such inferences are drawn and that they influence responses. In the opinion poll context it seems to influence the opinion of the respondents. There is sufficient reason to believe in the possibility of this inaccuracy in the survey. India Today and the other media have already declared the ranking of different chief ministers. All the ruling state`s CMs, secured excellent rankings for example Ashok Gehlot got the ranking of number one in the entire country. Being in the mind of the investigators, there is possibility that they communicated to respondents that the Congress is going to form government in the next elections. This inaccuracy can be reduced by giving special training to the investigators that deliberate or non-deliberate behavioural gestures doesn`t give any clue to the respondent about what their opinions are.
1.3 Unwillingness of the Investigator
So far we have considered only the unwillingness of the respondent to provide accurate information. The investigator`s side of the coin should also be considered. The investigator may be willing to obtain accurate information, even if the respondent is willing to provide it.
The most common form of this problem is interviewer`s cheating. The ways in which the investigator may obtain accurate information deliberately and his motives for doing so are limited only by his ingenuity and personality. It may be for example that an interviewer finds that a particular question is embarrassing . As a result he may decide to supply his own answer to make an estimate or inference of what the respondent`s answer should be. At the other extremes reports of interviews are occasionally submitted without the interviewer having taken trouble to contact respondents. A compromise between these extremes is the interviewing of friends but listing the names of people that were supposed to be interviewed. Such incidents in Indian Research industry seems to be very high as no research agency maintains its permanent field staff for the purpose of cost efficiency. They bank on fresh and untrained people at a nominal cost. Most of the field staff supplied by these research agencies consist of young unemployed people who want to make some money for time being by putting in their least efforts. Since they don`t have any long term objectives to remain in the research field, they tend to finish the task of interview as fast as they can whether it is done qualitatively or not. For opinion poll survey these research agencies shouldn`t rely solely on these local manpower suppliers. Rather, they should recruit qualitative staff by paying them higher money, higher compensations so that such type of inaccuracies can be eliminated.
Interviewer cheating can be kept to a low revel incidence but not eliminated completely. Careful selection, training and supervision of interviewers would eliminate much of the problem. In addition, control procedures can and should be established that would reduce it even more.
Other control procedure include the analysis of responses obtained by each investigator and the use of cheater questions. In studies when the volume of information obtained makes it worthwhile to use machines for tabulation, analysis of the patterns of responses obtained by each interviewer can be made at very little additional cost .Significant variations from expected norms can then be investigated.
The use of cheater questions in the questionnaire is a less used and widely publicised device. Questions can be devised that will disclose fabricated answer and with a reasonably high probability of answers.
1.4 Ambiguity:
Ambiguity may be defined as the errors made in spoken, written words or behaviour. Ambiguity, therefore, occurs in the transmission of information either through communication or observation. Ambiguity is present in all languages specially in ours. Ambiguity is present in many questions such as "Do you feel better off today than two years ago?"
The word "better off" is highly confusing and need strict definition instead the word "improve" should have been used.
Similarly in another question "How do you rate government`s performance?"- the word ¿performance¿ has to be described in a simplified way to the respondents .In use of such type of heavy words chances are there that neutral or positive answers are given by the respondents. The different methodologies are to be adopted to get the fair idea about what is going on in the state on those parameters. Moreover, the answers to these questions can be obsessed with the theory of recency which can be explained saying that the people remember things of immediate past more distinctively then things of distant past. Question should be designed in such a manner that he recalls both the time horizons. |
Conclusion
The list of mistakes committed by both these surveys is endless. In Outlook survey for example the respondents who preferred Vasundhara Raje as a befitting chief ministreal candidate were asked-"Reasons for preferring Vasundhara Raje". In response majority (20 per cent) of the respondents opted for the "development work". The irony is apparent from this answer itself because Ms. Raje has never donned any political position in the state, leave apart contributing anything to the welfare of the state.
Such paradoxes in the surveys demand urgent remedial measured in future as the whole political career of party person is deeply affected by these predictions, especially when the top media groups highlight them.
Contact:
Dr. Arvind Kalia, MBA, Ph.D , AEP (Kellogg)
Assistant Professor,
R.A. Podar Institute of Management,
University of Rajasthan, Jaipur
e-mail - arvindkalia@yahoo.com
Ms. Shipra Mathur,
Lecturer, International College of Girls,
Department of Journalism, Jaipur
e-mail- shipramathur@icfia.org