DOCUMENTATION: TRAI consultation paper on TV audience measurement

An extract from the Consultation Paper On Guidelines/Accreditation Mechanism for Television Rating Agencies in India

17th April, 2013

Website: www.trai.gov.in

Methodology for Audience Measurement

4.9 Methodology plays an important role in audience measurement. A robust methodology would ensure the reliability of the ratings provided by the rating agencies whereas a poor methodology would distort the measurement results, adversely affecting the sector as a whole. Therefore, a rating agency should have in place a rating process/methodology that conforms to the conditions / standards / norms prescribed for the ratings process and reflects consistent and internationally accepted rating standards. It should work towards continuous improvement in quality and method of the rating system, to provide accurate, up to date and relevant findings. In this regard, the key issues pertain to selection of the households, panel size & secrecy of panel homes. The same are discussed in the subsequent paragraphs.

Selection of the households

4.10 Sample households are generally selected through a large-scale establishment survey that defines the population to be represented and its characteristics. The establishment survey is conducted in such a manner which remains representative of all television households across the country. It encompasses the full range of demographic and TV reception variations, amongst other variables, that are found across the country and in different regions. The selection process of the sample households for the measurement purposes should be random and every household in a particular area should have an equal chance of being chosen. The establishment survey results in selection of household samples which is around 10 times of the desired panel homes. This approach exists in some countries like Australia & UK. This survey is required to be carried out continuously in order to account for changing demographic profile, shifts in viewing platforms and changes in household characteristics. These changes need to be accounted for in order that the sample reflects true representation of the viewing universe. This will ensure that any changes in household characteristics are reflected in the selected sample. The panel homes are drawn for audience measurement from the representative sample determined by the establishment survey. Selection of panel homes is done through a ‘multi-stage, stratified and un-clustered' sample design to ensure that the panel is fully representative of all television households. The maximum period of time that a household may stay on the panel also needs to be clearly defined (say four years).

4.11 There is a need to select households for rating measurement in a transparent manner. The issue of lack of transparency in method adopted for selection of the households by the rating agencies in India has often been raised in the past. In order to ensure transparency in selection of panel homes, the rating agency should provide details pertaining to methodology, establishment survey size, periodicity of the establishment survey, method used to select panel homes from household samples arrived through establishment survey, etc. on its website and also report the same to the accrediting agency at a defined frequency. This data, should also be published on the websites of the rating agencies and the accrediting agency. Such an approach exists in Italy. Auditing of the process will also facilitate in ensuring transparency in selection of panel homes.

Panel size

4.12 Panel size is the number of panel homes, drawn from samples collected during establishment survey, where the audience measurement device is placed. It is an important parameter that determines the accuracy of statistical exercise. The panel size should be representative of age, socio-economic class, gender, working status, delivery platforms and geographical coverage (both urban & rural markets). Generally, larger the panel size the greater would be the accuracy of the results. Smaller panel size results in limited data for analysis and therefore is not truly representative, which compromises the accuracy of the findings.

4.13 Inadequate panel size has serious limitations of not adequately reflecting regions, rural and small towns, demographic profiles, socio-economic classes, plurality of platforms etc. However, increasing the panel size has cost implications especially for taking audience measurement through People meters. So the panel size should be such as to optimize the tradeoff between increased cost and more reliable results.

4.14 M/s TAM Media Research has informed TRAI that TAM measures in-home minute to minute TV viewing for all the TV platforms viz. terrestrial, cable (both analog and digital) and satellite in all states except J&K and North East. As per M/s TAM Media Research, their panel households are spread over 8150 home10. With 15.5 crore TV households in India, this corresponds to less than 0.005% of TV households. Internationally, the panel size is in the range of 0.016% to 0.059% of the TV households11. It is important to note here that India is a large and diverse country stratified into several socio-economic categories and regions. Therefore in our case the panel size needs to be large enough to adequately capture the diversity in demographic profile, cultures, languages etc. Further, M/s TAM Media Research measures in-home TV consumption across urban Indian markets and do not cover rural India as yet. Therefore, limited panel size and inadequate rural & geographic coverage are some of the main shortcomings of the present television rating system in India as it gives disproportionate weightage to viewership pattern of a small sample of viewers. Doordarshan being one of the major providers in the rural areas does not get adequate representation in the panel.

4.15 The Amit Mitra Committee, constituted by the MIB to review the existing Television Rating System in India, also recommended an increase in sample size (panel size or number of panel homes being measured) in terms of number of sample households, coverage of rural areas and addressing the need to cover multiple delivery platforms. The committee recommended that the sample size should be increased by almost 4 times from 8000 approximately to 30,000 households over a period of 5 years. To support the funds required for additional ‘People meters’, the committee suggested that BARC should involve the industry to pay an agreed upon annual fee depending on their size and the number of channels they beam. The committee further mentioned that steps like local manufacturing with indigenization and reduction in import duties may be considered for bringing down the cost of People meter.

Secrecy of panel homes

4.16 In the past, issues relating to secrecy of the panel homes have also been raised. Availability of the entire list of panel households with the broadcasters had also made news in the past. Accessibility of panel homes would mean that the outcome can be manipulated. To avoid any manipulation at household level, the rating agencies should have proper systems in place to safeguard the secrecy of the panel homes. Rotation of panel homes may also help in ensuring secrecy of panel households.

4.17 Some stakeholders had earlier opined that the rating agencies should be made responsible for any violation with regard to maintaining of secrecy of the panel homes. They were of the view that manipulation of the data / TV ratings is a serious offence and should carry appropriate penalties for the agency involved, similar to market manipulation and insider trading in stock market and a suitable legislative mechanism should also be put in place for the same.

Privacy

4.18 It is important that the privacy of panel households be maintained. Privacy is to be maintained at two levels. Firstly, personally identifiable information of panel homes should not be used by rating agencies to advertise, promote or market third-party goods or services. Also personal data like names, addresses or phone numbers of panel members should not be revealed to any third parties for promotion of their business. Secondly, the results of the measurement should not be disclosed on an individual basis. The rating agency should release the measurement results only on aggregate basis, for example the results could be released on demographic basis. In most countries, individual households and viewers are kept anonymous in the rating data and only demographic profile data is released.

4.19 Taking into account the discussion above, some of possible guidelines to arrive at a robust methodology for a television rating system could include the following:

a. An appropriate combination of measurement techniques i.e. surveys, People meters or a combination thereof to be used.

b. All weighting or data adjustment procedures utilized by a rating agency in the process of converting basic raw data to rating reports needs to be based on systematic, logical procedures, consistently applied by the rating agency and defensible by empirical analysis.

c. Ratings are required to be technology neutral and capture data across multiple viewing platforms viz. cable TV, DTH, Terrestrial TV etc. On line platforms to be covered wherever feasible.

d. Rating agency should submit detailed methodology to the accrediting agency and also publish the same on its website.

e. In the event that a rating agency identifies an attempt to bias measurement results by a respondent’s submission of fabricated information, it should eliminate such cases from analysis. In the event that such cases have been included in published data, the agency may be required to assess the effect on results and notify the users about the same along with indication of its practical significance.

f. Any shortcomings, deficiencies, limitations in the rating system needs to be clearly disclosed in the rating reports and also brought to the notice of users of the rating system.

g. The procedure adopted for selection of the panel households and the rotation of the panel households should be made transparent

h. Geographic representation should be provided in proportion to the TV viewing population. The panel should be based on distribution of target viewership for a particular segment like age group, socio-economic class, gender, working status, multiple delivery platforms, all states, urban & rural markets. It should be updated once in 6 months to reflect the developments taking place in the delivery platforms, growth in viewership etc.

i. A minimum panel size (say 15,000) for providing the rating results may be mandated, which can progressively be increased in steps (say 5000 increase every year) to the desired panel size (say to 30000).

j. For selecting the sample homes a large scale establishment survey will be carried out. The household sample covered through this survey should be large enough (say 10 times of the desired panel size) to remain representative of all television households in the country. The establishment survey should be carried out periodically (say annually) to reflect changes in growth of TV homes, changes in demographics, growth in new delivery platforms like internet, variations of growth across markets, etc.

k. A certain percentage (say 25%) of the panel homes should be rotated every year. The rotation should be in such a manner that older panel homes are removed first while maintaining the representativeness of the sample.

l. Rating agency should not include any office, employee or any other member, of broadcasters, advertisers and advertising agencies, in audience measurement sample.

m. Secrecy of the panel households should be maintained.

n. Privacy of panel households should be maintained.

Issues for consultation

Q3. Please give your comments on the guidelines for methodology for audience measurement, as discussed in para 4.19 above, for television rating systems. You are welcome to suggest modifications. Please elaborate your response with justifications.

Q4. What should be the minimum panel size (in terms of numbers of households) that may be mandated in order to ensure statistical accuracy and adequate coverage representing various genre, regions,  demographics etc. for robust television rating system? Should the desired panel size be achieved immediately or in a phased manner? In case of implementing the desired panel size in phased manner, what should be the quantum of increase and periodicity of such increase in size?

Q5. Please give your suggestions/ views on as to how secrecy of panel homes can be ensured?

Consultation Paper on Guidelines/Accreditation Mechanism for Television Rating Agencies in India

http://www.trai.gov.in/ConsultationDescription.aspx?CONSULT_ID=683&qid=0