European Social Survey

From 2004-2008, I was a member of the Central Scientific Team of the European Social Survey, based at ESS HQ at City University. In this role, I directed the methodological research programme on modes of data collection, responsible for designing and conducting (in collaboration with other CST members) a series of studies aimed at assessing: a) whether mixed-mode data collection should be allowed on future rounds of the ESS; b) which modes of data collection should be allowed; and c) within which kinds of overall survey design mixed modes could be employed.

The research began with analyses of data from an earlier ESS mode experiment designed to assess mode effects on measurement, conducted in Hungary by Gallup Europe in collaboration with researchers at NatCen in 2003. These analyses helped inform the design of the subsequent research, conducted in the context of work packages I led in European Commission-funded (under FP6 and FP7) ESS infrastructure projects (ESS Integrated Infrastructure Initiative (I3) and ESSprep).

The studies included a mapping exercise based on desk research and consultation among stakeholders in participating ESS countries to assess demand and capacity for mixing modes; an assessment of the questionnaire design challenges involved in mixing modes; and three large-scale, survey experiments assessing different challenges associated with mixing modes in the context of the ESS: a) how to separate out different types of mode effect on measurement comparability (see Experiment 1); b) how to field long face-to-face interviews in other modes of data collection (see Experiment 2); c) and how to mix web surveys with other modes (concurrently vs. sequentially), while minimising disadvantages for data comparability (see Experiment 3). During this period, I also led a short-term research project (in collaboration with Roger Jowell and Peter Lynn) on ‘Mixing modes of data collection in comparative social surveys’, funded by the UK’s ESRC National Centre for Research Methods.

Since this time, I have worked on a number of other methodological projects based on Swiss ESS data, including a large-scale mode experiment alongside Round 6 of the survey in 2012.

In 2025, in its 12th round, the ESS will finally make the switch from face-to-face interviewing to self-completion modes (web and paper questionnaires) – the culmination of more than two decades worth of careful preparatory research. In my role as Chair of the ESS ERIC Methods Advisory Board, I have been advising the CST and ERIC members on this transition and on appropriate methodological strategies to support the switch. Ensuring researchers using ESS data are equipped with the necessary auxiliary data to assess the impact of mode on selection and measurement errors is a key priority for ensuring the validity of conclusions relating to cross-country and over-time comparisons.

Experiment 1: Separating Causes of Mode Effects, 2005

Experiment 1 was designed and implemented in collaboration with Gallup Europe and Peter Lynn (ISER, Essex), and was focused on the comparability of data from face-to-face and telephone interviews. Specifically, the aim was to address the challenge of how to separate out different types of mode effect on selection and measurement errors. Two probability-based sample surveys were conducted in Budapest and Lisbon in 2005, based on an experimental design involving random assignment of sample members to one of three treatments: 1) a face-to-face interview using showcards (as in the standard ESS); 2) a face-to-face interview without showcards; and 3) a telephone interview. The questionnaire included a selection of questions from the main ESS questionnaire, deemed especially likely to be at risk of mode effects, and other questions aimed at assessing whether respondents on mobile and fixed-line telephones might respond differently.

Collaborators: Robert Manchin, Agnes Illyes, and Gergely Hideg (Gallup Europe); Roger Jowell (former ESS Director); Peter Lynn and Annette Jäckle (ISER, Essex).

Outputs

Publications

  • Roberts, C., Jäckle, A. and Lynn, P. (2007). Causes of Mode Effects: Separating out Interviewer and Stimulus Effects in Comparisons of Face-to-Face and Telephone Surveys. Proceedings of the Survey Research Methods Section of the American Statistical Association, JSM2006-000424, Washington DC: American Statistical Association.
  • Jäckle, A., Roberts, C. and Lynn, P. (2006) Telephone versus Face-to-Face Interviewing: Mode Effects on Data Quality and Likely Causes (Report on Phase II of the ESS-Gallup Mixed Mode Methodology Project). ISER Working Paper 2006-41. Colchester: University of Essex.
  • Jäckle, A.,Roberts, C. and Lynn, P. (2010). Assessing the Effect of Data Collection Mode on Measurement. International Statistical Review, vol. 78(1), 3-20. https://doi.org/10.1111/j.1751-5823.2010.00102.x
  • Lynn, P., and Kaminska, O. (2011). The impact of mobile phones on survey measurement. Public Opinion Quarterly, 77(2): 586-605. https://doi.org/10.1093/poq/nfs046
  • Roberts, C. and Jäckle, A. (2012). Causes of mode effects: Separating out interviewer and stimulus effects in comparisons of face-to-face and telephone surveys. ISER Working Paper 2012-27. Colchester: University of Essex.
  • Villar, A. & Fitzgerald, R. (2017). Using mixed modes in survey research: Evidence from six experiments in the ESS. In M. Breen (ed.), Values and Identities in Europe. Evidence from the European Social Survey. London: Routledge

Invited Talks (C. Roberts)

  • Apr. 2011  – Mixing modes in cross-national surveys – findings from the European Social Survey (ESS) methodological research programme. Workshop on Data Collection Modes in Cross-National Surveys, Eurofound, Dublin
  • Dec. 2009 – Mixing modes of data collection in social surveys. FORS/IMA/MISC Methods and Research Meetings, University of Lausanne, Switzerland
  • Sep. 2006 – Report on Phase II of the ESS-Gallup Mixed Mode Methodology Project. Flash Eurobarometer Methodology Seminar – Gallup, The Adelphi, 1-11 John Adam Street, London, UK
  • Feb. 2006 – Future data collection methods on the ESS.  L’Enquete Sociale Européene (ESS): Apports et Mise en Perspective – Centre de Recherches Politiques de Sciences Po, Paris, France
  • Dec. 2005 – Methodological advances on the ESS: A mixed mode future? Research Seminar, University of Winchester, UK
  • Oct. 2005 – Measuring attitudes cross-nationally: Lessons from the European Social Survey. Sociology Seminar, Nuffield College, University of Oxford, UK.

Conference presentations

  • Parutis, V., Jäckle, A., and Roberts, C. (2015). Effects of showcards on responses in face-to-face surveys surveys. Understanding Society Scientific Conference 2015, Colchester, UK – 21-23/7/2015
  • Jäckle, A. and Roberts, C. Assessing the effect of data collection mode on measurement – Invited paper at the 56th Session of the International Statistical Institute, Lisbon, Portugal – 22-29/8/2007
  • Roberts, C.  Mixing modes of data collection in surveys – Annual Conference of the European Survey Research Association, Prague, Czech Republic – 25-29/6/2007
  • Roberts, C., Jäckle, A. and Lynn, P. Causes of Mode Effects: Separating out Interviewer and Stimulus Effects in Comparisons of Face-to-Face and Telephone Surveys – Annual Conference of the European Survey Research Association, Prague, Czech Republic – 25-29/6/2007
  • Roberts, C.  Mixing modes on the European Social Survey – Implications for Data Quality – Economic and Social Research Council (ESRC) Methods Festival, University of Oxford, UK – 17-20/7/2006
  • Roberts, C., Jäckle, A. and Lynn, P. Mixing modes on the European Social Survey: Implications for data quality – American Association for Public Opinion Research (AAPOR) 61st Annual Conference, Montreal, Canada – 18-21/5/2006
  • Roberts, C., Lynn, P. and Jäckle, A. Mixing modes on the European Social Survey: Implications for data quality – One day conference on mixed mode data collection in comparative social surveys, City University, London, UK – 15/9/2005
  • Roberts, C. and Lynn, P. (2005) Methodological advances on the ESS: A mixed mode future? – American Association for Public Opinion Research (AAPOR) 60th Annual Conference, Miami Beach, USA – 12-15/5/2005

Experiment 2: ESS Telephone Experiment, 2006

The second major mode experiment I designed and carried out for the ESS was focused on the feasibility of using telephone interviewing, focusing on the effect of varying interview length on respondents’ willingness to participate in the survey. The study was carried out in 2006 in five countries: Cyprus, Germany, Hungary, Poland, and Switzerland. Given that telephone interviewing was still the dominant mode for conducting surveys in several ESS countries at the time, we needed to evaluate whether the full-length ESS questionnaire (taking around 1 hour to complete in person) could be fielded by telephone, and how it would impact selection and measurement errors. The experimental design involved randomly assigning sample members to one of three treatments: 1) an invitation to participate in the full 1-hour interview (fully adapted for telephone interviewing); 2) an invitation to take part in a 45 minute interview (the full ESS minus one rotating module); and 3) an invitation to take part in a 30 minute interview with an invitation at the end of this to a subsequent 30-minute follow-up interview (i.e., the full ESS split in half).

Collaborators: Gillian Eva, Peter Lynn, Jerry Johnson, Nick Allum and Roger Jowell; National Coordinators and fieldwork agencies in Cyprus, Germany, Hungary, Poland and Switzerland.

Outputs

Publications

  • Roberts, C., Sarrasin, O. and Ernst Stähli, M (2020). The relative impact of different sources of measurement non-equivalence in comparative surveys: Examples of item, method and construct bias. Conditional acceptance from Survey Research Methods, special issue on ‘Measurement Invariance: Testing for it and explaining why it is absent’. https://doi.org/10.18148/srm/2020.v14i4.7416
  • Roberts, C., Eva, G., Lynn, P. and Johnson, J (2011). Measuring the effect of interview length on response propensity and response quality in a telephone survey – Final report of the ESS CATI experiment. ESSi JRA1 Deliverable 5. Centre for Comparative Social Surveys, City University London.
  • Roberts, C., Eva, G., Allum, N. and Lynn, P. (2010). Data quality in telephone surveys and the effect of questionnaire length: a cross-national experiment. ISER Working Paper 2010-36. Colchester: University of Essex.
  • Roberts, C. (2008) Designing equivalent questionnaires for a mixed mode European Social Survey: Report on the findings of ESS mode experiments. Centre for Comparative Social Surveys, City University London.

Presentations

  • Roberts, C. and Eva, G. Challenges in adapting face-to-face questionnaires for mixed mode data collection. 7th International Conference on Social Science Methodology, Naples, Italy – 1-5/9/2008
  • Eva, G., Roberts, C., Lynn, P. and Allum, N. Telephone interviews. The effect of length on data quality and response rates.  International Conference on Survey Methods in Multinational, Multiregional, and Multicultural Contexts (3MC), Berlin, Germany – 25-28/6/2008
  • Roberts, C., Eva, G., Allum, N. and Lynn, P. Does length matter? The effect of interview duration on data quality in telephone interviews – American Association for Public Opinion Research (AAPOR) 63rd Annual Conference, New Orleans, USA – 15-18/5/2008
  • Eva, G., Roberts, C. and Lynn, P. Measuring the Effect of Interview Length on Telephone Survey Cooperation – American Association for Public Opinion Research (AAPOR) 63rd Annual Conference, New Orleans, USA – 15-18/5/2008

Experiment 3: ESS Concurrent vs. Sequential Mixed Mode Experiment, 2008

The third experiment for the ESS I was involved in the design of (in collaboration with Willem Saris) was carried out in the Netherlands in 2008, shortly after I moved to Stanford University to work on the American National Election Studies. The study was funded by the EC’s 7th Framework Programme for the preparatory phase of the ESS infrastructure (‘ESSPrep’). The aim of the experiment was to evaluate effects of alternative mixed-mode designs (‘concurrent’ vs. ‘sequential’ ways of mixing modes) on response rates, the representativeness of samples, survey costs, and data quality. This study was conducted in collaboration with other members of the CST at the time (notably, Roger Jowell, Willem Saris, Geert Loosvelt, Dirk Heerweg, Peter Lynn, Gillian Eva, and Sally Widdop), and the resulting datasets have been used widely by other ESS CST members and associated researchers (including Rory Fitzgerald – now Director of the ESS ERIC, Ana Villar, Peter Martin, Melanie Revilla, and Jorre Vannieuwenhuyze).

Selected Outputs

  • Roberts, C. (2008). Assessing the cost-effectiveness of different modes for ESS data collection. ESSPrep WP6 Deliverable 1 (Mode Experiment Proposal). Centre for Comparative Social Surveys, City University, London.
  • Eva, G., Loosveldt, G., Lynn, P., Martin, P., Revilla, M., Saris, W., and Vannieuwenhuyze, J. (2010). ESS Prep WP6 – Mixed Mode Experiment. Deliverable 21 (Final report). London: City University.
  • Villar, A. & Fitzgerald, R. (2017). Using mixed modes in survey research: Evidence from six experiments in the ESS. In M. Breen (ed.), Values and Identities in Europe. Evidence from the European Social Survey. London: Routledge
  • Martin, P. & Lynn, P. (2011). The effect of mixed mode survey designs on simple and complex analyses. Centre for Comparative Social Surveys, City, University of London, Working Paper no. 04, November 2011
  • Martin, P. (2011). What makes a good mix? Chances and challenges of mixed mode data collection in the ESS. Centre for Comparative Social Surveys, City, University of London, Working Paper no. 02, February 2011
  • Revilla, M. (2010). Quality in Unimode and Mixed-Mode designs: A Multitrait-Multimethod approach. Survey Research Methods, 4 (3), 151-164. DOI: https://doi.org/10.18148/srm/2010.v4i3.4278
  • Vannieuwenhuyze, J., Loosveldt, G., & Molenberghs, G. (2010).
    A Method for Evaluating Mode Effects in Mixed-mode Surveys. Public Opinion Quarterly, 74 (5), 1027-45. DOI: https://doi.org/10.1093/poq/nfq059

Mapping Exercise to Assess the Demand and Capacity for Mixing Modes on the European Social Survey, 2007

In 2007, as part of the ESS I3 infrastructure funded ‘Joint Research Activity 1’ (a Work Package designed and led by me), I undertook – together with Gillian Eva and Sally Widdop – a major mapping exercise to assess the demand and capacity for mixing modes of data collection on the ESS across participating European countries. The mapping exercise involved a broad consultation exercise with different stakeholders (national coordinators, NSI’s and fieldwork agencies) and desk research methods to gather information relating to diverse indicators of demand and capacity, including mode penetration, mode preferences and infractructure capacity for conducting survey fieldwork in different modes and associated costs.

Outputs

  • Roberts, C., Eva, G. and Widdop, S. (2008) Assessing the demand and capacity for mixing modes of data collection on the European Social Survey: Final report of the mapping exercise. Centre for Comparative Social Surveys, City University London.

Designing Equivalent Questionnaires for a Mixed Mode ESS

Based on the accumulated findings of the different phases of ESS mixed mode experimentation, and preparatory work adapting ESS questionnaires for telephone interviewing and for online surveys, I undertook a review of the main lessons learned. The review formed part of the activities of the ESS I3 JRA1 project. The focus of the report (a work package deliverable) was on the implications of mode effects for the equivalence of measurement across modes and recommendations for future research.

Outputs

  • Roberts, C. (2008). Designing equivalent questionnaires for a mixed mode European Social Survey: Report on the findings of ESS mode experiments. ESSi JRA1, Deliverable 3. Centre for Comparative Social Surveys, City University, London.

Developing mixed mode methods in comparative social surveys, 2005

In 2005, in collaboration with Roger Jowell and Peter Lynn, we obtained an ESRC National Centre for Research Methods Short-term Project grant for a new ‘Network for Methodological Innovation’ on the topic of Developing mixed mode methods in comparative social surveys. The project consisted of three main activities led by me: 1) a major ‘methodological review’ – a review of the literature on mixed mode data collection; 2) a one-day international conference on mixed mode surveys; and 3) a one-day expert workshop with leading survey methodologists, both held at City University in September 2005. The report on the workshop informed the design of future mode research on the ESS, as well as helping to establish the topic of mixed mode data collection on the survey methodology research agenda more generally. The project also helped to build a broader international network of researchers interested in and working on the challenge of survey mode effects.

Outputs

  • Roberts, C. (2007). Mixing modes of data collection in surveys: A methodological review. National Centre for Research Methods (NCRM) Methods Review Papers, NCRM/008, Southampton: University of Southampton.
  • Roberts, C. (2005). Final report on the expert workshop on mixed mode data collection in comparative social surveys. In: Mixed Mode Data Collection in Comparative Social Surveys, 15-16 September 2005, City University, London.

Other ESS-related research

Since joining the Institute of Social Sciences at UNIL, I have continued to conduct methodological research using ESS data – from experiments linked to, or conducted alongside the survey, administrative data from the sampling frame, and survey paradata (call records documenting outcomes of different phases of fieldwork) in collaboration with colleagues from FORS and UNIL.

Collaborators: Michèle Ernst Stähli, Dominque Joye, Caroline Vandenplas, Oriane Sarrasin, and Jessica Herzing

Outputs

Publications

  • Roberts, C., Vandenplas, C., and Herzing, J. (2020). A validation of R-indicators as a measure of the risk of bias using data from a non-response follow-up survey. Journal of Official Statistics, 36(3): 675-701. https://doi.org/10.2478/jos-2020-0034
  • Roberts, C., Sarrasin, O. and Ernst Stähli, M (2020). The relative impact of different sources of measurement non-equivalence in comparative surveys: Examples of item, method and construct bias. Conditional acceptance from Survey Research Methods, special issue on ‘Measurement Invariance: Testing for it and explaining why it is absent’. https://doi.org/10.18148/srm/2020.v14i4.7416
  • Roberts, C., and Vandenplas, C. (2017). Estimating components of mean-squared error to evaluate the benefits of mixing data collection modes. Journal of Official Statistics, 33(2): 303-334. http://dx.doi.org/10.1515/JOS-2017-0016

Presentations

  • Roberts, C., Sarrasin, O, and Ernst Stähli, M. (2018) Sources of error in 3MC surveys – an assessment of the relative impact of mode and translation error on measurement invariance. Paper presented at the 73rd Annual Conference of the American Association for Public Opinion Research, 15th-19th May, 2018, Denver, Colorado
  • Ernst Stähli, M., Sarrasin, O. and Roberts, C. (2017). Do mode effects matter in cross-national surveys?  An assessment of measurement invariance across data collection methods and countries. Paper presented at the 7th Conference of the European Survey Research Association, Lisbon, Portugal, 17th-21st July 2017.
  • Roberts, C., Villar, A., and Yang, Y. (2017). Do 11-point scales guarantee better measurement quality than shorter scales in multi-lingual surveys? Paper presented at the 7th Conference of the European Survey Research Association, Lisbon, Portugal, 17th-21st July 2017.
  • Roberts, C., and Vandenplas, C. (2016). Estimating components of mean-squared error to evaluate the benefits of mixing data collection modes.   Paper presented at the 71st Annual Conference of the American Association for Public Opinion Research, Austin, TX., USA – 12-15/5/2016.
  • Roberts, C. and Vandenplas, C. (2015). A validation of R-indicators as a measure of the risk of bias using data from a non-response follow-up survey. Paper presented at the 70th Annual Conference of the American Association for Public Opinion Research, Miami, FL., USA – 14-17/5/2015
  • Parutis, V., Jäckle, A., and Roberts, C. (2015). Effects of showcards on responses in face-to-face surveys surveys. Understanding Society Scientific Conference 2015, Colchester, UK – 21-23/7/2015
  • Roberts, C., Vandenplas, C., and Ernst Staehli, M. (2012). Evaluating the impact of response enhancement methods on non-response bias in the Swiss European Social Survey. Paper presented at the International Workshop on Total Survey Error, Sanpoort, Netherlands – 2-4/9/2012
  • Roberts, C. Mode effects in Switzerland: non-response and measurement error on the ESS. Paper presented at the Annual Conference of the World Association for Public Opinion Research, Lausanne, Switzerland -11-13/9/2009
  • Roberts, C. Using survey paradata to evaluate the benefits of mixed mode data collection – Annual Conference of the European Survey Research Association, Warsaw, Poland – 29/6-3/7/2009

LIVES-FORS Mixed Mode Experiment 2012

In 2012, I collaborated with Dominique Joye (UNIL) and Michèle Ernst Stähli (FORS), leading on the design, implementation and analysis of a large-scale (n=3,600) mixed mode experiment in the French-speaking region of Switzerland. The study was conducted in the context of a work package of the National Centre for Competence in Research “LIVES – Overcoming Vulnerability: Life Course Perspectives”, funded by the Swiss National Science Foundation (Individual Project 15 – Life transitions in context: Towards an integrated methodological framework for studying the impact of critical events, led by Dominique Joye). The experiment was conducted alongside Round 6 of the European Social Survey (2012), enabling comparisons between the standard ESS data based on face-to-face interviewing, and alternative single and mixed mode designs (telephone only, telephone plus mail; web plus mail; mail only). Further information about the study and its outputs is available here.