From minor data gaps to major errors

Simulation study to demonstrate potential bias of online surveys

Abstract

Online data collection is one of the new directions for surveys. With online-only data collection, a probability sample is almost infeasible, which may compromise the generalisability of the results and bias the estimates. In this paper the authors present a case study of Hungary, where internet access is far from reaching full penetration (80 percent) but considered average by European standards. The study focuses on two main points: (1) the extent to which estimates from online surveys are biased in general, and (2) the socio-demographic and attitudinal aspects relevant to the magnitude of the bias. The main method of analysis is simulation, which is based on multiple data sources. Based on administrative data the demographic composition is modeled for both offline and online populations, while for the attitude dimensions face-to-face survey data of the European Social Survey is used. The study evaluates estimates from simulated online and face-to-face data collections involving several post-stratification processes. The overall conclusion of the study is that although online data collection seems to be a convenient data collection tool for social research, given the relatively high internet penetration, it is not yet a suitable method. The study found that even a minor data gap from the offline population leads to major error in the estimates: based on the characteristics of internet penetration in Hungary, in 67 percent of the cases erroneous estimates were obtained. For relevant research dimensions such as interest in politics, religiosity, health and marital status, the online data collection significantly under- or overestimates the likely real population proportions.