Skip to main content
SpringerLink
Log in

Measurement bias in self-reports of offending: a systematic review of experiments

  • Published:
Journal of Experimental Criminology Aims and scope Submit manuscript

Abstract

Objectives

Self-reported offending is one of the primary measurement methods in criminology. In this article, we aimed to systematically review the experimental evidence regarding measurement bias in self-reports of offending.

Methods

We carried out a systematic search for studies that (a) included a measure of offending, (b) compared self-reported data on offending between different methods, and (c) used an experimental design. Effect sizes were used to summarize the results.

Results

The 21 pooled experiments provided evidence regarding 18 different types of measurement manipulations which were grouped into three categories, i.e., Modes of administration, Procedures of data collection, and Questionnaire design. An analysis of the effect sizes for each experimental manipulation revealed, on the one hand, that self-reports are reliable across several ways of collecting data and, on the other hand, self-reports are influenced by a wide array of biasing factors. Within these measurement biases, we found that participants’ reports of offending are influenced by modes of administration, characteristics of the interviewer, anonymity, setting, bogus pipeline, response format, and size of the questionnaire.

Conclusions

This review provides evidence that allows us to better understand and improve crime measurements. However, many of the experiments presented in this review are not replicated and additional research is needed to test further aspects of how asking questions may impact participants’ answers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. “Item count technique” or “Unmatched count technique” are methods to reduce response bias, in which participants are randomly divided into at least two groups. The control group receives a list of questions without the sensitive item while the experimental group receives the same questions including the sensitive item. The prevalence estimate is calculated by the subtraction of the mean sum of the control group from the mean sum of the experimental group (Wolter and Laier 2014).

  2. “Random response technique” is a method to reduce response bias, in which participants are presented with a pair of questions, one sensitive and one innocuous. Participants use a randomization device, such as a dice or a coin, to either give a predetermined answer (e.g., yes or no) or to answer the sensitive question truthfully. A prevalence estimation is possible from knowing the probability of the predetermined outcome (Wolter and Preisendörfer 2013).

References

  • Auty, K. M., Farrington, D. P., & Coid, J. W. (2015). The validity of self-reported convictions in a community sample: findings from the Cambridge study in delinquent development. European Journal of Criminology, 12, 562–580.

    Article  Google Scholar 

  • Baier, D. (2017). Computer-assisted versus paper-and-pencil self-report delinquency surveys: results of an experimental study. European Journal of Criminology, 15, 385–402.

    Article  Google Scholar 

  • Baly, M. W., & Cornell, D. G. (2011). Effects of an educational video on the measurement of bullying by self-report. Journal of School Violence, 10, 221–238.

    Article  Google Scholar 

  • Beatty, J. R., Chase, S. K., & Ondersma, S. J. (2014). A randomized study of the effect of anonymity, quasi-anonymity, and certificates of confidentiality on postpartum women’s disclosure of sensitive information. Drug and Alcohol Dependence, 134, 280–284.

    Article  Google Scholar 

  • Beebe, T. J., Harrison, P. A., Mcrae, J. A., Anderson, R. E., & Fulkerson, J. A. (1998). An evaluation of computer-assisted self-interviews in a school setting. Public Opinion Quarterly, 62, 623–632.

    Article  Google Scholar 

  • Beebe, T. J., Harrison, P. A., Park, E., McRae, J. A., Jr., & Evans, J. (2006). The effects of data collection mode and disclosure on adolescent reporting of health behavior. Social Science Computer Review, 24, 476–488.

    Article  Google Scholar 

  • Bender, R., Friede, T., Koch, A., Kuss, O., Schlattmann, P., Schwarzer, G., & Skipka, G. (2018). Methods for evidence synthesis in the case of very few studies. Research Synthesis Methods, 9, 382–392.

    Google Scholar 

  • Biglan, M., Gilpin, E. A., Rohrbach, L. A., & Pierce, J. P. (2004). Is there a simple correction factor for comparing adolescent tobacco-use estimates from school-and home-based surveys? Nicotine and Tobacco Research, 6, 427–437.

    Article  Google Scholar 

  • Borenstein, M., Hedges, L., Higgins, J., & Rothstein, H. (2009). Introduction to meta-analysis. Hoboken: John Wiley & Sons.

    Book  Google Scholar 

  • Borenstein, M., Hedges, L., Higgins, J., & Rothstein, H. (2014). Comprehensive meta-analysis, version 3. Englewood: Biostat.

    Google Scholar 

  • Brener, N. D., Eaton, D. K., Kann, L., Grunbaum, J. A., Gross, L. A., Kyle, T. M., & Ross, J. G. (2006). The association of survey setting and mode with self-reported health risk behaviors among high school students. Public Opinion Quarterly, 70, 354–374.

    Article  Google Scholar 

  • Chaiken, J. M., & Chaiken, M. R. (1982). Varieties of criminal behavior. Santa Monica: Rand Corporation.

    Google Scholar 

  • Chan, J. H., Myron, R., & Crawshaw, M. (2005). The efficacy of non-anonymous measures of bullying. School Psychology International, 26, 443–458.

    Article  Google Scholar 

  • Clark, J. P., & Tifft, L. L. (1966). Polygraph and interview validation of self-reported deviant behavior. American Sociological Review, 31, 516–523.

    Article  Google Scholar 

  • Cops, D., De Boeck, A., & Pleysier, S. (2016). School vs. mail surveys: disentangling selection and measurement effects in self-reported juvenile delinquency. European Journal of Criminology, 13, 92–110.

    Article  Google Scholar 

  • Cutler, S. F., Wallace, P. G., & Haines, A. P. (1988). Assessing alcohol consumption in general practice patients - a comparison between questionnaire and interview (findings of the Medical Research Council’s general practice research framework study on lifestyle and health). Alcohol and Alcoholism, 23, 441–450.

    Google Scholar 

  • Dalton, D. R., Wimbush, J. C., & Daily, C. M. (1994). Using the unmatched count technique (UCT) to estimate base rates for sensitive behavior. Personnel Psychology, 47, 817–829.

    Article  Google Scholar 

  • Daylor, J. M., Blalock, D. V., Davis, T., Klauberg, W. X., Stuewig, J., & Tangney, J. P. (2019). Who tells the truth? Former inmates’ self-reported arrests vs. official records. Journal of Criminal Justice. https://doi.org/10.1016/j.jcrimjus.2019.04.002.

    Article  Google Scholar 

  • Durant, L. E., Carey, M. P., & Schroder, K. E. (2002). Effects of anonymity, gender, and erotophilia on the quality of data obtained from self-reports of socially sensitive behaviors. Journal of Behavioral Medicine, 25, 439–467.

    Article  Google Scholar 

  • Eaton, D. K., Brener, N. D., Kann, L., Denniston, M. M., McManus, T., Kyle, T. M., Roberts, A. M., Flint, K. H., & Ross, J. G. (2010). Comparison of paper-and-pencil versus web administration of the youth risk behavior survey (YRBS): risk behavior prevalence estimates. Evaluation Review, 34, 137–153.

    Article  Google Scholar 

  • Enzmann, D. (2013). The impact of questionnaire design on prevalence and incidence rates of self-reported delinquency: results of an experiment modifying the ISRD-2 questionnaire. Journal of Contemporary Criminal Justice, 29, 147–177.

    Article  Google Scholar 

  • Farrington, D. P. (1973). Self-reports of deviant behavior: predictive and stable? Journal of Criminal Law and Criminology, 64, 99–110.

    Article  Google Scholar 

  • Gomes, H. S., Maia, A., & Farrington, D. P. (2018a). Measuring offending: self-reports, official records, systematic observation and experimentation. Crime Psychology Review, 4, 26–44.

    Article  Google Scholar 

  • Gomes, H. S., Maia, A. & Farrington, D. P. (2018b). Method effects in measuring self-reported offending: an experimental comparison of personal, paper-and-pencil, and computer assisted interviews. (Paper presented at the 74th Annual Meeting of the American Society of criminology, Atlanta, GA).

  • Gribble, J. N., Rogers, S. M., Miller, H. G., & Turner, C. R. (1998). Measuring AIDS-related behaviors in older populations: methodological issues. Research on Aging, 20, 798–821.

    Article  Google Scholar 

  • Gribble, J. N., Miller, H. G., Cooley, P. C., Catania, J. A., Pollack, L., & Turner, C. F. (2000). The impact of T-ACASI interviewing on reported drug use among men who have sex with men. Substance Use & Misuse, 35, 869–890.

    Article  Google Scholar 

  • Grysman, B. & Johnson, C. (2010). Effects of value affirmation on drug use disclosure in patients entering a community mental health center. Doctoral dissertation, Hofstra University, Long Island, NY.

  • Hamby, S., Sugarman, D. B., & Boney-McCoy, S. (2006). Does questionnaire format impact reported partner violence rates?: an experimental study. Violence and Victims, 21, 507–518.

    Article  Google Scholar 

  • Hardt, R. H., & Peterson-Hardt, S. (1977). On determining the quality of the delinquency self-report method. Journal of Research in Crime and Delinquency, 14, 247–259.

    Article  Google Scholar 

  • Hindelang, M. J., Hirschi, T., & Weis, J. G. (1981). Measuring delinquency. London: Sage.

    Google Scholar 

  • Horney, J., & Marshall, I. H. (1992). An experimental comparison of two self-report methods for measuring lambda. Journal of Research in Crime and Delinquency, 29, 102–121.

    Article  Google Scholar 

  • Huang, F. L., & Cornell, D. G. (2015). The impact of definition and question order on the prevalence of bullying victimization using student self-reports. Psychological Assessment, 27, 1484–1493.

    Article  Google Scholar 

  • Jolliffe, D., & Farrington, D. P. (2014). Self-reported offending: reliability and validity. In G. J. N. Bruinsma & D. L. Weisburd (Eds.), Encyclopedia of criminology and criminal justice (pp. 4716–4723). New York: Springer-Verlag.

    Chapter  Google Scholar 

  • King, C. A., Hill, R. M., Wynne, H. A., & Cunningham, R. M. (2012). Adolescent suicide risk screening: the effect of communication about type of follow-up on adolescents' screening responses. Journal of Clinical Child and Adolescent Psychology, 41, 508–515.

    Article  Google Scholar 

  • Kivivuori, J., Salmi, V., & Walser, S. (2013). Supervision mode effects in computerized delinquency surveys at school: Finnish replication of a Swiss experiment. Journal of Experimental Criminology, 9, 91–107.

    Article  Google Scholar 

  • Kleck, G., & Roberts, K. (2012). What survey modes are most effective in eliciting self-reports of criminal or delinquent behavior? In L. Gideon (Ed.), Handbook of survey methodology for the social sciences (pp. 417–439). New York: Springer.

    Chapter  Google Scholar 

  • Knapp, H., & Kirk, S. A. (2003). Using pencil and paper, internet and touch-tone phones for self-administered surveys: does methodology matter? Computers in Human Behavior, 19, 117–134.

    Article  Google Scholar 

  • Krohn, M. D., Waldo, G. P., & Chiricos, T. G. (1974). Self-reported delinquency: a comparison of structured interviews and self-administered checklists. Journal of Criminal Law and Criminology, 65, 545–553.

    Article  Google Scholar 

  • Krohn, M. D., Lizotte, A. J., Phillips, M. D., Thornberry, T. P., & Bell, K. A. (2013). Explaining systematic bias in self-reported measures: factors that affect the under-and over-reporting of self-reported arrests. Justice Quarterly, 30, 501–528.

    Article  Google Scholar 

  • Krosnick, J. A., & Presser, S. (2010). Question and questionnaire design. In P. V. Marsden & J. D. Wright (Eds.), Handbook of survey research (2nd ed., pp. 263–313). Bingley: Emerald.

    Google Scholar 

  • Krumpal, I. (2013). Determinants of social desirability bias in sensitive surveys: a literature review. Quality and Quantity, 47, 2025–2047.

    Article  Google Scholar 

  • Kulik, J. A., Stein, K. B., & Sarbin, T. R. (1968). Disclosure of delinquent behavior under conditions of anonymity and nonanonymity. Journal of Consulting and Clinical Psychology, 32, 506–509.

    Article  Google Scholar 

  • Loeber, R., Farrington, D. P., Hipwell, A. E., Stepp, S. D., Pardini, D., & Ahonen, L. (2015). Constancy and change in the prevalence and frequency of offending when based on longitudinal self-reports or official records: comparisons by gender, race, and crime type. Journal of Developmental and Life-Course Criminology, 1, 150–168.

    Article  Google Scholar 

  • Loftus, E. F., & Marburger, W. (1983). Since the eruption of Mt. St. Helens, has anyone beaten you up? Improving the accuracy of retrospective reports with landmarkevents. Memory and Cognition, 11, 114–120.

    Article  Google Scholar 

  • Lucia, S., Herrmann, L., & Killias, M. (2007). How important are interview methods and questionnaire designs in research on self-reported juvenile delinquency? An experimental comparison of Internet vs paper-and-pencil questionnaires and different definitions of the reference period. Journal of Experimental Criminology, 3, 39–64.

    Article  Google Scholar 

  • Maxfield, M. G., Weiler, B. L., & Widom, C. S. (2000). Comparing self-reports and official records of arrests. Journal of Quantitative Criminology, 16, 87–110.

    Article  Google Scholar 

  • Moskowitz, J. M. (2004). Assessment of cigarette smoking and smoking susceptibility among youth: telephone computer-assisted self-interviews versus computer-assisted telephone interviews. Public Opinion Quarterly, 68, 565–587.

    Article  Google Scholar 

  • Nye, F. I., & Short, J. F. (1957). Scaling delinquent behavior. American Sociological Review, 22, 326–331.

    Article  Google Scholar 

  • Ong, A. D., & Weiss, D. J. (2000). The impact of anonymity on responses to sensitive questions. Journal of Applied Social Psychology, 30, 1691–1708.

    Article  Google Scholar 

  • Piquero, A. R., Schubert, C. A., & Brame, R. (2014). Comparing official and self-report records of offending across gender and race/ethnicity in a longitudinal study of serious youthful offenders. Journal of Research in Crime and Delinquency, 51, 526–556.

    Article  Google Scholar 

  • Porterfield, A. L. (1943). Delinquency and its outcome in court and college. American Journal of Sociology, 49, 199–208.

    Article  Google Scholar 

  • Potdar, R., & Koenig, M. A. (2005). Does audio-CASI improve reports of risky behavior? Evidence from a randomized field trial among young urban men in India. Studies in Family Planning, 36, 107–116.

    Article  Google Scholar 

  • Rehm, J., & Spuhler, T. (1993). Measurement error in alcohol consumption: the Swiss health survey. European Journal of Clinical Nutrition, 47, S25–S30.

    Google Scholar 

  • Richman, W., Kiesler, S., Weisband, S., & Drasgow, F. (1999). A meta-analytic study of social desirability distortion in computer-administered questionnaires, traditional questionnaires, and interviews. Journal of Applied Psychology, 84, 754–775.

    Article  Google Scholar 

  • Schore, J., Maynard, R., & Piliavin, I. (1979). The accuracy of self-reported arrest data. Princeton: Mathematica Policy Research.

    Google Scholar 

  • Schwarz, N. (1999). Self-reports: how the questions shape the answers. American Psychologist, 54, 93–105.

    Article  Google Scholar 

  • Sobell, L. C., & Sobell, M. B. (1981). Effects of three interview factors on the validity of alcohol abusers’ self-reports. American Journal of Drug and Alcohol Abuse, 8, 225–237.

    Article  Google Scholar 

  • Strang, E., & Peterson, Z. D. (2016). Use of a bogus pipeline to detect men’s underreporting of sexually aggressive behavior. Journal of Interpersonal Violence (in press). https://doi.org/10.1177/0886260516681157.

  • Thornberry, T. P., & Krohn, M. D. (2000). The self-report method for measuring delinquency and crime. In D. Duffee (Ed.), Criminal justice (pp. 33–84). Washington, DC: National Institute of Justice.

    Google Scholar 

  • Tourangeau, R., & McNeeley, M. E. (2003). Measuring crime and crime victimization: methodological issues. In J. V. Pepper & C. V. Petrie (Eds.), Measurement problems in criminal research: Workshop summary (pp. 10–42). Washington, DC: National Academies Press.

    Google Scholar 

  • Tourangeau, R., & Smith, T. W. (1996). Asking sensitive questions: the impact of data collection mode, question format, and question context. Public Opinion Quarterly, 60, 275–304.

    Article  Google Scholar 

  • Tourangeau, R., & Yan, T. (2007). Sensitive questions in surveys. Psychological Bulletin, 133, 859–883.

    Article  Google Scholar 

  • Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. New York: Cambridge University Press.

    Book  Google Scholar 

  • Trapl, E. S., Taylor, H. G., Colabianchi, N., Litaker, D., & Borawski, E. A. (2013). Value of audio-enhanced handheld computers over paper surveys with adolescents. American Journal of Health Behavior, 37, 62–69.

    Article  Google Scholar 

  • Turner, C. F., & Miller, H. G. (1997). Monitoring trends in drug use: strategies for the 21st century. Substance Use and Misuse, 32, 2093–2103.

    Article  Google Scholar 

  • Turner, C. F., Ku, L., Rogers, S. M., Lindberg, L. D., Pleck, J. H., & Sonenstein, F. L. (1998). Adolescent sexual behavior, drug use, and violence: increased reporting with computer survey technology. Science, 280, 867–873.

    Article  Google Scholar 

  • van De Looij-Jansen, P. M., & De Wilde, E. J. (2008). Comparison of web-based versus paper-and-pencil self-administered questionnaire: Effects on health indicators in Dutch adolescents. Health Services Research, 43, 1708–1721.

    Article  Google Scholar 

  • van de Looij-Jansen, P. M., Goldschmeding, J. E., & de Wilde, E. J. (2006). Comparison of anonymous versus confidential survey procedures: effects on health indicators in Dutch adolescents. Journal of Youth and Adolescence, 35, 652–658.

    Article  Google Scholar 

  • Walser, S., & Killias, M. (2012). Who should supervise students during self-report interviews? A controlled experiment on response behavior in online questionnaires. Journal of Experimental Criminology, 8, 17–28.

    Article  Google Scholar 

  • Wilson, D. B. (2009). Missing a critical piece of the pie: simple document search strategies inadequate for systematic reviews. Journal of Experimental Criminology, 5, 429–440.

    Article  Google Scholar 

  • Wolter, F., & Laier, B. (2014). The effectiveness of the item count technique in eliciting valid answers to sensitive questions: an evaluation in the context of self-reported delinquency. Survey Research Methods, 8, 153–168.

    Google Scholar 

  • Wolter, F., & Preisendörfer, P. (2013). Asking sensitive questions: an evaluation of the randomized response technique versus direct questioning using individual validation data. Sociological Methods and Research, 42, 321–353.

    Article  Google Scholar 

Download references

Funding

This study was conducted at the Psychology Research Centre (PSI/01662), School of Psychology, University of Minho, and supported by the Portuguese Foundation for Science and Technology and the Portuguese Ministry of Science, Technology and Higher Education (UID/PSI/01662/2019), through the national funds (PIDDAC).

The first author was supported by a doctoral grant from the Portuguese Foundation for Science and Technology (FCT - SFRH/BD/122919/2016).

Author information

Authors and Affiliations

  1. Psychology Research Centre (CIPsi), University of Minho, School of Psychology - Campus de Gualtar, 4710-057, Braga, Portugal

    Hugo S. Gomes & Ângela Maia

  2. Institute of Criminology, Cambridge University, Cambridge, UK

    David P. Farrington

  3. Sociology and Criminology & Law, University of Florida, Gainesville, FL, USA

    Marvin D. Krohn

Authors
  1. Hugo S. Gomes
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David P. Farrington
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ângela Maia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marvin D. Krohn
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hugo S. Gomes.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gomes, H.S., Farrington, D.P., Maia, Â. et al. Measurement bias in self-reports of offending: a systematic review of experiments. J Exp Criminol 15, 313–339 (2019). https://doi.org/10.1007/s11292-019-09379-w

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11292-019-09379-w

Keywords

Search

Navigation