Measuring Effectiveness in Digital Game-Based Learning: A Methodological Review.

  • Anissa All iMinds-MICT-Ghent University
  • Elena Patricia Nunez Castellar iMinds-MICT-Ghent University
  • Jan Van Looy iMinds-MICT-Ghent University
Keywords: Digital Game-Based Learning, Effectiveness assessment, cognitive learning outcomes

Abstract

In recent years, a growing number of studies are being conducted into the effectiveness of digital game-based learning (DGBL). Despite this growing interest, there is a lack of sound empirical evidence on the effectiveness of DGBL due to different outcome measures for assessing effectiveness, varying methods of data collection and inconclusive or difficult to interpret results. This has resulted in a need for an overarching methodology for assessing the effectiveness of DGBL. The present study took a first step in this direction by mapping current methods used for assessing the effectiveness of DGBL. Results showed that currently, comparison of results across studies and thus looking at effectiveness of DGBL on a more general level is problematic due to diversity in and suboptimal study designs. Variety in study design relates to three issues, namely different activities that are implemented in the control groups, different measures for assessing the effectiveness of DGBL and the use of different statistical techniques for analyzing learning outcomes. Suboptimal study designs are the result of variables confounding study results. Possible confounds that were brought forward in this review are elements that are added to the game as part of the educational intervention (e.g., required reading, debriefing session), instructor influences and practice effects when using the same test pre- and post-intervention. Lastly, incomplete information on the study design impedes replication of studies and thus falsification of study results.

Author Biographies

Anissa All, iMinds-MICT-Ghent University

Anissa All graduated in July 2011 as a Master New Media & Society at Ghent University. In February 2011 Anissa followed an internship at MICT. Since July 2011 she is part of the MICT team a junior researcher, working on living lab research and projects concerning digital game-based learning.

Since January 2013, Anissa is working on a PhD through a IWT grant. In this PhD research, she will develop a standard procedure to measure effectiveness of serious games aimed at cognitive learning outcomes. 

Elena Patricia Nunez Castellar, iMinds-MICT-Ghent University

Elena Núñez Castellar received her degree in Psychology at the Universidad Nacional de Colombia in 2006. During the last year of her studies she worked as a clinical neuropsychology assistant getting expertise in the selection and interpretation of standardized neuropsychological tests. In 2011 she obtained her degree of Doctor in Psychology from the Ghent University. During her PhD she got broad experience with research methods in cognitive neuroscience, namely EEG (electro-encephalography) and reaction times research. Since 2012 she joined the research group for Media & ICT (iMinds-MICT). She conducts experimental studies to evaluate the effectiveness of educational games and quality of experience in immersive media.

Jan Van Looy, iMinds-MICT-Ghent University

Jan Van Looy is assistant professor at the research group for Media and ICT (iMinds-MICT) at Ghent University, Belgium, where his team conducts multidisciplinary research into digital games and immersive media. Topics include digital game usage and experience, serious game effectiveness, serious games for raising awareness of societal issues, for language learning, for mathematics education, for increasing traffic safety and for improving energy efficiency, gender and gaming, stereoscopic and omnidirectional video.

References

[1] Sawyer, B. and P. Smith, "Taxonomy for Serious Games". Digitalmil, Inc& Serious Games Initiative/Univ. of Central Florida, RETRO Lab, 2008.
[2] Clark, R., "Learning from serious games? Arguments, evidence, and research suggestions". Educational Technology, 2007. 47(3): p. 56-59.
[3] Mayer, I., et al., "A Brief Methodology for Researching and Evaluating Serious Games and Game-Based Learning". In Psychology, Pedagogy, and Assessment in Serious Games, T.C.T.H.E.B.G.B.P. Moreno-Ger, Editor. 2013, ICI Global.
[4 ] Mayer, R., "Multimedia Learning and Games", In Computer Games and Instruction, S. Tobias, Editor. 2011, Information Age Publishing: Charlotte, NC. p. 281-306.
[5] O'Neil, H.F.,Wainess, R, Baker E.L., "Classification of learning outcomes: Evidence from the computer games literature". The Curriculum Journal, 2005. 16(4): p. 455-474. http://dx.doi.org/10.1080/09585170500384529
[6] Ke, F., "A qualitative meta-analysis of computer games as learning tools. Handbook of research on effective electronic gaming in education", Hershey: IGI, 2009. 1: p. 1-32. http://dx.doi.org/10.4018/978-1-59904-808-6.ch001
[7] Brom, C., et al., "Turning high-schools into laboratories? Lessons learnt from studies of instructional effectiveness of digital games in the curricular schooling system", in E-Learning and Games for Training, Education, Health and Sports, Berlin-Heidelberg: Springer, 2012, p. 41-53.
[8] Hays, R.T., "The effectiveness of instructional games: a literature review and discussion". 2005.
[9] Stewart, J., et al., "The Potential of Digital Games for Empowerment and Social Inclusion of Groups at Risk of Social and Economic Exclusion: Evidence and Opportunity for Policy", Institute for Prospective and Technological Studies, Joint Research Centre, 2013.
[10] Nunez Castellar, E., et al., "Improving arithmetic skills through gameplay: assessment of the effectiveness of an educational game in terms of cognitive and affective learning outcomes". Information sciences, 246, 19-31, 2013.
[11] Yip, F.W.M. and. Kwan A.C.M, "Online vocabulary games as a tool for teaching and learning English vocabularyù2. Educational Media International, 43(3): p. 233-249, 2006. http://dx.doi.org/10.1080/09523980600641445
[12] Kretschmann, R., "Digital Sport-Management Games and Their Contribution to Prospective Sport-Managers’ Competence Development". Advances in Physical Education,. 2(4): p. 179-186, 2012. http://dx.doi.org/10.4236/ape.2012.24031
[13] Corsi, T.M., et al., "The real-time global supply chain game: New educational tool for developing supply chain management professionals". Transportation Journal, p. 61-73, 2006.
[14] Neys, J., et al., "Poverty is not a game: behavioral changes and long term effects after playing PING". In 13th annual conference on the International Speech Communication Association. Portland, 2012.
[15] Baranowski, T., et al., "Playing for real: video games and stories for health-related behavior change". American journal of preventive medicine, 34(1): p. 74, 2008. http://dx.doi.org/10.1016/j.amepre.2007.09.027
[16] Korteling, J.E., et al., "Transfer of Gaming: transfer of training in serious gaming": TNO innovation for life, 2011.
[17] Gunter, G.A., Kenny, R.F., and Vick, E.H., "A case for a formal design paradigm for serious games." The Journal of the International Digital Media and Arts Association, 3(1): p. 93-105, 2006.
[18] Kozma, R.B., "Will Media Influence Learning? Reframing the Debate". Educational Technology Research and Development, 42(2): p. 7-19, 1994. http://dx.doi.org/10.1007/BF02299087
[19] Ryan, R.M. and Deci, E.L., "Intrinsic and extrinsic motivations: Classic definitions and new directions.", Contemporary educational psychology,. 25(1): p. 54-67, 2000. http://dx.doi.org/10.1006/ceps.1999.1020
[20] Garris, R., Ahlers, R. and Driskell J.E., "Games, Motivation, and Learning: A Research and Practice Model.", Simulation & Gaming. 33(4): p. 441-467, 2002. http://dx.doi.org/10.1177/1046878102238607
[21] Ryan, R.M. and Deci, E.L., "Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being.", American psychologist, 2000. 55(1): p. 68. http://dx.doi.org/10.1037/0003-066X.55.1.68
[22] Wilkes, M. and Bligh, J., "Evaluating educational interventions.", BMJ: British Medical Journal,. 318 (7193): p. 1269, 1999. http://dx.doi.org/10.1136/bmj.318.7193.1269
[23] Calder, J., "Programme evaluation and quality: A comprehensive guide to setting up an evaluation system": London: Routledge, 2013.
[24] Taras, M., "Assessment–summative and formative–some theoretical reflections". British Journal of Educational Studies, 53(4): p. 466-478, 2005. http://dx.doi.org/10.1111/j.1467-8527.2005.00307.x
[25] Hutchinson, L., "Evaluating and researching the effectiveness of educational interventions.", BMJ: British Medical Journal, 318 (7193): p. 1267, 1999. http://dx.doi.org/10.1136/bmj.318.7193.1267
[26] Hainey, T., "Using Games-Based Learning to Teach Requirements Collection and Analysis at Tertiary Education Level", PhD thesis, University of the West of Scotland, 2010.
[27] Wouters, P., van der Spek, E. and Van Oostendorp, H., "Current practices in serious game research: A review from a learning outcomes perspective.", Games-based learning advancements for multi-sensory human computer interfaces: techniques and effective practices, Hershey: IGI, p. 232-250, 2009. http://dx.doi.org/10.4018/978-1-60566-360-9.ch014
[28] Shute, V.J., Rieber, L. and Van Eck R., "Games... and... learning. Trends and issues in instructional design and technology", 3, 2011.
[29] Shute, V.J., "Stealth assessment in computer-based games to support learning. Computer games and instruction", 55(2): p. 503-524, 2011.
[30] Higgins, J.P., Green, S. and Collaboration, C., "Cochrane handbook for systematic reviews of interventions". Vol. 5,. Wiley Online Library, 2008. http://dx.doi.org/10.1002/9780470712184
[31] Kraiger, K., Ford, J.K., and. Salas E, "Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation.", Journal of applied psychology, 78(2): p. 311, 1993. [28] Shute, V.J., Rieber, L. and Van Eck R., "Games... and... learning. Trends and issues in instructional design and technology", 3, 2011.
[29] Shute, V.J., "Stealth assessment in computer-based games to support learning. Computer games and instruction", 55(2): p. 503-524, 2011.

[30] Higgins, J.P., Green, S. and Collaboration, C., "Cochrane handbook for systematic reviews of interventions". Vol. 5, Wiley Online Library, 2008.
http://dx.doi.org/10.1002/9780470712184
[31] Kraiger, K., Ford, J.K., and. Salas E, "Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation.", Journal of applied psychology, 78(2): p. 311, 1993.
http://dx.doi.org/10.1037/0021-9010.78.2.311
[32] Campbell, D.T., Stanley, J.C. and. Gage, N.L, "Experimental and quasi-experimental designs for research", Boston: Houghton Mifflin, 1963.
[33] Yang, Y.-T.C., "Building virtual cities, inspiring intelligent citizens: Digital games for developing students’ problem solving and learning motivation". Computers & Education, 59(2): p. 365-377, 2012. http://dx.doi.org/10.1016/j.compedu.2012.01.012
[34] Miller, D.J. and Robertson, D.P., "Using a games console in the primary classroom: Effects of ‘Brain Training’ programme on computation and self-esteem", British Journal of Educational Technology, 41(2): p. 242-255, 2010. http://dx.doi.org/10.1111/j.1467-8535.2008.00918.x
[35] Miller, D.J. and Robertson, D.P., "Educational benefits of using game consoles in a primary classroom: A randomised controlled trial.", British Journal of Educational Technology, 42(5): p. 850-864, 2011. http://dx.doi.org/10.1111/j.1467-8535.2010.01114.x
[36] Schønau-Fog, H. and Bjørner, T., "“Sure, I Would Like to Continue” A Method for Mapping the Experience of Engagement in Video Games.", Bulletin of Science, Technology & Society,. 32(5): p. 405-412, 2012. http://dx.doi.org/10.1177/0270467612469068
[37] Singer, J.D. and. Willett, J.B., "Applied longitudinal data analysis: Modeling change and event occurrence.", Oxford university press, 2003.
[38] Dimitrov, D.M, Rumrill, J., Phillip D, "Pretest-posttest designs and measurement of change.", Work: A Journal of Prevention, Assessment and Rehabilitation, 20(2): p. 159-165, p. 2003.
[39] Leary, M.R., "Introduction to behavioral research methods.", Brooks/Cole Pacific Grove, CA, 1995.
[40] Serrano-Laguna, Á., et al., "Learning Analytics and Educational Games: Lessons Learned from Practical Experience.", In Games and Learning Alliance Conference. Paris., 2013.
[41] Joy, E.H. and Garcia, F.E, "Measuring Learning Effectiveness: A New Look at No-Significant-Difference Findings", JALN,. 4(1): p. 33-39, 2000.
[42] Crawford, J., Stewart, L. and Moore J., "Demonstration of savings on the AVLT and development of a parallel form.", Journal of Clinical and Experimental Neuropsychology, 11(6): p. 975-981, 1989. http://dx.doi.org/10.1080/01688638908400950
[43] Anastasi, A., "Differential psychology: Individual and group differences in behavior.", London: Macmillan, 1961.
[44] Popper, K., "Science: conjectures and refutations. Readings in the Philosophy of Science: From Positivism to Postmodernism", p. 9-13, 2000.
Published
2014-06-19
How to Cite
All, A., Nunez Castellar, E. P., & Van Looy, J. (2014). Measuring Effectiveness in Digital Game-Based Learning: A Methodological Review. International Journal of Serious Games, 1(2). https://doi.org/10.17083/ijsg.v1i2.18
Section
Articles

Most read articles by the same author(s)

Obs.: This plugin requires at least one statistics/report plugin to be enabled. If your statistics plugins provide more than one metric then please also select a main metric on the admin's site settings page and/or on the journal manager's settings pages.