Measuring Effectiveness in Digital Game-Based Learning: A Methodological Review.
DOI:
https://doi.org/10.17083/ijsg.v1i2.18Keywords:
Digital Game-Based Learning, Effectiveness assessment, cognitive learning outcomesAbstract
In recent years, a growing number of studies are being conducted into the effectiveness of digital game-based learning (DGBL). Despite this growing interest, there is a lack of sound empirical evidence on the effectiveness of DGBL due to different outcome measures for assessing effectiveness, varying methods of data collection and inconclusive or difficult to interpret results. This has resulted in a need for an overarching methodology for assessing the effectiveness of DGBL. The present study took a first step in this direction by mapping current methods used for assessing the effectiveness of DGBL. Results showed that currently, comparison of results across studies and thus looking at effectiveness of DGBL on a more general level is problematic due to diversity in and suboptimal study designs. Variety in study design relates to three issues, namely different activities that are implemented in the control groups, different measures for assessing the effectiveness of DGBL and the use of different statistical techniques for analyzing learning outcomes. Suboptimal study designs are the result of variables confounding study results. Possible confounds that were brought forward in this review are elements that are added to the game as part of the educational intervention (e.g., required reading, debriefing session), instructor influences and practice effects when using the same test pre- and post-intervention. Lastly, incomplete information on the study design impedes replication of studies and thus falsification of study results.
References
[1] Sawyer, B. and P. Smith, "Taxonomy for Serious Games". Digitalmil, Inc& Serious Games Initiative/Univ. of Central Florida, RETRO Lab, 2008.
[2] Clark, R., "Learning from serious games? Arguments, evidence, and research suggestions". Educational Technology, 2007. 47(3): p. 56-59.
[3] Mayer, I., et al., "A Brief Methodology for Researching and Evaluating Serious Games and Game-Based Learning". In Psychology, Pedagogy, and Assessment in Serious Games, T.C.T.H.E.B.G.B.P. Moreno-Ger, Editor. 2013, ICI Global.
[4 ] Mayer, R., "Multimedia Learning and Games", In Computer Games and Instruction, S. Tobias, Editor. 2011, Information Age Publishing: Charlotte, NC. p. 281-306.
[5] O'Neil, H.F.,Wainess, R, Baker E.L., "Classification of learning outcomes: Evidence from the computer games literature". The Curriculum Journal, 2005. 16(4): p. 455-474. http://dx.doi.org/10.1080/09585170500384529
[6] Ke, F., "A qualitative meta-analysis of computer games as learning tools. Handbook of research on effective electronic gaming in education", Hershey: IGI, 2009. 1: p. 1-32. http://dx.doi.org/10.4018/978-1-59904-808-6.ch001
[7] Brom, C., et al., "Turning high-schools into laboratories? Lessons learnt from studies of instructional effectiveness of digital games in the curricular schooling system", in E-Learning and Games for Training, Education, Health and Sports, Berlin-Heidelberg: Springer, 2012, p. 41-53.
[8] Hays, R.T., "The effectiveness of instructional games: a literature review and discussion". 2005.
[9] Stewart, J., et al., "The Potential of Digital Games for Empowerment and Social Inclusion of Groups at Risk of Social and Economic Exclusion: Evidence and Opportunity for Policy", Institute for Prospective and Technological Studies, Joint Research Centre, 2013.
[10] Nunez Castellar, E., et al., "Improving arithmetic skills through gameplay: assessment of the effectiveness of an educational game in terms of cognitive and affective learning outcomes". Information sciences, 246, 19-31, 2013.
[11] Yip, F.W.M. and. Kwan A.C.M, "Online vocabulary games as a tool for teaching and learning English vocabularyù2. Educational Media International, 43(3): p. 233-249, 2006. http://dx.doi.org/10.1080/09523980600641445
[12] Kretschmann, R., "Digital Sport-Management Games and Their Contribution to Prospective Sport-Managers’ Competence Development". Advances in Physical Education,. 2(4): p. 179-186, 2012. http://dx.doi.org/10.4236/ape.2012.24031
[13] Corsi, T.M., et al., "The real-time global supply chain game: New educational tool for developing supply chain management professionals". Transportation Journal, p. 61-73, 2006.
[14] Neys, J., et al., "Poverty is not a game: behavioral changes and long term effects after playing PING". In 13th annual conference on the International Speech Communication Association. Portland, 2012.
[15] Baranowski, T., et al., "Playing for real: video games and stories for health-related behavior change". American journal of preventive medicine, 34(1): p. 74, 2008. http://dx.doi.org/10.1016/j.amepre.2007.09.027
[16] Korteling, J.E., et al., "Transfer of Gaming: transfer of training in serious gaming": TNO innovation for life, 2011.
[17] Gunter, G.A., Kenny, R.F., and Vick, E.H., "A case for a formal design paradigm for serious games." The Journal of the International Digital Media and Arts Association, 3(1): p. 93-105, 2006.
[18] Kozma, R.B., "Will Media Influence Learning? Reframing the Debate". Educational Technology Research and Development, 42(2): p. 7-19, 1994. http://dx.doi.org/10.1007/BF02299087
[19] Ryan, R.M. and Deci, E.L., "Intrinsic and extrinsic motivations: Classic definitions and new directions.", Contemporary educational psychology,. 25(1): p. 54-67, 2000. http://dx.doi.org/10.1006/ceps.1999.1020
[20] Garris, R., Ahlers, R. and Driskell J.E., "Games, Motivation, and Learning: A Research and Practice Model.", Simulation & Gaming. 33(4): p. 441-467, 2002. http://dx.doi.org/10.1177/1046878102238607
[21] Ryan, R.M. and Deci, E.L., "Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being.", American psychologist, 2000. 55(1): p. 68. http://dx.doi.org/10.1037/0003-066X.55.1.68
[22] Wilkes, M. and Bligh, J., "Evaluating educational interventions.", BMJ: British Medical Journal,. 318 (7193): p. 1269, 1999. http://dx.doi.org/10.1136/bmj.318.7193.1269
[23] Calder, J., "Programme evaluation and quality: A comprehensive guide to setting up an evaluation system": London: Routledge, 2013.
[24] Taras, M., "Assessment–summative and formative–some theoretical reflections". British Journal of Educational Studies, 53(4): p. 466-478, 2005. http://dx.doi.org/10.1111/j.1467-8527.2005.00307.x
[25] Hutchinson, L., "Evaluating and researching the effectiveness of educational interventions.", BMJ: British Medical Journal, 318 (7193): p. 1267, 1999. http://dx.doi.org/10.1136/bmj.318.7193.1267
[26] Hainey, T., "Using Games-Based Learning to Teach Requirements Collection and Analysis at Tertiary Education Level", PhD thesis, University of the West of Scotland, 2010.
[27] Wouters, P., van der Spek, E. and Van Oostendorp, H., "Current practices in serious game research: A review from a learning outcomes perspective.", Games-based learning advancements for multi-sensory human computer interfaces: techniques and effective practices, Hershey: IGI, p. 232-250, 2009. http://dx.doi.org/10.4018/978-1-60566-360-9.ch014
[28] Shute, V.J., Rieber, L. and Van Eck R., "Games... and... learning. Trends and issues in instructional design and technology", 3, 2011.
[29] Shute, V.J., "Stealth assessment in computer-based games to support learning. Computer games and instruction", 55(2): p. 503-524, 2011.
[30] Higgins, J.P., Green, S. and Collaboration, C., "Cochrane handbook for systematic reviews of interventions". Vol. 5,. Wiley Online Library, 2008. http://dx.doi.org/10.1002/9780470712184
[31] Kraiger, K., Ford, J.K., and. Salas E, "Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation.", Journal of applied psychology, 78(2): p. 311, 1993. [28] Shute, V.J., Rieber, L. and Van Eck R., "Games... and... learning. Trends and issues in instructional design and technology", 3, 2011.
[29] Shute, V.J., "Stealth assessment in computer-based games to support learning. Computer games and instruction", 55(2): p. 503-524, 2011.
[30] Higgins, J.P., Green, S. and Collaboration, C., "Cochrane handbook for systematic reviews of interventions". Vol. 5, Wiley Online Library, 2008.
http://dx.doi.org/10.1002/9780470712184
[31] Kraiger, K., Ford, J.K., and. Salas E, "Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation.", Journal of applied psychology, 78(2): p. 311, 1993.
http://dx.doi.org/10.1037/0021-9010.78.2.311
[32] Campbell, D.T., Stanley, J.C. and. Gage, N.L, "Experimental and quasi-experimental designs for research", Boston: Houghton Mifflin, 1963.
[33] Yang, Y.-T.C., "Building virtual cities, inspiring intelligent citizens: Digital games for developing students’ problem solving and learning motivation". Computers & Education, 59(2): p. 365-377, 2012. http://dx.doi.org/10.1016/j.compedu.2012.01.012
[34] Miller, D.J. and Robertson, D.P., "Using a games console in the primary classroom: Effects of ‘Brain Training’ programme on computation and self-esteem", British Journal of Educational Technology, 41(2): p. 242-255, 2010. http://dx.doi.org/10.1111/j.1467-8535.2008.00918.x
[35] Miller, D.J. and Robertson, D.P., "Educational benefits of using game consoles in a primary classroom: A randomised controlled trial.", British Journal of Educational Technology, 42(5): p. 850-864, 2011. http://dx.doi.org/10.1111/j.1467-8535.2010.01114.x
[36] Schønau-Fog, H. and Bjørner, T., "“Sure, I Would Like to Continue” A Method for Mapping the Experience of Engagement in Video Games.", Bulletin of Science, Technology & Society,. 32(5): p. 405-412, 2012. http://dx.doi.org/10.1177/0270467612469068
[37] Singer, J.D. and. Willett, J.B., "Applied longitudinal data analysis: Modeling change and event occurrence.", Oxford university press, 2003.
[38] Dimitrov, D.M, Rumrill, J., Phillip D, "Pretest-posttest designs and measurement of change.", Work: A Journal of Prevention, Assessment and Rehabilitation, 20(2): p. 159-165, p. 2003.
[39] Leary, M.R., "Introduction to behavioral research methods.", Brooks/Cole Pacific Grove, CA, 1995.
[40] Serrano-Laguna, Á., et al., "Learning Analytics and Educational Games: Lessons Learned from Practical Experience.", In Games and Learning Alliance Conference. Paris., 2013.
[41] Joy, E.H. and Garcia, F.E, "Measuring Learning Effectiveness: A New Look at No-Significant-Difference Findings", JALN,. 4(1): p. 33-39, 2000.
[42] Crawford, J., Stewart, L. and Moore J., "Demonstration of savings on the AVLT and development of a parallel form.", Journal of Clinical and Experimental Neuropsychology, 11(6): p. 975-981, 1989. http://dx.doi.org/10.1080/01688638908400950
[43] Anastasi, A., "Differential psychology: Individual and group differences in behavior.", London: Macmillan, 1961.
[44] Popper, K., "Science: conjectures and refutations. Readings in the Philosophy of Science: From Positivism to Postmodernism", p. 9-13, 2000.
[2] Clark, R., "Learning from serious games? Arguments, evidence, and research suggestions". Educational Technology, 2007. 47(3): p. 56-59.
[3] Mayer, I., et al., "A Brief Methodology for Researching and Evaluating Serious Games and Game-Based Learning". In Psychology, Pedagogy, and Assessment in Serious Games, T.C.T.H.E.B.G.B.P. Moreno-Ger, Editor. 2013, ICI Global.
[4 ] Mayer, R., "Multimedia Learning and Games", In Computer Games and Instruction, S. Tobias, Editor. 2011, Information Age Publishing: Charlotte, NC. p. 281-306.
[5] O'Neil, H.F.,Wainess, R, Baker E.L., "Classification of learning outcomes: Evidence from the computer games literature". The Curriculum Journal, 2005. 16(4): p. 455-474. http://dx.doi.org/10.1080/09585170500384529
[6] Ke, F., "A qualitative meta-analysis of computer games as learning tools. Handbook of research on effective electronic gaming in education", Hershey: IGI, 2009. 1: p. 1-32. http://dx.doi.org/10.4018/978-1-59904-808-6.ch001
[7] Brom, C., et al., "Turning high-schools into laboratories? Lessons learnt from studies of instructional effectiveness of digital games in the curricular schooling system", in E-Learning and Games for Training, Education, Health and Sports, Berlin-Heidelberg: Springer, 2012, p. 41-53.
[8] Hays, R.T., "The effectiveness of instructional games: a literature review and discussion". 2005.
[9] Stewart, J., et al., "The Potential of Digital Games for Empowerment and Social Inclusion of Groups at Risk of Social and Economic Exclusion: Evidence and Opportunity for Policy", Institute for Prospective and Technological Studies, Joint Research Centre, 2013.
[10] Nunez Castellar, E., et al., "Improving arithmetic skills through gameplay: assessment of the effectiveness of an educational game in terms of cognitive and affective learning outcomes". Information sciences, 246, 19-31, 2013.
[11] Yip, F.W.M. and. Kwan A.C.M, "Online vocabulary games as a tool for teaching and learning English vocabularyù2. Educational Media International, 43(3): p. 233-249, 2006. http://dx.doi.org/10.1080/09523980600641445
[12] Kretschmann, R., "Digital Sport-Management Games and Their Contribution to Prospective Sport-Managers’ Competence Development". Advances in Physical Education,. 2(4): p. 179-186, 2012. http://dx.doi.org/10.4236/ape.2012.24031
[13] Corsi, T.M., et al., "The real-time global supply chain game: New educational tool for developing supply chain management professionals". Transportation Journal, p. 61-73, 2006.
[14] Neys, J., et al., "Poverty is not a game: behavioral changes and long term effects after playing PING". In 13th annual conference on the International Speech Communication Association. Portland, 2012.
[15] Baranowski, T., et al., "Playing for real: video games and stories for health-related behavior change". American journal of preventive medicine, 34(1): p. 74, 2008. http://dx.doi.org/10.1016/j.amepre.2007.09.027
[16] Korteling, J.E., et al., "Transfer of Gaming: transfer of training in serious gaming": TNO innovation for life, 2011.
[17] Gunter, G.A., Kenny, R.F., and Vick, E.H., "A case for a formal design paradigm for serious games." The Journal of the International Digital Media and Arts Association, 3(1): p. 93-105, 2006.
[18] Kozma, R.B., "Will Media Influence Learning? Reframing the Debate". Educational Technology Research and Development, 42(2): p. 7-19, 1994. http://dx.doi.org/10.1007/BF02299087
[19] Ryan, R.M. and Deci, E.L., "Intrinsic and extrinsic motivations: Classic definitions and new directions.", Contemporary educational psychology,. 25(1): p. 54-67, 2000. http://dx.doi.org/10.1006/ceps.1999.1020
[20] Garris, R., Ahlers, R. and Driskell J.E., "Games, Motivation, and Learning: A Research and Practice Model.", Simulation & Gaming. 33(4): p. 441-467, 2002. http://dx.doi.org/10.1177/1046878102238607
[21] Ryan, R.M. and Deci, E.L., "Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being.", American psychologist, 2000. 55(1): p. 68. http://dx.doi.org/10.1037/0003-066X.55.1.68
[22] Wilkes, M. and Bligh, J., "Evaluating educational interventions.", BMJ: British Medical Journal,. 318 (7193): p. 1269, 1999. http://dx.doi.org/10.1136/bmj.318.7193.1269
[23] Calder, J., "Programme evaluation and quality: A comprehensive guide to setting up an evaluation system": London: Routledge, 2013.
[24] Taras, M., "Assessment–summative and formative–some theoretical reflections". British Journal of Educational Studies, 53(4): p. 466-478, 2005. http://dx.doi.org/10.1111/j.1467-8527.2005.00307.x
[25] Hutchinson, L., "Evaluating and researching the effectiveness of educational interventions.", BMJ: British Medical Journal, 318 (7193): p. 1267, 1999. http://dx.doi.org/10.1136/bmj.318.7193.1267
[26] Hainey, T., "Using Games-Based Learning to Teach Requirements Collection and Analysis at Tertiary Education Level", PhD thesis, University of the West of Scotland, 2010.
[27] Wouters, P., van der Spek, E. and Van Oostendorp, H., "Current practices in serious game research: A review from a learning outcomes perspective.", Games-based learning advancements for multi-sensory human computer interfaces: techniques and effective practices, Hershey: IGI, p. 232-250, 2009. http://dx.doi.org/10.4018/978-1-60566-360-9.ch014
[28] Shute, V.J., Rieber, L. and Van Eck R., "Games... and... learning. Trends and issues in instructional design and technology", 3, 2011.
[29] Shute, V.J., "Stealth assessment in computer-based games to support learning. Computer games and instruction", 55(2): p. 503-524, 2011.
[30] Higgins, J.P., Green, S. and Collaboration, C., "Cochrane handbook for systematic reviews of interventions". Vol. 5,. Wiley Online Library, 2008. http://dx.doi.org/10.1002/9780470712184
[31] Kraiger, K., Ford, J.K., and. Salas E, "Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation.", Journal of applied psychology, 78(2): p. 311, 1993. [28] Shute, V.J., Rieber, L. and Van Eck R., "Games... and... learning. Trends and issues in instructional design and technology", 3, 2011.
[29] Shute, V.J., "Stealth assessment in computer-based games to support learning. Computer games and instruction", 55(2): p. 503-524, 2011.
[30] Higgins, J.P., Green, S. and Collaboration, C., "Cochrane handbook for systematic reviews of interventions". Vol. 5, Wiley Online Library, 2008.
http://dx.doi.org/10.1002/9780470712184
[31] Kraiger, K., Ford, J.K., and. Salas E, "Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation.", Journal of applied psychology, 78(2): p. 311, 1993.
http://dx.doi.org/10.1037/0021-9010.78.2.311
[32] Campbell, D.T., Stanley, J.C. and. Gage, N.L, "Experimental and quasi-experimental designs for research", Boston: Houghton Mifflin, 1963.
[33] Yang, Y.-T.C., "Building virtual cities, inspiring intelligent citizens: Digital games for developing students’ problem solving and learning motivation". Computers & Education, 59(2): p. 365-377, 2012. http://dx.doi.org/10.1016/j.compedu.2012.01.012
[34] Miller, D.J. and Robertson, D.P., "Using a games console in the primary classroom: Effects of ‘Brain Training’ programme on computation and self-esteem", British Journal of Educational Technology, 41(2): p. 242-255, 2010. http://dx.doi.org/10.1111/j.1467-8535.2008.00918.x
[35] Miller, D.J. and Robertson, D.P., "Educational benefits of using game consoles in a primary classroom: A randomised controlled trial.", British Journal of Educational Technology, 42(5): p. 850-864, 2011. http://dx.doi.org/10.1111/j.1467-8535.2010.01114.x
[36] Schønau-Fog, H. and Bjørner, T., "“Sure, I Would Like to Continue” A Method for Mapping the Experience of Engagement in Video Games.", Bulletin of Science, Technology & Society,. 32(5): p. 405-412, 2012. http://dx.doi.org/10.1177/0270467612469068
[37] Singer, J.D. and. Willett, J.B., "Applied longitudinal data analysis: Modeling change and event occurrence.", Oxford university press, 2003.
[38] Dimitrov, D.M, Rumrill, J., Phillip D, "Pretest-posttest designs and measurement of change.", Work: A Journal of Prevention, Assessment and Rehabilitation, 20(2): p. 159-165, p. 2003.
[39] Leary, M.R., "Introduction to behavioral research methods.", Brooks/Cole Pacific Grove, CA, 1995.
[40] Serrano-Laguna, Á., et al., "Learning Analytics and Educational Games: Lessons Learned from Practical Experience.", In Games and Learning Alliance Conference. Paris., 2013.
[41] Joy, E.H. and Garcia, F.E, "Measuring Learning Effectiveness: A New Look at No-Significant-Difference Findings", JALN,. 4(1): p. 33-39, 2000.
[42] Crawford, J., Stewart, L. and Moore J., "Demonstration of savings on the AVLT and development of a parallel form.", Journal of Clinical and Experimental Neuropsychology, 11(6): p. 975-981, 1989. http://dx.doi.org/10.1080/01688638908400950
[43] Anastasi, A., "Differential psychology: Individual and group differences in behavior.", London: Macmillan, 1961.
[44] Popper, K., "Science: conjectures and refutations. Readings in the Philosophy of Science: From Positivism to Postmodernism", p. 9-13, 2000.
Downloads
Additional Files
Published
2014-06-19
Issue
Section
Articles
License
IJSG copyright information is provided here.
How to Cite
Measuring Effectiveness in Digital Game-Based Learning: A Methodological Review. (2014). International Journal of Serious Games, 1(2). https://doi.org/10.17083/ijsg.v1i2.18
