Metrics Feedback Cycle: measuring and improving user engagement in gamified eLearning systems
DOI:
https://doi.org/10.17083/ijsg.v4i4.192Keywords:
gamification, eLearning, metricsAbstract
This paper presents the identification, design and implementation of a set of metrics of user engagement in a gamified eLearning application. The 'Metrics Feedback Cycle' (MFC) is introduced as a formal process prescribing the iterative evaluation and improvement of application-wide engagement, using data collected from metrics as input to improve related engagement features. This framework was showcased using a gamified eLearning application as a case study. In this paper, we designed a prototype and tested it with thirty-six (N=36) students to validate the effectiveness of the MFC. The analysis and interpretation of metrics data shows that the gamification features had a positive effect on user engagement, and helped identify areas in which this could be improved. We conclude that the MFC has applications in gamified systems that seek to maximise engagement by iteratively evaluating implemented features against a set of evolving metrics.
References
[1] Adkins, S., “The 2016-2021 Global Game-based Learning Market,” in Serious Play Conference, 2016.
[2] Hamari, H., Koivisto, J., Sarsa, J., “Does Gamification Work? - A Literature Review of Empirical Studies on Gamification,” in System Sciences (HICSS), 2014 47th Hawaii International Conference on, 2014, pp. 3025–3034. https://doi.org/10.1109/HICSS.2014.377
[3] Dicheva, D., Dichev, C., Agre, G. and Angelova, G., “Gamification in Education: A Systematic Mapping Study,” J. Educ. Technol. Soc., vol. 18, pp. 75–88, 2015.
[4] Deterding, S., Sicart, M., Nacke, L., O’Hara, K. and Dixon D., “Gamification. using game-design elements in non-gaming contexts,” in the 2011 annual conference extended abstracts, 2011, pp. 2425–2428. https://doi.org/10.1145/1979742.1979575
[5] Deterding, S., “Situated motivational affordances of game elements?: A conceptual model,” in CHI2011, 2011, pp. 3–6.
[6] G. Zichermann and C. Cunningham, “Gamification by Design: Implementing Game Mechanics in Web and Mobile Apps,” in Gamification by Design: Implementing Game Mechanics in Web and Mobile Apps, O’Reilly Media, Inc, 2011, p. 208.
[7] O’Brien, H. L and Toms, E. G., “What is user engagement? A Conceptual Framework for defining user engagement with technology,” J. Am. Soc. Inf. Sci. Technol., vol. 59, no. 6, pp. 938–955, 2008. https://doi.org/10.1002/asi.20801
[8] AlMarshedi, A., Wanick, V., Ranchhod, A., and others, “SGI: A framework for increasing the sustainability of gamification impact,” Int. J. Infonomics, vol. 8, no. 1/2, pp. 1044–1052, 2015. https://doi.org/10.20533/iji.1742.4712.2015.0123
[9] Kapp, K. M., “The Gamification of Learning and Instruction: Game-based Methods and Strategies for Training and Education,” in The Gamification of Learning and Instruction: Game-based Methods and Strategies for Training and Education, San Francisco: John Wiley & Sons, 2012, pp. 1–22.
[10] I. R. M. Association, Gamification: Concepts, Methodologies, Tools, and Applications, 1st ed. Hershey, PA, USA: IGI Global, 2015.
[11] Gee, P. J., “Good video games and good learning,” in Phi Kappa Phi Forum, 2005, vol. 85, no. 2, p. 33.
[12] Squire, D., “‘All I Know I Learned from Zelda’: Immersive Gaming and Learning and Why The Legend of Zelda Is a Perfect Learning Game,” in Learning to Play: Exploring the Future of Education with Video Games, M. Khine Swe, Ed. Peter Lang, 2011, pp. 167--176.
[13] Deterding, S. “The Lens of Intrinsic Skill Atoms: A Method for Gameful Design,” Human-Computer Interact., vol. 30, no. 3–4, pp. 294–335, 2015. https://doi.org/10.1080/07370024.2014.993471
[14] Clifton, B., Advanced Web Metrics with Google Analytics. 2010.
[15] Xu, Y., “Literature Review on Web Application Gamification and Analytics,” 2011.
[16] Gee, J. P., “Learning by design: Games as learning machines,” Interact. Educ. Multimed., vol. 8, pp. 15–23, 2004.
[17] Garris, R., Ahlers, R. and Driskell, J. E. ,“Games, Motivation, and Learning: A Research and Practice Model,” Simul. Gaming, vol. 33, no. 4, pp. 441–467, Dec. 2002. https://doi.org/10.1177/1046878102238607
[18] Ryan, R. M. and Deci, E. L. “Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions,” Contemp. Educ. Psychol., vol. 25, no. 1, pp. 54–67, 2000. https://doi.org/10.1006/ceps.1999.1020
[19] AAlmarshedi, A., Wanick, V. , Wills, G., and Ranchhod, A., “Gamification and Behaviour,” in More than just games, Stieglitz et al. (eds.), Ed. Switzerland: Springer International Publishing, 2016, pp. 19–29.
[20] Squire, K., “Video games in education.,” Games Simul., vol. 2, no. 1, pp. 49–62, 2003.
[21] Herz, J. C., Joystick Nation: How Videogames Ate Our Quarters, Won Our Hearts, and Rewired Our Minds, 1st ed. Boston, MA: Little, Brown & Co. Inc., 1997.
[22] Lazzaro, N., “Why We Play Games?: Four Keys to More Emotion Without Story,” 2004.
[23] A. Phippen, L. Sheppard, and S. Furnell, “A practical evaluation of Web analytics,” Internet Res., vol. 14, no. 4, pp. 284--293, 2004. https://doi.org/10.1108/10662240410555306
[24] O’Donovan, S., Gain, J., Marais, P., Donovan, S. O. and Marais, P., “A Case Study in the Gamification of a University-level Games Development Course,” in ACM International Conference Proceeding Series, 2013, pp. 242–251. https://doi.org/10.1145/2513456.2513469
[25] Ibanez, M.-B., Di-Serio, A., and Delgado-Kloos, C., “Gamification for Engaging Computer Science Students in Learning Activities: A Case Study,” IEEE Trans. Learn. Technol., vol. 7, no. 3, pp. 291–301, 2014. https://doi.org/10.1109/TLT.2014.2329293
[26] Heilbrunn, B., Herzig, P. and Schill, A., “Towards gamification analytics-requirements for monitoring and adapting gamification designs,” 44th Annu. Meet. Soc. Comput. Sci. INFORMATICS 2014, vol. P-232, pp. 333–344, 2014.
[27] Heilbrunn, B., Herzig, P. and Schill, A., “Tools for gamification analytics: A survey,” in Proceedings - 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing, UCC 2014, 2014, pp. 603–608. https://doi.org/10.1109/UCC.2014.93
[28] Boehm, B. W., “A spiral model of software development and enhancement,” Computer (Long. Beach. Calif)., vol. 21, no. 5, pp. 61–72, 1988. https://doi.org/10.1109/2.59
[29] Google, “Firebase.” [Online]. Available: https://firebase.google.com/.
[30] Yee, N., “Ariadne - Understanding MMORPG Addiction,” 2002. .
[31] Smahel, D., Blinka, L. and Ledabyl, O., “Playing MMORPGs: connections between addiction and identifying with a character.,” Cyber Psychol. Behav., vol. 11, no. 6, pp. 715--718, 2008. https://doi.org/10.1089/cpb.2007.0210
[32] Wen, D. M.-H., Chang, D. J.-W., Lin, Y.-T., Liang, C.-W., and Yang, S.-Y. “Gamification design for increasing customer purchase intention in a mobile marketing campaign app,” in International Conference on HCI in Business, 2014, pp. 440–448. https://doi.org/10.1007/978-3-319-07293-7_43
[33] Meder, M., Plumbaum, T., and Hopfgartner, F., “Perceived and actual role of gamification principles,” in Proceedings of the 2013 IEEE/ACM 6th International Conference on Utility and Cloud Computing, 2013, pp. 488–493. https://doi.org/10.1109/UCC.2013.95
[34] Denny, P. “The Effect of Virtual Achievements on Student Engagement,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2013, pp. 763–772.
[35] Anderson, A., Huttenlocher, D., Kleinberg, J., and Leskovec, J.“ Steering User Behavior with Badges,” in Proceedings of the 22Nd International Conference on World Wide Web, 2013, pp. 95–106.
[36] Domínguez, A., Saenz-de-Navarrete, J., de-Marcos, L., Fernández-Sanz, L., Pagés, C. , and Martínez-Herráiz, J.-J., “Gamifying learning experiences: Practical implications and outcomes,” Comput. Educ., vol. 63, pp. 380–392, 2013. https://doi.org/10.1016/j.compedu.2012.12.020
[37] Pinchbeck, D., “An affordance based model for gameplay,” Break. New Gr. Innov. Games, Play. Pract. Theory Proc. 2009 Digit. Games Res. Assoc. Conf., 2009.
[38] Salen, K. and Zimmerman, E., Rules of Play: Game Design Fundamentals, vol. 37, no. 5. Massachusetts Institute of Technology, 2004.
[39] Tondello, G. F., Wehbe, R. R., Diamond, L., Busch, M., Marczewski, A. and Nacke, L. E. “The Gamification User Types Hexad Scale,” Proc. 2016 Annu. Symp. Comput. Interact. Play - CHI Play ’16, pp. 229–243, 2016. https://doi.org/10.1145/2967934.2968082
[40] Monterrat, B., Lavoué, É. and George, S. ,“Motivation for learning: Adaptive gamification for web-based learning environments,” in CSEDU 2014 - Proceedings of the 6th International Conference on Computer Supported Education, 2014, vol. 1, pp. 117–125.
[41] Malone, T. W., “What makes things fun to learn? heuristics for designing instructional computer games,” in Proceedings of the 3rd ACM SIGSMALL symposium and the first SIGPC symposium on Small systems - SIGSMALL ’80, 1980, pp. 162–169. https://doi.org/10.1145/800088.802839
[42] Muntean, C. C. I., “Raising engagement in e-learning through gamification,” 6th Int. Conf. Virtual Learn. ICVL 2011, no. 1, pp. 323–329, 2011.
[2] Hamari, H., Koivisto, J., Sarsa, J., “Does Gamification Work? - A Literature Review of Empirical Studies on Gamification,” in System Sciences (HICSS), 2014 47th Hawaii International Conference on, 2014, pp. 3025–3034. https://doi.org/10.1109/HICSS.2014.377
[3] Dicheva, D., Dichev, C., Agre, G. and Angelova, G., “Gamification in Education: A Systematic Mapping Study,” J. Educ. Technol. Soc., vol. 18, pp. 75–88, 2015.
[4] Deterding, S., Sicart, M., Nacke, L., O’Hara, K. and Dixon D., “Gamification. using game-design elements in non-gaming contexts,” in the 2011 annual conference extended abstracts, 2011, pp. 2425–2428. https://doi.org/10.1145/1979742.1979575
[5] Deterding, S., “Situated motivational affordances of game elements?: A conceptual model,” in CHI2011, 2011, pp. 3–6.
[6] G. Zichermann and C. Cunningham, “Gamification by Design: Implementing Game Mechanics in Web and Mobile Apps,” in Gamification by Design: Implementing Game Mechanics in Web and Mobile Apps, O’Reilly Media, Inc, 2011, p. 208.
[7] O’Brien, H. L and Toms, E. G., “What is user engagement? A Conceptual Framework for defining user engagement with technology,” J. Am. Soc. Inf. Sci. Technol., vol. 59, no. 6, pp. 938–955, 2008. https://doi.org/10.1002/asi.20801
[8] AlMarshedi, A., Wanick, V., Ranchhod, A., and others, “SGI: A framework for increasing the sustainability of gamification impact,” Int. J. Infonomics, vol. 8, no. 1/2, pp. 1044–1052, 2015. https://doi.org/10.20533/iji.1742.4712.2015.0123
[9] Kapp, K. M., “The Gamification of Learning and Instruction: Game-based Methods and Strategies for Training and Education,” in The Gamification of Learning and Instruction: Game-based Methods and Strategies for Training and Education, San Francisco: John Wiley & Sons, 2012, pp. 1–22.
[10] I. R. M. Association, Gamification: Concepts, Methodologies, Tools, and Applications, 1st ed. Hershey, PA, USA: IGI Global, 2015.
[11] Gee, P. J., “Good video games and good learning,” in Phi Kappa Phi Forum, 2005, vol. 85, no. 2, p. 33.
[12] Squire, D., “‘All I Know I Learned from Zelda’: Immersive Gaming and Learning and Why The Legend of Zelda Is a Perfect Learning Game,” in Learning to Play: Exploring the Future of Education with Video Games, M. Khine Swe, Ed. Peter Lang, 2011, pp. 167--176.
[13] Deterding, S. “The Lens of Intrinsic Skill Atoms: A Method for Gameful Design,” Human-Computer Interact., vol. 30, no. 3–4, pp. 294–335, 2015. https://doi.org/10.1080/07370024.2014.993471
[14] Clifton, B., Advanced Web Metrics with Google Analytics. 2010.
[15] Xu, Y., “Literature Review on Web Application Gamification and Analytics,” 2011.
[16] Gee, J. P., “Learning by design: Games as learning machines,” Interact. Educ. Multimed., vol. 8, pp. 15–23, 2004.
[17] Garris, R., Ahlers, R. and Driskell, J. E. ,“Games, Motivation, and Learning: A Research and Practice Model,” Simul. Gaming, vol. 33, no. 4, pp. 441–467, Dec. 2002. https://doi.org/10.1177/1046878102238607
[18] Ryan, R. M. and Deci, E. L. “Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions,” Contemp. Educ. Psychol., vol. 25, no. 1, pp. 54–67, 2000. https://doi.org/10.1006/ceps.1999.1020
[19] AAlmarshedi, A., Wanick, V. , Wills, G., and Ranchhod, A., “Gamification and Behaviour,” in More than just games, Stieglitz et al. (eds.), Ed. Switzerland: Springer International Publishing, 2016, pp. 19–29.
[20] Squire, K., “Video games in education.,” Games Simul., vol. 2, no. 1, pp. 49–62, 2003.
[21] Herz, J. C., Joystick Nation: How Videogames Ate Our Quarters, Won Our Hearts, and Rewired Our Minds, 1st ed. Boston, MA: Little, Brown & Co. Inc., 1997.
[22] Lazzaro, N., “Why We Play Games?: Four Keys to More Emotion Without Story,” 2004.
[23] A. Phippen, L. Sheppard, and S. Furnell, “A practical evaluation of Web analytics,” Internet Res., vol. 14, no. 4, pp. 284--293, 2004. https://doi.org/10.1108/10662240410555306
[24] O’Donovan, S., Gain, J., Marais, P., Donovan, S. O. and Marais, P., “A Case Study in the Gamification of a University-level Games Development Course,” in ACM International Conference Proceeding Series, 2013, pp. 242–251. https://doi.org/10.1145/2513456.2513469
[25] Ibanez, M.-B., Di-Serio, A., and Delgado-Kloos, C., “Gamification for Engaging Computer Science Students in Learning Activities: A Case Study,” IEEE Trans. Learn. Technol., vol. 7, no. 3, pp. 291–301, 2014. https://doi.org/10.1109/TLT.2014.2329293
[26] Heilbrunn, B., Herzig, P. and Schill, A., “Towards gamification analytics-requirements for monitoring and adapting gamification designs,” 44th Annu. Meet. Soc. Comput. Sci. INFORMATICS 2014, vol. P-232, pp. 333–344, 2014.
[27] Heilbrunn, B., Herzig, P. and Schill, A., “Tools for gamification analytics: A survey,” in Proceedings - 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing, UCC 2014, 2014, pp. 603–608. https://doi.org/10.1109/UCC.2014.93
[28] Boehm, B. W., “A spiral model of software development and enhancement,” Computer (Long. Beach. Calif)., vol. 21, no. 5, pp. 61–72, 1988. https://doi.org/10.1109/2.59
[29] Google, “Firebase.” [Online]. Available: https://firebase.google.com/.
[30] Yee, N., “Ariadne - Understanding MMORPG Addiction,” 2002. .
[31] Smahel, D., Blinka, L. and Ledabyl, O., “Playing MMORPGs: connections between addiction and identifying with a character.,” Cyber Psychol. Behav., vol. 11, no. 6, pp. 715--718, 2008. https://doi.org/10.1089/cpb.2007.0210
[32] Wen, D. M.-H., Chang, D. J.-W., Lin, Y.-T., Liang, C.-W., and Yang, S.-Y. “Gamification design for increasing customer purchase intention in a mobile marketing campaign app,” in International Conference on HCI in Business, 2014, pp. 440–448. https://doi.org/10.1007/978-3-319-07293-7_43
[33] Meder, M., Plumbaum, T., and Hopfgartner, F., “Perceived and actual role of gamification principles,” in Proceedings of the 2013 IEEE/ACM 6th International Conference on Utility and Cloud Computing, 2013, pp. 488–493. https://doi.org/10.1109/UCC.2013.95
[34] Denny, P. “The Effect of Virtual Achievements on Student Engagement,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2013, pp. 763–772.
[35] Anderson, A., Huttenlocher, D., Kleinberg, J., and Leskovec, J.“ Steering User Behavior with Badges,” in Proceedings of the 22Nd International Conference on World Wide Web, 2013, pp. 95–106.
[36] Domínguez, A., Saenz-de-Navarrete, J., de-Marcos, L., Fernández-Sanz, L., Pagés, C. , and Martínez-Herráiz, J.-J., “Gamifying learning experiences: Practical implications and outcomes,” Comput. Educ., vol. 63, pp. 380–392, 2013. https://doi.org/10.1016/j.compedu.2012.12.020
[37] Pinchbeck, D., “An affordance based model for gameplay,” Break. New Gr. Innov. Games, Play. Pract. Theory Proc. 2009 Digit. Games Res. Assoc. Conf., 2009.
[38] Salen, K. and Zimmerman, E., Rules of Play: Game Design Fundamentals, vol. 37, no. 5. Massachusetts Institute of Technology, 2004.
[39] Tondello, G. F., Wehbe, R. R., Diamond, L., Busch, M., Marczewski, A. and Nacke, L. E. “The Gamification User Types Hexad Scale,” Proc. 2016 Annu. Symp. Comput. Interact. Play - CHI Play ’16, pp. 229–243, 2016. https://doi.org/10.1145/2967934.2968082
[40] Monterrat, B., Lavoué, É. and George, S. ,“Motivation for learning: Adaptive gamification for web-based learning environments,” in CSEDU 2014 - Proceedings of the 6th International Conference on Computer Supported Education, 2014, vol. 1, pp. 117–125.
[41] Malone, T. W., “What makes things fun to learn? heuristics for designing instructional computer games,” in Proceedings of the 3rd ACM SIGSMALL symposium and the first SIGPC symposium on Small systems - SIGSMALL ’80, 1980, pp. 162–169. https://doi.org/10.1145/800088.802839
[42] Muntean, C. C. I., “Raising engagement in e-learning through gamification,” 6th Int. Conf. Virtual Learn. ICVL 2011, no. 1, pp. 323–329, 2011.
Downloads
Published
2017-12-19
Issue
Section
Articles
License
IJSG copyright information is provided here.
How to Cite
Metrics Feedback Cycle: measuring and improving user engagement in gamified eLearning systems. (2017). International Journal of Serious Games, 4(4). https://doi.org/10.17083/ijsg.v4i4.192
