Metrics Feedback Cycle: measuring and improving user engagement in gamified eLearning systems
Keywords:gamification, eLearning, metrics
This paper presents the identification, design and implementation of a set of metrics of user engagement in a gamified eLearning application. The 'Metrics Feedback Cycle' (MFC) is introduced as a formal process prescribing the iterative evaluation and improvement of application-wide engagement, using data collected from metrics as input to improve related engagement features. This framework was showcased using a gamified eLearning application as a case study. In this paper, we designed a prototype and tested it with thirty-six (N=36) students to validate the effectiveness of the MFC. The analysis and interpretation of metrics data shows that the gamification features had a positive effect on user engagement, and helped identify areas in which this could be improved. We conclude that the MFC has applications in gamified systems that seek to maximise engagement by iteratively evaluating implemented features against a set of evolving metrics.
 Hamari, H., Koivisto, J., Sarsa, J., “Does Gamification Work? - A Literature Review of Empirical Studies on Gamification,” in System Sciences (HICSS), 2014 47th Hawaii International Conference on, 2014, pp. 3025–3034. https://doi.org/10.1109/HICSS.2014.377
 Dicheva, D., Dichev, C., Agre, G. and Angelova, G., “Gamification in Education: A Systematic Mapping Study,” J. Educ. Technol. Soc., vol. 18, pp. 75–88, 2015.
 Deterding, S., Sicart, M., Nacke, L., O’Hara, K. and Dixon D., “Gamification. using game-design elements in non-gaming contexts,” in the 2011 annual conference extended abstracts, 2011, pp. 2425–2428. https://doi.org/10.1145/1979742.1979575
 Deterding, S., “Situated motivational affordances of game elements : A conceptual model,” in CHI2011, 2011, pp. 3–6.
 G. Zichermann and C. Cunningham, “Gamification by Design: Implementing Game Mechanics in Web and Mobile Apps,” in Gamification by Design: Implementing Game Mechanics in Web and Mobile Apps, O’Reilly Media, Inc, 2011, p. 208.
 O’Brien, H. L and Toms, E. G., “What is user engagement? A Conceptual Framework for defining user engagement with technology,” J. Am. Soc. Inf. Sci. Technol., vol. 59, no. 6, pp. 938–955, 2008. https://doi.org/10.1002/asi.20801
 AlMarshedi, A., Wanick, V., Ranchhod, A., and others, “SGI: A framework for increasing the sustainability of gamification impact,” Int. J. Infonomics, vol. 8, no. 1/2, pp. 1044–1052, 2015. https://doi.org/10.20533/iji.1742.4712.2015.0123
 Kapp, K. M., “The Gamification of Learning and Instruction: Game-based Methods and Strategies for Training and Education,” in The Gamification of Learning and Instruction: Game-based Methods and Strategies for Training and Education, San Francisco: John Wiley & Sons, 2012, pp. 1–22.
 I. R. M. Association, Gamification: Concepts, Methodologies, Tools, and Applications, 1st ed. Hershey, PA, USA: IGI Global, 2015.
 Gee, P. J., “Good video games and good learning,” in Phi Kappa Phi Forum, 2005, vol. 85, no. 2, p. 33.
 Squire, D., “‘All I Know I Learned from Zelda’: Immersive Gaming and Learning and Why The Legend of Zelda Is a Perfect Learning Game,” in Learning to Play: Exploring the Future of Education with Video Games, M. Khine Swe, Ed. Peter Lang, 2011, pp. 167--176.
 Deterding, S. “The Lens of Intrinsic Skill Atoms: A Method for Gameful Design,” Human-Computer Interact., vol. 30, no. 3–4, pp. 294–335, 2015. https://doi.org/10.1080/07370024.2014.993471
 Clifton, B., Advanced Web Metrics with Google Analytics. 2010.
 Xu, Y., “Literature Review on Web Application Gamification and Analytics,” 2011.
 Gee, J. P., “Learning by design: Games as learning machines,” Interact. Educ. Multimed., vol. 8, pp. 15–23, 2004.
 Garris, R., Ahlers, R. and Driskell, J. E. ,“Games, Motivation, and Learning: A Research and Practice Model,” Simul. Gaming, vol. 33, no. 4, pp. 441–467, Dec. 2002. https://doi.org/10.1177/1046878102238607
 Ryan, R. M. and Deci, E. L. “Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions,” Contemp. Educ. Psychol., vol. 25, no. 1, pp. 54–67, 2000. https://doi.org/10.1006/ceps.1999.1020
 AAlmarshedi, A., Wanick, V. , Wills, G., and Ranchhod, A., “Gamification and Behaviour,” in More than just games, Stieglitz et al. (eds.), Ed. Switzerland: Springer International Publishing, 2016, pp. 19–29.
 Squire, K., “Video games in education.,” Games Simul., vol. 2, no. 1, pp. 49–62, 2003.
 Herz, J. C., Joystick Nation: How Videogames Ate Our Quarters, Won Our Hearts, and Rewired Our Minds, 1st ed. Boston, MA: Little, Brown & Co. Inc., 1997.
 Lazzaro, N., “Why We Play Games : Four Keys to More Emotion Without Story,” 2004.
 A. Phippen, L. Sheppard, and S. Furnell, “A practical evaluation of Web analytics,” Internet Res., vol. 14, no. 4, pp. 284--293, 2004. https://doi.org/10.1108/10662240410555306
 O’Donovan, S., Gain, J., Marais, P., Donovan, S. O. and Marais, P., “A Case Study in the Gamification of a University-level Games Development Course,” in ACM International Conference Proceeding Series, 2013, pp. 242–251. https://doi.org/10.1145/2513456.2513469
 Ibanez, M.-B., Di-Serio, A., and Delgado-Kloos, C., “Gamification for Engaging Computer Science Students in Learning Activities: A Case Study,” IEEE Trans. Learn. Technol., vol. 7, no. 3, pp. 291–301, 2014. https://doi.org/10.1109/TLT.2014.2329293
 Heilbrunn, B., Herzig, P. and Schill, A., “Towards gamification analytics-requirements for monitoring and adapting gamification designs,” 44th Annu. Meet. Soc. Comput. Sci. INFORMATICS 2014, vol. P-232, pp. 333–344, 2014.
 Heilbrunn, B., Herzig, P. and Schill, A., “Tools for gamification analytics: A survey,” in Proceedings - 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing, UCC 2014, 2014, pp. 603–608. https://doi.org/10.1109/UCC.2014.93
 Boehm, B. W., “A spiral model of software development and enhancement,” Computer (Long. Beach. Calif)., vol. 21, no. 5, pp. 61–72, 1988. https://doi.org/10.1109/2.59
 Google, “Firebase.” [Online]. Available: https://firebase.google.com/.
 Yee, N., “Ariadne - Understanding MMORPG Addiction,” 2002. .
 Smahel, D., Blinka, L. and Ledabyl, O., “Playing MMORPGs: connections between addiction and identifying with a character.,” Cyber Psychol. Behav., vol. 11, no. 6, pp. 715--718, 2008. https://doi.org/10.1089/cpb.2007.0210
 Wen, D. M.-H., Chang, D. J.-W., Lin, Y.-T., Liang, C.-W., and Yang, S.-Y. “Gamification design for increasing customer purchase intention in a mobile marketing campaign app,” in International Conference on HCI in Business, 2014, pp. 440–448. https://doi.org/10.1007/978-3-319-07293-7_43
 Meder, M., Plumbaum, T., and Hopfgartner, F., “Perceived and actual role of gamification principles,” in Proceedings of the 2013 IEEE/ACM 6th International Conference on Utility and Cloud Computing, 2013, pp. 488–493. https://doi.org/10.1109/UCC.2013.95
 Denny, P. “The Effect of Virtual Achievements on Student Engagement,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2013, pp. 763–772.
 Anderson, A., Huttenlocher, D., Kleinberg, J., and Leskovec, J.“ Steering User Behavior with Badges,” in Proceedings of the 22Nd International Conference on World Wide Web, 2013, pp. 95–106.
 Domínguez, A., Saenz-de-Navarrete, J., de-Marcos, L., Fernández-Sanz, L., Pagés, C. , and Martínez-Herráiz, J.-J., “Gamifying learning experiences: Practical implications and outcomes,” Comput. Educ., vol. 63, pp. 380–392, 2013. https://doi.org/10.1016/j.compedu.2012.12.020
 Pinchbeck, D., “An affordance based model for gameplay,” Break. New Gr. Innov. Games, Play. Pract. Theory Proc. 2009 Digit. Games Res. Assoc. Conf., 2009.
 Salen, K. and Zimmerman, E., Rules of Play: Game Design Fundamentals, vol. 37, no. 5. Massachusetts Institute of Technology, 2004.
 Tondello, G. F., Wehbe, R. R., Diamond, L., Busch, M., Marczewski, A. and Nacke, L. E. “The Gamification User Types Hexad Scale,” Proc. 2016 Annu. Symp. Comput. Interact. Play - CHI Play ’16, pp. 229–243, 2016. https://doi.org/10.1145/2967934.2968082
 Monterrat, B., Lavoué, É. and George, S. ,“Motivation for learning: Adaptive gamification for web-based learning environments,” in CSEDU 2014 - Proceedings of the 6th International Conference on Computer Supported Education, 2014, vol. 1, pp. 117–125.
 Malone, T. W., “What makes things fun to learn? heuristics for designing instructional computer games,” in Proceedings of the 3rd ACM SIGSMALL symposium and the first SIGPC symposium on Small systems - SIGSMALL ’80, 1980, pp. 162–169. https://doi.org/10.1145/800088.802839
 Muntean, C. C. I., “Raising engagement in e-learning through gamification,” 6th Int. Conf. Virtual Learn. ICVL 2011, no. 1, pp. 323–329, 2011.
How to Cite
IJSG copyright information is provided here.