Metrics Feedback Cycle: measuring and improving user engagement in gamified eLearning systems

Adam Atkins, Vanissa Wanick, Gary Wills

Abstract


This paper presents the identification, design and implementation of a set of metrics of user engagement in a gamified eLearning application. The 'Metrics Feedback Cycle' (MFC) is introduced as a formal process prescribing the iterative evaluation and improvement of application-wide engagement, using data collected from metrics as input to improve related engagement features. This framework was showcased using a gamified eLearning application as a case study. In this paper, we designed a prototype and tested it with thirty-six (N=36) students to validate the effectiveness of the MFC. The analysis and interpretation of metrics data shows that the gamification features had a positive effect on user engagement, and helped identify areas in which this could be improved. We conclude that the MFC has applications in gamified systems that seek to maximise engagement by iteratively evaluating implemented features against a set of evolving metrics.


Keywords


gamification; eLearning; metrics

Full Text:

PDF

References


Adkins, S., “The 2016-2021 Global Game-based Learning Market,” in Serious Play Conference, 2016.

Hamari, H., Koivisto, J., Sarsa, J., “Does Gamification Work? - A Literature Review of Empirical Studies on Gamification,” in System Sciences (HICSS), 2014 47th Hawaii International Conference on, 2014, pp. 3025–3034. https://doi.org/10.1109/HICSS.2014.377

Dicheva, D., Dichev, C., Agre, G. and Angelova, G., “Gamification in Education: A Systematic Mapping Study,” J. Educ. Technol. Soc., vol. 18, pp. 75–88, 2015.

Deterding, S., Sicart, M., Nacke, L., O’Hara, K. and Dixon D., “Gamification. using game-design elements in non-gaming contexts,” in the 2011 annual conference extended abstracts, 2011, pp. 2425–2428. https://doi.org/10.1145/1979742.1979575

Deterding, S., “Situated motivational affordances of game elements : A conceptual model,” in CHI2011, 2011, pp. 3–6.

G. Zichermann and C. Cunningham, “Gamification by Design: Implementing Game Mechanics in Web and Mobile Apps,” in Gamification by Design: Implementing Game Mechanics in Web and Mobile Apps, O’Reilly Media, Inc, 2011, p. 208.

O’Brien, H. L and Toms, E. G., “What is user engagement? A Conceptual Framework for defining user engagement with technology,” J. Am. Soc. Inf. Sci. Technol., vol. 59, no. 6, pp. 938–955, 2008. https://doi.org/10.1002/asi.20801

AlMarshedi, A., Wanick, V., Ranchhod, A., and others, “SGI: A framework for increasing the sustainability of gamification impact,” Int. J. Infonomics, vol. 8, no. 1/2, pp. 1044–1052, 2015. https://doi.org/10.20533/iji.1742.4712.2015.0123

Kapp, K. M., “The Gamification of Learning and Instruction: Game-based Methods and Strategies for Training and Education,” in The Gamification of Learning and Instruction: Game-based Methods and Strategies for Training and Education, San Francisco: John Wiley & Sons, 2012, pp. 1–22.

I. R. M. Association, Gamification: Concepts, Methodologies, Tools, and Applications, 1st ed. Hershey, PA, USA: IGI Global, 2015.

Gee, P. J., “Good video games and good learning,” in Phi Kappa Phi Forum, 2005, vol. 85, no. 2, p. 33.

Squire, D., “‘All I Know I Learned from Zelda’: Immersive Gaming and Learning and Why The Legend of Zelda Is a Perfect Learning Game,” in Learning to Play: Exploring the Future of Education with Video Games, M. Khine Swe, Ed. Peter Lang, 2011, pp. 167--176.

Deterding, S. “The Lens of Intrinsic Skill Atoms: A Method for Gameful Design,” Human-Computer Interact., vol. 30, no. 3–4, pp. 294–335, 2015. https://doi.org/10.1080/07370024.2014.993471

Clifton, B., Advanced Web Metrics with Google Analytics. 2010.

Xu, Y., “Literature Review on Web Application Gamification and Analytics,” 2011.

Gee, J. P., “Learning by design: Games as learning machines,” Interact. Educ. Multimed., vol. 8, pp. 15–23, 2004.

Garris, R., Ahlers, R. and Driskell, J. E. ,“Games, Motivation, and Learning: A Research and Practice Model,” Simul. Gaming, vol. 33, no. 4, pp. 441–467, Dec. 2002. https://doi.org/10.1177/1046878102238607

Ryan, R. M. and Deci, E. L. “Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions,” Contemp. Educ. Psychol., vol. 25, no. 1, pp. 54–67, 2000. https://doi.org/10.1006/ceps.1999.1020

AAlmarshedi, A., Wanick, V. , Wills, G., and Ranchhod, A., “Gamification and Behaviour,” in More than just games, Stieglitz et al. (eds.), Ed. Switzerland: Springer International Publishing, 2016, pp. 19–29.

Squire, K., “Video games in education.,” Games Simul., vol. 2, no. 1, pp. 49–62, 2003.

Herz, J. C., Joystick Nation: How Videogames Ate Our Quarters, Won Our Hearts, and Rewired Our Minds, 1st ed. Boston, MA: Little, Brown & Co. Inc., 1997.

Lazzaro, N., “Why We Play Games : Four Keys to More Emotion Without Story,” 2004.

A. Phippen, L. Sheppard, and S. Furnell, “A practical evaluation of Web analytics,” Internet Res., vol. 14, no. 4, pp. 284--293, 2004. https://doi.org/10.1108/10662240410555306

O’Donovan, S., Gain, J., Marais, P., Donovan, S. O. and Marais, P., “A Case Study in the Gamification of a University-level Games Development Course,” in ACM International Conference Proceeding Series, 2013, pp. 242–251. https://doi.org/10.1145/2513456.2513469

Ibanez, M.-B., Di-Serio, A., and Delgado-Kloos, C., “Gamification for Engaging Computer Science Students in Learning Activities: A Case Study,” IEEE Trans. Learn. Technol., vol. 7, no. 3, pp. 291–301, 2014. https://doi.org/10.1109/TLT.2014.2329293

Heilbrunn, B., Herzig, P. and Schill, A., “Towards gamification analytics-requirements for monitoring and adapting gamification designs,” 44th Annu. Meet. Soc. Comput. Sci. INFORMATICS 2014, vol. P-232, pp. 333–344, 2014.

Heilbrunn, B., Herzig, P. and Schill, A., “Tools for gamification analytics: A survey,” in Proceedings - 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing, UCC 2014, 2014, pp. 603–608. https://doi.org/10.1109/UCC.2014.93

Boehm, B. W., “A spiral model of software development and enhancement,” Computer (Long. Beach. Calif)., vol. 21, no. 5, pp. 61–72, 1988. https://doi.org/10.1109/2.59

Google, “Firebase.” [Online]. Available: https://firebase.google.com/.

Yee, N., “Ariadne - Understanding MMORPG Addiction,” 2002. .

Smahel, D., Blinka, L. and Ledabyl, O., “Playing MMORPGs: connections between addiction and identifying with a character.,” Cyber Psychol. Behav., vol. 11, no. 6, pp. 715--718, 2008. https://doi.org/10.1089/cpb.2007.0210

Wen, D. M.-H., Chang, D. J.-W., Lin, Y.-T., Liang, C.-W., and Yang, S.-Y. “Gamification design for increasing customer purchase intention in a mobile marketing campaign app,” in International Conference on HCI in Business, 2014, pp. 440–448. https://doi.org/10.1007/978-3-319-07293-7_43

Meder, M., Plumbaum, T., and Hopfgartner, F., “Perceived and actual role of gamification principles,” in Proceedings of the 2013 IEEE/ACM 6th International Conference on Utility and Cloud Computing, 2013, pp. 488–493. https://doi.org/10.1109/UCC.2013.95

Denny, P. “The Effect of Virtual Achievements on Student Engagement,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2013, pp. 763–772.

Anderson, A., Huttenlocher, D., Kleinberg, J., and Leskovec, J.“ Steering User Behavior with Badges,” in Proceedings of the 22Nd International Conference on World Wide Web, 2013, pp. 95–106.

Domínguez, A., Saenz-de-Navarrete, J., de-Marcos, L., Fernández-Sanz, L., Pagés, C. , and Martínez-Herráiz, J.-J., “Gamifying learning experiences: Practical implications and outcomes,” Comput. Educ., vol. 63, pp. 380–392, 2013. https://doi.org/10.1016/j.compedu.2012.12.020

Pinchbeck, D., “An affordance based model for gameplay,” Break. New Gr. Innov. Games, Play. Pract. Theory Proc. 2009 Digit. Games Res. Assoc. Conf., 2009.

Salen, K. and Zimmerman, E., Rules of Play: Game Design Fundamentals, vol. 37, no. 5. Massachusetts Institute of Technology, 2004.

Tondello, G. F., Wehbe, R. R., Diamond, L., Busch, M., Marczewski, A. and Nacke, L. E. “The Gamification User Types Hexad Scale,” Proc. 2016 Annu. Symp. Comput. Interact. Play - CHI Play ’16, pp. 229–243, 2016. https://doi.org/10.1145/2967934.2968082

Monterrat, B., Lavoué, É. and George, S. ,“Motivation for learning: Adaptive gamification for web-based learning environments,” in CSEDU 2014 - Proceedings of the 6th International Conference on Computer Supported Education, 2014, vol. 1, pp. 117–125.

Malone, T. W., “What makes things fun to learn? heuristics for designing instructional computer games,” in Proceedings of the 3rd ACM SIGSMALL symposium and the first SIGPC symposium on Small systems - SIGSMALL ’80, 1980, pp. 162–169. https://doi.org/10.1145/800088.802839

Muntean, C. C. I., “Raising engagement in e-learning through gamification,” 6th Int. Conf. Virtual Learn. ICVL 2011, no. 1, pp. 323–329, 2011.




DOI: http://dx.doi.org/10.17083/ijsg.v4i4.192

Refbacks

  • There are currently no refbacks.





Serious Games Society


Creative Commons LicenseThe International Journal of Serious Games (IJSG) by Serious Games Society is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.