Serious Gaming Analytics: What Students´ Log Files Tell Us about Gaming and Learning
DOI:
https://doi.org/10.17083/ijsg.v1i2.9Keywords:
Serious gaming, learning analytics, logging, behaviour, videoAbstract
In this paper we explore existing log files of the VIBOA environmental policy game. Our aim is to identify relevant player behaviours and performance patterns. The VIBOA game is a 50 hours master level serious game that supports inquiry-based learning: students adopt the role of an environmental consultant in the (fictitious) consultancy agency VIBOA, and have to deal with complex, multi-faceted environmental problems in an academic and methodologically sound way. A sample of 118 master students played the game. We used learning analytics to extract relevant data from the logging and find meaningful patterns and relationships. We observed substantial behavioural variability across students. Correlation analysis suggest a behavioural trade that reflects the rate of “switching” between different game objects or activities. We were able to establish a model that uses switching indicators as predictors for the efficiency of learning. Also we found slight evidence that students who display increased switching behaviours need more time to complete the games. We conclude the paper by critically evaluating our findings, making explicit the limitations of our study and making suggestions for future research that links together learning analytics and serious gaming.References
[1] VandeWalle, D., Brown, S.P., Cron, W.L., Slocum, L.W., “The influence of goal orientation and self-regulation tactics on sales performance: A longitudinal field test”. Journal of Applied Psychology (APA), vol. 84, pp.249–259, 1999. http://dx.doi.org/10.1037/0021-9010.84.2.249
[2] Fisher, S.L., Ford, J.K., “Differential effects of learner effort and goal orientation on two learning outcomes”, Personnel Psychology (Wiley) vol. 51, pp.397–420, 1998. http://dx.doi.org/10.1111/j.1744-6570.1998.tb00731.x
[3] Redeker, C., Punie, Y., Ferrari, A., “eAssessment for 21st century learning and skills”, In Ravenscroft, A., Lindsteadt, S. Kloos, C.D., Hernandez-Leo, D. (eds.). 21st Century Learning for 21st Century Skills. Proceedings of the 7th European Conference on technology-enhanced learning EC-TEL. Springer, Heidelberg, pp.292-305, 2012. http://dx.doi.org/10.1007/978-3-642-33263-0_23
[4] Westera, W., Nadolski, R., Hummel, H., Wopereis, I., “Serious Games for Higher Education: a Framework for Reducing Design Complexity”, Journal of Computer-Assisted Learning (Wiley), vol. 24, no. 5, pp. 420-432, 2008. http://dx.doi.org/10.1111/j.1365-2729.2008.00279.x
[5] Westera, W., Nadolski, R, Hummel, H., “Learning analytics in serious gaming: uncovering the hidden treasury of game log files”, Paper presented at the GALA Serious Gaming Conference October 24-25 in Paris, France, 2013.
[6] Buckingham Shum, S., Ferguson, R., “Social Learning Analytics”, Educational Technology & Society (IFETS), vol. 15, no. 3, pp.3–26, 2012.
[7] Baker, R.S.J.D., Yacef, K., “The state of educational data mining in 2009: A review and future visions”, Journal of Educational Data Mining vol. 1, no. 1, pp.3-17, 2009.
[8] Romero, C., Ventura, S., “Educational data mining: A survey from 1995 to 2005”, Expert Systems with Applications (Elsevier), vol. 33, no. 1, pp.135-146, 2007. http://dx.doi.org/10.1016/j.eswa.2006.04.005
[9] Bienkowski, M., Feng, M., Means, B., “Enhancing Teaching and Learning Through Educational Data Mining and Learning Analytics: An Issue Brief”. SRI International U.S. Department of Education, Office of Educational Technology, Washington, 2012. Retrieved from http://www.ed.gov/edblogs/technology/files/2012/03/edm-la-brief.pdf
[10] Zouaq, A., Jocsimovíc, S., Gasevíc, D., “Ontology Learning to Analyze Research Trends in Learning Analytics Publications”. In Mathieu d'Aquin, M., Dietze, S., Drachsler, H., Herder, E., Taibi. D. (eds), Proceedings of the LAK Data Challenge, held at LAK 2013, the Third Conference on Learning Analytics and Knowledge, Leuven, Belgium, April 9, 2013. Available at http://ceur-ws.org/Vol-974/lakdatachallenge2013_08.pdf
[11] Martin, T., Sherin,B., “Learning Analytics and Computational Techniques for Detecting and Evaluating Patterns in Learning: An Introduction to the Special Issue”, Journal of the Learning Sciences (Taylor & Francis), vol. 22, no. 4, pp.511-520, 2013. DOI: 10.1080/10508406.2013.840466
[12] Arnold, K.E., “Signals: Applying academic analytics”, Educause Quarterly, vol. 33, no. 1, pp.10-15, 2010.. Retrieved from http://www.educause.edu/ero/article/signals-applying-academic-analytics
[13] Pistilli, M., Arnold, K., “Course signals at Purdue: Using learning analytics to increase student success”, 2nd International Conference on Learning Analytics and Knowledge, Vancouver, 2012.
[14] Long, P.D., Siemens, G., “Penetrating the Fog: Analytics in Learning and Education”, Educause Review, vol. 46. No. 5, pp.31-40, 2011. Retrieved from http://www.educause.edu/ero/article/penetrating-fog-analytics-learning-and-education
[15] Cooper, A., Powell, S., MacNeill, S., “ Survey of the State of Analytics in UK Higher and Further Institutions 2013”, White Paper 2013:WP03, Bolton: CETIS. Retrieved from http://publications.cetis.ac.uk/2013/884
[16] Kraan, W., Sherlock, D., “Analytics Tools and Infrastructure. CETIS Analytics Series, vol.1, no.11, 2013, Bolton: JISC/CETIS. Retrieved from http://publications.cetis.ac.uk/2013/535
[17] Hung, J.-L., Hsu, Y.-C., Rice, K., “Integrating Data Mining in Program Evaluation of K-12 Online Education”, Educational Technology & Society (IFETS), vol. 15, no. 3, pp.27–41, 2012.
[18] Greller, W., Drachsler, H., “Translating Learning into Numbers: A Generic Framework for Learning Analytics”, Educational Technology & Society (IFETS), vol. 15, no. 3, pp.42–57, 2012.
[19] Siemens, G., “Learning analytics: A foundation for informed change in higher education”, Educause, slides retrieved from http://www.slideshare.net/gsiemens/learning-analytics-educause
[20] Elias, T., “Learning analytics: Definitions, processes and potentials”, retrieved from http://learninganalytics.net/LearningAnalyticsDefinitionsProcessesPotential.pdf
[21] Campbell, J.P., Oblinger, D.G., “Academic Analytics”, Educause White Paper 2007, retrieved from http://www.educause.edu/ir/library/pdf/PUB6101.pdf
[22] Dyckhoff, A.L., Zielke, D., Bültmann, M., Chatti, M.A., Schroeder, U., “Design and Implementation of a Learning Analytics Toolkit for Teachers”. Educational Technology & Society (IFETS), vol. 15, no. 3, pp.58–76, 2012.
[23] Boyd, D., Crawford, K., “Six Provocations for Big Data, A Decade in Internet Time”, Symposium on the Dynamics of the Internet and Society, Oxford Internet Institute, Oxford, 2012. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1926431
[24] Bluemink,J.,Hämäläinen, R., Manninen, T., Järvelä, S., “Group-level analysis on multiplayer game collaboration: how do the individuals shape the group interaction?”, Interactive Learning Environments (Taylor & Francis), vol. 18, no. 4, pp.365–383, 2010.
[25] Fernández-Gallego, B., Lama, M., Vidal, J.C., Mucientes, M., “Learning Analytics Framework for Educational Virtual Worlds”. In Proceedings of the 2013 International Conference on Virtual and Augmented Reality in Education, Procedia Computer Science (Elsevier), vol. 25 , pp.443-447, 2013. DOI: 10.1016/j.procs.2013.11.056
[26] Koper, R., Tattersall, C., “Learning Design: A handbook on modelling and delivering networked education and training”, Springer, Berlin, 2005.
[27] Gobert, J.D., Sao Pedro, M., Raziuddin, J., Baker, R.S., “From Log Files to Assessment Metrics: Measuring Students' Science Inquiry Skills Using Educational Data Mining”, Journal of the Learning Sciences (Taylor & Francis), vol. 22, no. 4, pp.521-563, 2013. DOI: 10.1080/10508406.2013.837391
[28] Westera, W., Hommes, M.A., Houtmans, M., Kurvers, H.J., “Computer-Supported Training of Psychodiagnostic Skills”, Interactive Learning Environments (Taylor & Francis), vol. 11, no. 3, pp.215-231, 2003.
[29] Kickmeier-Rust, M.D., Albert, D., “Micro Adaptivity: Protecting Immersion in Didactically Adaptive Digital Educational Games”, Journal of Computer Assisted Learning (Wiley),vol. 26, pp.95-105, 2010. http://dx.doi.org/10.1111/j.1365-2729.2009.00332.x
[30] Reese, D.D., Seward, R.J., Tabachnick, B.G., Hitt, B., Harrison, A., McFarland, L., “Timed Report Measures Learning: Game-Based Embedded Assessment”, In Ifenthaler, D., Eseryel, D., Ge, X. (eds.), Assessment in Game-Based Learning: Foundations, Innovations, and Perspectives. Springer, New York, 2013.
[31] Serrano-Laguna, A., Torrente, J., Moreno-Ger, P., Fernández-Manjón, B., “Tracing a Little for Big Improvements: Application of Learning Analytics and Videogames for Student Assessment”, Procedia Computer Science (Elsevier), vol. 15, pp.203-209, 2012. Retrieved from http://www.sciencedirect.com/science/article/pii/S1877050912008344 http://dx.doi.org/10.1016/j.procs.2012.10.072
[32] Shute, V.J., Ventura, M., Bauer, M., Zapata-Rivera, D., “Melding the Power of Serious Games and Embedded Assessment to Monitor and Foster Learning: Flow and Grow”, In Ritterfeld, U., Cody, M., Vorderer, M. (eds.). Serious Games: Mechanisms and Effects, Routledge, New York, pp.295-321, 2009.
[33] Mislevy, R.J., Steinberg, L.S., Almond, R.G., “On the structure of educational assessment”, Measurement: Interdisciplinary Research and Perspective (Taylor& Francis), vol. 10, pp.3-62, 2003.
[34] Pearl, J., “Probabilistic reasoning in intelligent systems: Networks of plausible inference”, Kaufmann, San Mateo CA, Unites States, 1988.
[35] Slootmaker, A., Kurvers, H.J., Hummel, H.G.K., Koper, E.J.R., “Developing scenario-based serious games for complex cognitive skills acquisition: Design, development and evaluation of the EMERGO platform”, in preparation, 2014.
[36] Westera, W., “A didactic framework for audiovisual design”, Journal of Educational Media (Taylor& Francis), vol. 24, no. 2, pp.87-102, 1999.
[37] Koumi, J., “Designing Video and Multimedia for Open and Flexible Learning”, Routledge, London, 2006.
[38] Green, S.B., “How many subjects does it take to do a regression analysis?”, Multivariate Behavioral Research (Taylor& Francis), vol. 26, pp.499–510, 1991. http://dx.doi.org/10.1207/s15327906mbr2603_7
[2] Fisher, S.L., Ford, J.K., “Differential effects of learner effort and goal orientation on two learning outcomes”, Personnel Psychology (Wiley) vol. 51, pp.397–420, 1998. http://dx.doi.org/10.1111/j.1744-6570.1998.tb00731.x
[3] Redeker, C., Punie, Y., Ferrari, A., “eAssessment for 21st century learning and skills”, In Ravenscroft, A., Lindsteadt, S. Kloos, C.D., Hernandez-Leo, D. (eds.). 21st Century Learning for 21st Century Skills. Proceedings of the 7th European Conference on technology-enhanced learning EC-TEL. Springer, Heidelberg, pp.292-305, 2012. http://dx.doi.org/10.1007/978-3-642-33263-0_23
[4] Westera, W., Nadolski, R., Hummel, H., Wopereis, I., “Serious Games for Higher Education: a Framework for Reducing Design Complexity”, Journal of Computer-Assisted Learning (Wiley), vol. 24, no. 5, pp. 420-432, 2008. http://dx.doi.org/10.1111/j.1365-2729.2008.00279.x
[5] Westera, W., Nadolski, R, Hummel, H., “Learning analytics in serious gaming: uncovering the hidden treasury of game log files”, Paper presented at the GALA Serious Gaming Conference October 24-25 in Paris, France, 2013.
[6] Buckingham Shum, S., Ferguson, R., “Social Learning Analytics”, Educational Technology & Society (IFETS), vol. 15, no. 3, pp.3–26, 2012.
[7] Baker, R.S.J.D., Yacef, K., “The state of educational data mining in 2009: A review and future visions”, Journal of Educational Data Mining vol. 1, no. 1, pp.3-17, 2009.
[8] Romero, C., Ventura, S., “Educational data mining: A survey from 1995 to 2005”, Expert Systems with Applications (Elsevier), vol. 33, no. 1, pp.135-146, 2007. http://dx.doi.org/10.1016/j.eswa.2006.04.005
[9] Bienkowski, M., Feng, M., Means, B., “Enhancing Teaching and Learning Through Educational Data Mining and Learning Analytics: An Issue Brief”. SRI International U.S. Department of Education, Office of Educational Technology, Washington, 2012. Retrieved from http://www.ed.gov/edblogs/technology/files/2012/03/edm-la-brief.pdf
[10] Zouaq, A., Jocsimovíc, S., Gasevíc, D., “Ontology Learning to Analyze Research Trends in Learning Analytics Publications”. In Mathieu d'Aquin, M., Dietze, S., Drachsler, H., Herder, E., Taibi. D. (eds), Proceedings of the LAK Data Challenge, held at LAK 2013, the Third Conference on Learning Analytics and Knowledge, Leuven, Belgium, April 9, 2013. Available at http://ceur-ws.org/Vol-974/lakdatachallenge2013_08.pdf
[11] Martin, T., Sherin,B., “Learning Analytics and Computational Techniques for Detecting and Evaluating Patterns in Learning: An Introduction to the Special Issue”, Journal of the Learning Sciences (Taylor & Francis), vol. 22, no. 4, pp.511-520, 2013. DOI: 10.1080/10508406.2013.840466
[12] Arnold, K.E., “Signals: Applying academic analytics”, Educause Quarterly, vol. 33, no. 1, pp.10-15, 2010.. Retrieved from http://www.educause.edu/ero/article/signals-applying-academic-analytics
[13] Pistilli, M., Arnold, K., “Course signals at Purdue: Using learning analytics to increase student success”, 2nd International Conference on Learning Analytics and Knowledge, Vancouver, 2012.
[14] Long, P.D., Siemens, G., “Penetrating the Fog: Analytics in Learning and Education”, Educause Review, vol. 46. No. 5, pp.31-40, 2011. Retrieved from http://www.educause.edu/ero/article/penetrating-fog-analytics-learning-and-education
[15] Cooper, A., Powell, S., MacNeill, S., “ Survey of the State of Analytics in UK Higher and Further Institutions 2013”, White Paper 2013:WP03, Bolton: CETIS. Retrieved from http://publications.cetis.ac.uk/2013/884
[16] Kraan, W., Sherlock, D., “Analytics Tools and Infrastructure. CETIS Analytics Series, vol.1, no.11, 2013, Bolton: JISC/CETIS. Retrieved from http://publications.cetis.ac.uk/2013/535
[17] Hung, J.-L., Hsu, Y.-C., Rice, K., “Integrating Data Mining in Program Evaluation of K-12 Online Education”, Educational Technology & Society (IFETS), vol. 15, no. 3, pp.27–41, 2012.
[18] Greller, W., Drachsler, H., “Translating Learning into Numbers: A Generic Framework for Learning Analytics”, Educational Technology & Society (IFETS), vol. 15, no. 3, pp.42–57, 2012.
[19] Siemens, G., “Learning analytics: A foundation for informed change in higher education”, Educause, slides retrieved from http://www.slideshare.net/gsiemens/learning-analytics-educause
[20] Elias, T., “Learning analytics: Definitions, processes and potentials”, retrieved from http://learninganalytics.net/LearningAnalyticsDefinitionsProcessesPotential.pdf
[21] Campbell, J.P., Oblinger, D.G., “Academic Analytics”, Educause White Paper 2007, retrieved from http://www.educause.edu/ir/library/pdf/PUB6101.pdf
[22] Dyckhoff, A.L., Zielke, D., Bültmann, M., Chatti, M.A., Schroeder, U., “Design and Implementation of a Learning Analytics Toolkit for Teachers”. Educational Technology & Society (IFETS), vol. 15, no. 3, pp.58–76, 2012.
[23] Boyd, D., Crawford, K., “Six Provocations for Big Data, A Decade in Internet Time”, Symposium on the Dynamics of the Internet and Society, Oxford Internet Institute, Oxford, 2012. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1926431
[24] Bluemink,J.,Hämäläinen, R., Manninen, T., Järvelä, S., “Group-level analysis on multiplayer game collaboration: how do the individuals shape the group interaction?”, Interactive Learning Environments (Taylor & Francis), vol. 18, no. 4, pp.365–383, 2010.
[25] Fernández-Gallego, B., Lama, M., Vidal, J.C., Mucientes, M., “Learning Analytics Framework for Educational Virtual Worlds”. In Proceedings of the 2013 International Conference on Virtual and Augmented Reality in Education, Procedia Computer Science (Elsevier), vol. 25 , pp.443-447, 2013. DOI: 10.1016/j.procs.2013.11.056
[26] Koper, R., Tattersall, C., “Learning Design: A handbook on modelling and delivering networked education and training”, Springer, Berlin, 2005.
[27] Gobert, J.D., Sao Pedro, M., Raziuddin, J., Baker, R.S., “From Log Files to Assessment Metrics: Measuring Students' Science Inquiry Skills Using Educational Data Mining”, Journal of the Learning Sciences (Taylor & Francis), vol. 22, no. 4, pp.521-563, 2013. DOI: 10.1080/10508406.2013.837391
[28] Westera, W., Hommes, M.A., Houtmans, M., Kurvers, H.J., “Computer-Supported Training of Psychodiagnostic Skills”, Interactive Learning Environments (Taylor & Francis), vol. 11, no. 3, pp.215-231, 2003.
[29] Kickmeier-Rust, M.D., Albert, D., “Micro Adaptivity: Protecting Immersion in Didactically Adaptive Digital Educational Games”, Journal of Computer Assisted Learning (Wiley),vol. 26, pp.95-105, 2010. http://dx.doi.org/10.1111/j.1365-2729.2009.00332.x
[30] Reese, D.D., Seward, R.J., Tabachnick, B.G., Hitt, B., Harrison, A., McFarland, L., “Timed Report Measures Learning: Game-Based Embedded Assessment”, In Ifenthaler, D., Eseryel, D., Ge, X. (eds.), Assessment in Game-Based Learning: Foundations, Innovations, and Perspectives. Springer, New York, 2013.
[31] Serrano-Laguna, A., Torrente, J., Moreno-Ger, P., Fernández-Manjón, B., “Tracing a Little for Big Improvements: Application of Learning Analytics and Videogames for Student Assessment”, Procedia Computer Science (Elsevier), vol. 15, pp.203-209, 2012. Retrieved from http://www.sciencedirect.com/science/article/pii/S1877050912008344 http://dx.doi.org/10.1016/j.procs.2012.10.072
[32] Shute, V.J., Ventura, M., Bauer, M., Zapata-Rivera, D., “Melding the Power of Serious Games and Embedded Assessment to Monitor and Foster Learning: Flow and Grow”, In Ritterfeld, U., Cody, M., Vorderer, M. (eds.). Serious Games: Mechanisms and Effects, Routledge, New York, pp.295-321, 2009.
[33] Mislevy, R.J., Steinberg, L.S., Almond, R.G., “On the structure of educational assessment”, Measurement: Interdisciplinary Research and Perspective (Taylor& Francis), vol. 10, pp.3-62, 2003.
[34] Pearl, J., “Probabilistic reasoning in intelligent systems: Networks of plausible inference”, Kaufmann, San Mateo CA, Unites States, 1988.
[35] Slootmaker, A., Kurvers, H.J., Hummel, H.G.K., Koper, E.J.R., “Developing scenario-based serious games for complex cognitive skills acquisition: Design, development and evaluation of the EMERGO platform”, in preparation, 2014.
[36] Westera, W., “A didactic framework for audiovisual design”, Journal of Educational Media (Taylor& Francis), vol. 24, no. 2, pp.87-102, 1999.
[37] Koumi, J., “Designing Video and Multimedia for Open and Flexible Learning”, Routledge, London, 2006.
[38] Green, S.B., “How many subjects does it take to do a regression analysis?”, Multivariate Behavioral Research (Taylor& Francis), vol. 26, pp.499–510, 1991. http://dx.doi.org/10.1207/s15327906mbr2603_7
Downloads
Published
2014-06-19
Issue
Section
Articles
License
IJSG copyright information is provided here.
How to Cite
Serious Gaming Analytics: What Students´ Log Files Tell Us about Gaming and Learning. (2014). International Journal of Serious Games, 1(2). https://doi.org/10.17083/ijsg.v1i2.9
