-
Delving Into the Psychology of Machines: Exploring the Structure of Self-Regulated Learning via LLM-Generated Survey Responses
Authors:
Leonie V. D. E. Vogelsmeier,
Eduardo Oliveira,
Kamila Misiejuk,
Sonsoles López-Pernas,
Mohammed Saqr
Abstract:
Large language models (LLMs) offer the potential to simulate human-like responses and behaviors, creating new opportunities for psychological science. In the context of self-regulated learning (SRL), if LLMs can reliably simulate survey responses at scale and speed, they could be used to test intervention scenarios, refine theoretical models, augment sparse datasets, and represent hard-to-reach po…
▽ More
Large language models (LLMs) offer the potential to simulate human-like responses and behaviors, creating new opportunities for psychological science. In the context of self-regulated learning (SRL), if LLMs can reliably simulate survey responses at scale and speed, they could be used to test intervention scenarios, refine theoretical models, augment sparse datasets, and represent hard-to-reach populations. However, the validity of LLM-generated survey responses remains uncertain, with limited research focused on SRL and existing studies beyond SRL yielding mixed results. Therefore, in this study, we examined LLM-generated responses to the 44-item Motivated Strategies for Learning Questionnaire (MSLQ; Pintrich \& De Groot, 1990), a widely used instrument assessing students' learning strategies and academic motivation. Particularly, we used the LLMs GPT-4o, Claude 3.7 Sonnet, Gemini 2 Flash, LLaMA 3.1-8B, and Mistral Large. We analyzed item distributions, the psychological network of the theoretical SRL dimensions, and psychometric validity based on the latent factor structure. Our results suggest that Gemini 2 Flash was the most promising LLM, showing considerable sampling variability and producing underlying dimensions and theoretical relationships that align with prior theory and empirical findings. At the same time, we observed discrepancies and limitations, underscoring both the potential and current constraints of using LLMs for simulating psychological survey data and applying it in educational contexts.
△ Less
Submitted 16 June, 2025;
originally announced June 2025.
-
Human-AI Collaboration or Academic Misconduct? Measuring AI Use in Student Writing Through Stylometric Evidence
Authors:
Eduardo Araujo Oliveira,
Madhavi Mohoni,
Sonsoles López-Pernas,
Mohammed Saqr
Abstract:
As human-AI collaboration becomes increasingly prevalent in educational contexts, understanding and measuring the extent and nature of such interactions pose significant challenges. This research investigates the use of authorship verification (AV) techniques not as a punitive measure, but as a means to quantify AI assistance in academic writing, with a focus on promoting transparency, interpretab…
▽ More
As human-AI collaboration becomes increasingly prevalent in educational contexts, understanding and measuring the extent and nature of such interactions pose significant challenges. This research investigates the use of authorship verification (AV) techniques not as a punitive measure, but as a means to quantify AI assistance in academic writing, with a focus on promoting transparency, interpretability, and student development. Building on prior work, we structured our investigation into three stages: dataset selection and expansion, AV method development, and systematic evaluation. Using three datasets - including a public dataset (PAN-14) and two from University of Melbourne students from various courses - we expanded the data to include LLM-generated texts, totalling 1,889 documents and 540 authorship problems from 506 students. We developed an adapted Feature Vector Difference AV methodology to construct robust academic writing profiles for students, designed to capture meaningful, individual characteristics of their writing. The method's effectiveness was evaluated across multiple scenarios, including distinguishing between student-authored and LLM-generated texts and testing resilience against LLMs' attempts to mimic student writing styles. Results demonstrate the enhanced AV classifier's ability to identify stylometric discrepancies and measure human-AI collaboration at word and sentence levels while providing educators with a transparent tool to support academic integrity investigations. This work advances AV technology, offering actionable insights into the dynamics of academic writing in an AI-driven era.
△ Less
Submitted 12 May, 2025;
originally announced May 2025.
-
Classroom Activities and New Classroom Apps for Enhancing Children's Understanding of Social Media Mechanisms
Authors:
Henriikka Vartiainen,
Nicolas Pope,
Juho Kahila,
Sonsoles López-Pernas,
Matti Tedre
Abstract:
Young people are increasingly exposed to adverse effects of data-driven profiling, recommending, and manipulation on social media platforms, most of them without adequate understanding of the mechanisms that drive these platforms. In the context of computing education, educating learners about mechanisms and data practices of social media may improve young learners' data agency, digital literacy,…
▽ More
Young people are increasingly exposed to adverse effects of data-driven profiling, recommending, and manipulation on social media platforms, most of them without adequate understanding of the mechanisms that drive these platforms. In the context of computing education, educating learners about mechanisms and data practices of social media may improve young learners' data agency, digital literacy, and understanding how their digital services work. A four-hour technology -- supported intervention was designed and implemented in 12 schools involving 209 5th and 8th grade learners. Two new classroom apps were developed to support the classroom activities. Using Likert-scale questions borrowed from a data agency questionnaire and open-ended questions that mapped learners' data-driven reasoning on social media phenomena, this article shows significant improvement between pre- and post-tests in learners' data agency and data-driven explanations of social media mechanisms. Results present an example of improving young learners' understanding of social media mechanisms.
△ Less
Submitted 27 January, 2025;
originally announced January 2025.
-
Complex Dynamic Systems in Education: Beyond the Static, the Linear and the Causal Reductionism
Authors:
Mohammed Saqr,
Daryn Dever,
Sonsoles López-Pernas,
Christophe Gernigon,
Gwen Marchand,
Avi Kaplan
Abstract:
Traditional methods in educational research often fail to capture the complex and evolving nature of learning processes. This chapter examines the use of complex systems theory in education to address these limitations. The chapter covers the main characteristics of complex systems such as non-linear relationships, emergent properties, and feedback mechanisms to explain how educational phenomena u…
▽ More
Traditional methods in educational research often fail to capture the complex and evolving nature of learning processes. This chapter examines the use of complex systems theory in education to address these limitations. The chapter covers the main characteristics of complex systems such as non-linear relationships, emergent properties, and feedback mechanisms to explain how educational phenomena unfold. Some of the main methodological approaches are presented, such as network analysis and recurrence quantification analysis to study relationships and patterns in learning. These have been operationalized by existing education research to study self-regulation, engagement, and academic emotions, among other learning-related constructs. Lastly, the chapter describes data collection methods that are suitable for studying learning processes from a complex systems' lens.
△ Less
Submitted 30 January, 2025; v1 submitted 18 December, 2024;
originally announced January 2025.
-
An XAI Social Media Platform for Teaching K-12 Students AI-Driven Profiling, Clustering, and Engagement-Based Recommending
Authors:
Nicolas Pope,
Juho Kahila,
Henriikka Vartiainen,
Mohammed Saqr,
Sonsoles Lopez-Pernas,
Teemu Roos,
Jari Laru,
Matti Tedre
Abstract:
This paper, submitted to the special track on resources for teaching AI in K-12, presents an explainable AI (XAI) education tool designed for K-12 classrooms, particularly for students in grades 4-9. The tool was designed for interventions on the fundamental processes behind social media platforms, focusing on four AI- and data-driven core concepts: data collection, user profiling, engagement metr…
▽ More
This paper, submitted to the special track on resources for teaching AI in K-12, presents an explainable AI (XAI) education tool designed for K-12 classrooms, particularly for students in grades 4-9. The tool was designed for interventions on the fundamental processes behind social media platforms, focusing on four AI- and data-driven core concepts: data collection, user profiling, engagement metrics, and recommendation algorithms. An Instagram-like interface and a monitoring tool for explaining the data-driven processes make these complex ideas accessible and engaging for young learners. The tool provides hands-on experiments and real-time visualizations, illustrating how user actions influence both their personal experience on the platform and the experience of others. This approach seeks to enhance learners' data agency, AI literacy, and sensitivity to AI ethics. The paper includes a case example from 12 two-hour test sessions involving 209 children, using learning analytics to demonstrate how they navigated their social media feeds and the browsing patterns that emerged.
△ Less
Submitted 18 December, 2024;
originally announced December 2024.
-
Transition Network Analysis: A Novel Framework for Modeling, Visualizing, and Identifying the Temporal Patterns of Learners and Learning Processes
Authors:
Mohammed Saqr,
Sonsoles López-Pernas,
Tiina Törmänen,
Rogers Kaliisa,
Kamila Misiejuk,
Santtu Tikka
Abstract:
This paper presents a novel learning analytics method: Transition Network Analysis (TNA), a method that integrates Stochastic Process Mining and probabilistic graph representation to model, visualize, and identify transition patterns in the learning process data. Combining the relational and temporal aspects into a single lens offers capabilities beyond either framework, including centralities to…
▽ More
This paper presents a novel learning analytics method: Transition Network Analysis (TNA), a method that integrates Stochastic Process Mining and probabilistic graph representation to model, visualize, and identify transition patterns in the learning process data. Combining the relational and temporal aspects into a single lens offers capabilities beyond either framework, including centralities to capture important learning events, community detection to identify behavior patterns, and clustering to reveal temporal patterns. Furthermore, TNA introduces several significance tests that go beyond either method and add rigor to the analysis. Here, we introduce the theoretical and mathematical foundations of TNA and we demonstrate the functionalities of TNA with a case study where students (n=191) engaged in small-group collaboration to map patterns of group dynamics using the theories of co-regulation and socially-shared regulated learning. The analysis revealed that TNA can map the regulatory processes as well as identify important events, patterns, and clusters. Bootstrap validation established the significant transitions and eliminated spurious transitions. As such, TNA can capture learning dynamics and provide a robust framework for investigating the temporal evolution of learning processes. Future directions include -- inter alia -- expanding estimation methods, reliability assessment, and building longitudinal TNA.
△ Less
Submitted 5 February, 2025; v1 submitted 23 November, 2024;
originally announced November 2024.
-
Empowering Database Learning Through Remote Educational Escape Rooms
Authors:
Enrique Barra,
Sonsoles López-Pernas,
Aldo Gordillo,
Alejandro Pozo,
Andres Muñoz-Arcentales,
Javier Conde
Abstract:
Learning about databases is indispensable for individuals studying software engineering or computer science or those involved in the IT industry. We analyzed a remote educational escape room for teaching about databases in four different higher education courses in two consecutive academic years. We employed three instruments for evaluation: a pre- and post-test to assess the escape room's effecti…
▽ More
Learning about databases is indispensable for individuals studying software engineering or computer science or those involved in the IT industry. We analyzed a remote educational escape room for teaching about databases in four different higher education courses in two consecutive academic years. We employed three instruments for evaluation: a pre- and post-test to assess the escape room's effectiveness for student learning, a questionnaire to gather students' perceptions, and a Web platform that unobtrusively records students' interactions and performance. We show novel evidence that educational escape rooms conducted remotely can be engaging as well as effective for teaching about databases.
△ Less
Submitted 28 August, 2024;
originally announced September 2024.
-
Have Learning Analytics Dashboards Lived Up to the Hype? A Systematic Review of Impact on Students' Achievement, Motivation, Participation and Attitude
Authors:
Rogers Kaliisa,
Kamila Misiejuk,
Sonsoles López-Pernas,
Mohammad Khalil,
Mohammed Saqr
Abstract:
While learning analytics dashboards (LADs) are the most common form of LA intervention, there is limited evidence regarding their impact on students learning outcomes. This systematic review synthesizes the findings of 38 research studies to investigate the impact of LADs on students' learning outcomes, encompassing achievement, participation, motivation, and attitudes. As we currently stand, ther…
▽ More
While learning analytics dashboards (LADs) are the most common form of LA intervention, there is limited evidence regarding their impact on students learning outcomes. This systematic review synthesizes the findings of 38 research studies to investigate the impact of LADs on students' learning outcomes, encompassing achievement, participation, motivation, and attitudes. As we currently stand, there is no evidence to support the conclusion that LADs have lived up to the promise of improving academic achievement. Most studies reported negligible or small effects, with limited evidence from well-powered controlled experiments. Many studies merely compared users and non-users of LADs, confounding the dashboard effect with student engagement levels. Similarly, the impact of LADs on motivation and attitudes appeared modest, with only a few exceptions demonstrating significant effects. Small sample sizes in these studies highlight the need for larger-scale investigations to validate these findings. Notably, LADs showed a relatively substantial impact on student participation. Several studies reported medium to large effect sizes, suggesting that LADs can promote engagement and interaction in online learning environments. However, methodological shortcomings, such as reliance on traditional evaluation methods, self-selection bias, the assumption that access equates to usage, and a lack of standardized assessment tools, emerged as recurring issues. To advance the research line for LADs, researchers should use rigorous assessment methods and establish clear standards for evaluating learning constructs. Such efforts will advance our understanding of the potential of LADs to enhance learning outcomes and provide valuable insights for educators and researchers alike.
△ Less
Submitted 22 December, 2023;
originally announced December 2023.
-
Modeling Digital Twin Data and Architecture: A Building Guide with FIWARE as Enabling Technology
Authors:
Javier Conde,
Andrés Munoz-Arcentales,
Álvaro Alonso,
Sonsoles López-Pernas,
Joaquín Salvachúa
Abstract:
The use of Digital Twins in the industry has become a growing trend in recent years, allowing to improve the lifecycle of any process by taking advantage of the relationship between the physical and the virtual world. Existing literature formulates several challenges for building Digital Twins, as well as some proposals for overcoming them. However, in the vast majority of the cases, the architect…
▽ More
The use of Digital Twins in the industry has become a growing trend in recent years, allowing to improve the lifecycle of any process by taking advantage of the relationship between the physical and the virtual world. Existing literature formulates several challenges for building Digital Twins, as well as some proposals for overcoming them. However, in the vast majority of the cases, the architectures and technologies presented are strongly bounded to the domain where the Digital Twins are applied. This article proposes the FIWARE Ecosystem, combining its catalog of components and its Smart Data Models, as a solution for the development of any Digital Twin. We also provide a use case to showcase how to use FIWARE for building Digital Twins through a complete example of a Parking Digital Twin. We conclude that the FIWARE Ecosystem constitutes a real reference option for developing DTs in any domain.
△ Less
Submitted 3 September, 2023;
originally announced September 2023.
-
A modern approach to transition analysis and process mining with Markov models: A tutorial with R
Authors:
Jouni Helske,
Satu Helske,
Mohammed Saqr,
Sonsoles López-Pernas,
Keefe Murphy
Abstract:
This chapter presents an introduction to Markovian modeling for the analysis of sequence data. Contrary to the deterministic approach seen in the previous sequence analysis chapters, Markovian models are probabilistic models, focusing on the transitions between states instead of studying sequences as a whole. The chapter provides an introduction to this method and differentiates between its most c…
▽ More
This chapter presents an introduction to Markovian modeling for the analysis of sequence data. Contrary to the deterministic approach seen in the previous sequence analysis chapters, Markovian models are probabilistic models, focusing on the transitions between states instead of studying sequences as a whole. The chapter provides an introduction to this method and differentiates between its most common variations: first-order Markov models, hidden Markov models, mixture Markov models, and mixture hidden Markov models. In addition to a thorough explanation and contextualization within the existing literature, the chapter provides a step-by-step tutorial on how to implement each type of Markovian model using the R package seqHMM. The chaper also provides a complete guide to performing stochastic process mining with Markovian models as well as plotting, comparing and clustering different process models.
△ Less
Submitted 2 September, 2023;
originally announced September 2023.
