دورية أكاديمية
Investigating the Reliability of Aggregate Measurements of Learning Process Data: From Theory to Practice
| العنوان: | Investigating the Reliability of Aggregate Measurements of Learning Process Data: From Theory to Practice |
|---|---|
| اللغة: | English |
| المؤلفون: | Yingbin Zhang (ORCID |
| المصدر: | Journal of Computer Assisted Learning. 2024 40(3):1295-1308. |
| الإتاحة: | Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-usTest |
| تمت مراجعته من قبل الزملاء: | Y |
| Page Count: | 14 |
| تاريخ النشر: | 2024 |
| Sponsoring Agency: | National Science Foundation (NSF) |
| Contract Number: | 1942962 |
| نوع الوثيقة: | Journal Articles Reports - Research |
| Education Level: | Higher Education Postsecondary Education |
| الواصفات: | Learning Analytics, Learning Processes, Test Reliability, Psychometrics, Undergraduate Students, Time Management, Programming, Computer Science Education |
| DOI: | 10.1111/jcal.12951 |
| تدمد: | 0266-4909 1365-2729 |
| مستخلص: | Background: Learning analytics (LA) research often aggregates learning process data to extract measurements indicating constructs of interest. However, the warranty that such aggregation will produce reliable measurements has not been explicitly examined. The reliability evidence of aggregate measurements has rarely been reported, leaving an implicit assumption that such measurements are free of errors. Objectives: This study addresses these gaps by investigating the psychometric pros and cons of aggregate measurements. Methods: This study proposes a framework for aggregating process data, which includes the conditions where aggregation is appropriate, and a guideline for selecting the proper reliability evidence and the computing procedure. We support and demonstrate the framework by analysing undergraduates' academic procrastination and programming proficiency in an introductory computer science course. Results and Conclusion: Aggregation over a period is acceptable and may improve measurement reliability only if the construct of interest is stable during the period. Otherwise, aggregation may mask meaningful changes in behaviours and should be avoided. While selecting the type of reliability evidence, a critical question is whether process data can be regarded as repeated measurements. Another question is whether the lengths of processes are unequal and individual events are unreliable. If the answer to the second question is no, segmenting each process into a fixed number of bins assists in computing the reliability coefficient. Major Takeaways: The proposed framework can be a general guideline for aggregating process data in LA research. Researchers should check and report the reliability evidence for aggregate measurements before the ensuing interpretation. |
| Abstractor: | As Provided |
| Entry Date: | 2024 |
| رقم الانضمام: | EJ1424211 |
| قاعدة البيانات: | ERIC |
| تدمد: | 0266-4909 1365-2729 |
|---|---|
| DOI: | 10.1111/jcal.12951 |