Meta-synthesis of a comprehensive framework of factors affecting the usability of learning management systems

Abdoli, S.; Nili Ahmadabadi, M.; Fardanesh, H.; Asgari, M.

doi:10.22061/tej.2023.9452.2848

Document Type : Original Research Paper

Authors

¹ Department of Educational Technology, Faculty of Psychology and Educational Sciences, Allameh Tabataba'i University, Tehran, Iran

² Department of Educational Technology, Faculty of humanity sciences, Tarbiyat Modarres University, Tehran, Iran

³ Department of Educational Measurement, Faculty of Psychology and Educational Sciences, Allameh Tabataba'i University, , Iran

https://doi.org/10.22061/tej.2023.9452.2848

Abstract

Background and Objectives:. Learning management systems are one of the most important platforms for virtual education, and usability is the most important feature that determines their quality. In the conducted research, various factors have been proposed as factors affecting the usability of learning management systems, but shortcomings in the field of usability of these systems and determining the factors affecting it in a comprehensive and practical guide map, based on which usability of learning management system can be evaluated and improved, is the challenge of various educational institutions in virtual education. Therefore, the purpose of this research is to extract the most important factors affecting usability and then present these factors in a comprehensive and practical framework.

Methods: In this research, the qualitative method and meta-synthesis approach were used. Based on the stages of the meta-synthesis approach, after stating the research questions and purposes, the systematic review method was used to search for research reports in the field of usability of learning management systems, which were the statistical population of this research, and the sample - the documents available in Scopus and Web of Science databases - was selected through purposive sampling. Then, the quality of research resources was examined through individual appraisal and comparative appraisal; the findings from the research were classified, these findings were synthesized by the taxonomic analysis approach, and finally, to examine the validity of the findings, the audit trail, consensual validity, and expert peer review methods were used.

Findings: The findings of this research included the usability factors of learning management systems and the presentation of these factors in a framework. In this framework, factors were categorized based on the responsible of each factor. Factors included navigation, visual design, learnability and memorability, efficiency, accessibility, errors correction, privacy and security, ease of use, ethical issues and compliance in the tasks of IT specialists and content, Instructor’s knowledge, and learning orientation in the task Instructors/instructional designers. Factors that both IT specialists and instructors/instructional designers were in charge of encompassed interactivity/interaction, assessment and feedback, help and documentation, usefulness, and satisfaction and enjoyment.

Conclusion: Based on the findings, by considering usability factors in designing and evaluation of learning management systems, it is possible to increase the efficiency and effectiveness of these platforms in virtual education and ultimately the level of learners' satisfaction. The results of this research, while enriching the research literature in the field of usability of learning management systems, help these systems to be used efficiently, effectively and satisfactorily by learners and teachers. The results also help system developers, instructional designers, and evaluators to adopt a more effective approach to improving system usability by designing and evaluating learning management systems based on the framework presented in this research. The time-consuming nature of testing the usability of learning management systems used in Iranian universities based on the proposed framework was one of the limitations of this research, and other researchers can focus on doing this test in future research. It is also suggested that future research, in addition to the Scopus and Web of Science databases, which have been selected as the sources for research reports, consider the resources available in other databases, such as ERIC and Google Scholar. In addition, increasing the number of reviewers of research in the systematic review and the number of evaluators in the phase of evaluating the quality of resources and validating findings make the results more generalizable. Finally, applying quantitative research methods as well as heuristic evaluation of usability factors can complement this qualitative research.

Keywords

20.1001.1.20080441.1402.17.4.11.6

Main Subjects

E-Lerning

COPYRIGHTS
©2023 The author(s). This is an open access article distributed under the terms of the Creative Commons Attribution (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, as long as the original authors and source are cited. No permission is required from the authors or the publishers.

References

[1] Syahid A, Kamri K A, Azizan S N. Usability of Massive Open Online Courses (MOOCs): Malaysian Undergraduates’ Perspective. Journal of Educators Online. 2021; 18(3): 2068-2080.

[2] Alshehri A, Rutter M, Smith S. The Effects of UTAUT and Usability Qualities on Students' Use of Learning Management Systems in Saudi Tertiary Education. Journal of Information Technology Education. 2020; 19(1): 891-930.

[3] Daramola O, Oladipupo O, Afolabi I, Olopade A. Heuristic Evaluation of an Institutional E-learning System: A Nigerian Case. International Journal of Emerging Technologies in Learning. 2017; 12(3): 26-42.

[4] Al-Fraihat D, Joy M, Masa'deh R, Sinclair J. Evaluating E-learning systems success: An empirical study. Computers in Human Behavior. 2020; 102(1): 67-86.

[5] Coates H, James R, Baldwin G. A critical examination of the effects of learning management systems on university teaching and learning. Tertiary Education and Management. 2005; 11(1): 19-36.

[6] Haghshenas M. A model for utilizing social softwares in learning management system of E-learning. Journal of Iranian Distance Education. 2019; 1(4): 25-38.

[7] Tere T, Seta H B, Hidayanto A N, Abidin, Z. Variables affecting e-learning services quality in Indonesian higher education: Students' perspectives. Journal of Information Technology Education: Research. 2020; 19(1): 259-286.

[8] Junus I S, Santoso H B, Isal, Y K, Utomo A. Y. Usability Evaluation of the Student-Centered e-Learning Environment. International Review of Research in Open and Distance Learning. 2015; 16(4): 62-82.

[9] Gunesekera A I, Bao Y, Kibelloh M. The role of usability on e-learning user interactions and satisfaction a literature review. Journal of Systems and Information Technology. 2019; 21(3): 368-394.

[10] Kapenieks J. User-friendly e-learning Environment for Educational Action Research. Procedia Computer Science. 2013; 26 (1): 121-142.

[11] Eltahir M E, Al-Qatawneh, S, Al-Ramahi N, Najeh Alsalhi. The perspective of students and faculty members on the efficiency and usability of e-learning courses at Ajman university: a case study. Journal of Technology and Science Education. 2019; 9(3): 38-403.

[12] Norm DI. Ergonomics of human-system interaction–Part 11: Usability: Definitions and concepts. ISO. 2018; 11.

[13] Freire L L, Arezes P M, Campos J C. A literature review about usability evaluation methods for e-learning platforms. Work. 2012; 41(1): 1038-1044.

[14] Cooper M, Colwell C, Jelfs A. Embedding accessibility and usability: considerations for e-learning research and development projects. Research in Learning Technology. 2007; 15(3): 231-245.

[15] Schoeffel R. The concept of product usability. ISO BULLETIN. 2003; 34(1): 5-7.

[16] Nielsen J. Usability Engineering. San Francisco: Morgan Kaufmann; 1993.

[17] Dahleez K A, El-Saleh A A, Al Alawi A M, Fattah F A M A. Student learning outcomes and online engagement in time of crisis: the role of e-learning system usability and teacher behavior. The International Journal of Information and Learning Technology. 2021-A; 38(5): 473-492.

[18] Emang D W A B, Lukman R N I R, Kamarulzaman M I S, Zaaba Z F. Usability studies on E-Learning Platforms: Preliminary Study in USM. The 2nd International Conference on Applied Science and Technology (Icast’17): 2017 October 1-8: New York, United States.

[19] Preece J, Sharp H, Rogers Y. Interaction Design: Beyond Human-Computer Interaction. Chichester: John Wiley & Sons; 2015.

[20] Davids M R, Chikte U M E, Halperin M L. Effect of improving the usability of an e-learning resource: a randomized trial. Advances in physiology education. 2014; 38(2): 155-160.

[21] Shee D Y, Wang Y S. Multi-criteria evaluation of the web-based e-learning system: A methodology based on learner satisfaction and its applications. Computers & Education. 2008; 50(3): 894-905.

[22] Alshehri A, Rutter M, Smith S. Assessing the Relative Importance of an E-learning system’s Usability Design Characteristics Based on Students' Preferences. European Journal of Educational Research. 2019; 8(3): 839-855.

[23] Altalbe A. Antecedents of actual usage of e-learning system in high education during COVID-19 pandemic: Moderation effect of instructor support. IEEE Access. 2021; 9(1): 93119-93136.

[24] Brade J, Lorenz M, Busch M, Hammer N. Being there again – Presence in real and virtual environments and its relation to usability and user experience using a mobile navigation task. International Journal of Human-Computer Studies. 2017; 101(1): 76-87.

[25] Dix A, Finlay J, Abowd G, Beale R. Human–Computer Interaction. England: Pearson/Prentice-Hall; 2004.

[26] Holden H, Rada R. Understanding the Influence of Perceived Usability and Technology Self-Efficacy on Teachers’ Technology Acceptance. Journal of Research on Technology in Education. 2011; 43(4): 343-367.

[27] Taat M S, Francis A. Factors influencing the students’ acceptance of e-learning at teacher education institute: An exploratory study in Malaysia. International Journal of Higher Education. 2020; 9(1): 133-141.

[28] Abuhlfaia K, Quincey E D. The Usability of E-learning Platforms in Higher Education: A Systematic Mapping Study. HCI '18: Proceedings of the 32nd International BCS Human Computer Interaction Conference: 2018 July 1-13: Belfast, United Kingdom.

[29] Gaupp R, Fabry G, Körner M. Self-regulated learning and critical reflection in an e-learning on patient safety for third-year medical students. International Journal of Medical Education. 2018; 9(1): 189-194.

[30] Rabin E, Kalman Y M, Kalz M. An empirical investigation of the antecedents of learner-centered outcome measures in MOOCs. International Journal of Educational Technology in Higher Education. 2019; 16(14):1-20.

[31] Joshi A, Vinay M, Bhaskar P. Impact of Coronavirus Pandemic on the Indian Education Sector: Perspectives of Teachers on online teaching and assessments. Interactive Technology and Smart Education. 2021; 18(2): 205-226.

[32] Chopra G, Madan P, Jaisingh P, Bhaskar P. Effectiveness of e-learning portal from students’ perspective: A structural equation model (SEM) approach. Interactive Technology and Smart Education. 2019; 16(4): 94-116.

[33] Pal D, Vanijja V. Perceived usability evaluation of Microsoft Teams as an online learning platform during COVID-19 using system usability scale and technology acceptance model in India. Children and Youth Services Review. 2020; 119: 1-12.

[34] Alsswey A, Al-Samarraie H, El-Qirem F, Zaqout F. M-learning technology in Arab Gulf countries: A systematic review of progress and recommendations. Education and Information Technologies. 2020; 25(3): 2919-2931.

[35] Dwivedi Y K, Rana N, Tamilmani K, Raman R. A meta-analysis based modified unified theory of acceptance and use of technology (meta-UTAUT): a review of emerging literature. Current Opinion in Psychology. 2020; 36(1): 13-18.

[36] Crompton H, Burke D. The use of mobile learning in higher education: A systematic review. Computers & Education. 2018; 123(1): 53-64.

[37] Ababtain M A, Khan A. R. Towards a Framework for Usability of Arabic-English Websites. Procedia Computer Science. 2017; 109(1): 1010-1015.

[38] Persico D, MancaS, Pozzi F. Adapting the Technology Acceptance Model to evaluate the innovative potential of e-learning systems. Computers in Human Behavior. 2014; 30(1): 614-622.

[39] Hargittai E. Digital Na(t)ives? Variation in internet skills and uses among members of the “net generation”. Sociological Inquiry. 2010; 80(1): 92-113

[40] Bhaskar P, Chandan K, Joshi A. Blockchain in education management: present and future applications. Interactive Technology and Smart Education. 2021; 18(1): 1-17.

[41] Başaran S, Mohammed R K H. Usability evaluation of open source learning management systems. International Journal of Advanced Computer Science and Applications (IJACSA). 2020; 11(6): 400-410.

[42] Wang L Y K, Lew S L, Lau S H, Leow, M C. Usability factors predicting continuance of intention to use cloud e-learning application. Heliyon. 2019; 5(6): 1-11.

[43] Rodrigues H, Almeida F, Figueiredo V, Lopes S L. Tracking e-learning through published papers: A systematic review. Computers & Education. 2019; 136(1): 87-98.

[44] Volika S, Fesakis G. To what extent is the use of interaction models as design patterns supported by current e-learning authoring tools A comparative analysis. International Conference on Technology and Innovation in Learning, Teaching and Education: 2019 May 49-61: Springer, Cham.

[45] Leeuw R D, Soet A D, Horst S V D, Walsh K, Westerman M, Scheele, F. How We Evaluate Postgraduate Medical E-Learning: Systematic Review. JMIR Medical Education. 2019; 5(1): 1-43.

[46] Alshehri A, Rutter M J, Smith S. The Effects of Gender and Age on Students’ Use of a Learning Management System in Saudi Arabia. International Journal of Learning and Teaching. 2020-B; 6(3): 135-145.

[47] Santoso H B, Schrepp M, Isal Y K, Utomo A. Measuring user experience of the student-centered e-learning environment. Journal of Educators Online. 2016; 13(1): 210-216.

[48] Kurucay M, Inan F. A. Examining the effects of learner-learner interactions on satisfaction and learning in an online undergraduate course. Computers & Education. 2017; 115(1): 20-37.

[49] Hong J. C, Tai K H, Yueh H M, Kuo Y C, Chen J S. Internet cognitive failure relevant to users' satisfaction with content and interface design to reflect continuance intention to use a government e-learning system. Computers in Human Behavior. 2017; 66(1): 353-362.

[50] Al-Samarraie H, Selim H, Zaqout F. The effect of content representation design principles on users’ intuitive beliefs and use of e-learning systems. Interactive Learning Environments. 2016; 24(8): 1758-1777.

[51] Jou M, Tennyson R D, Tennyson R D, Huang S Y. A study on the Usability of E-books and APP in Engineering Courses: A Case Study on Mechanical Drawing. Computers & Education. 2016; 92(1): 181-193.

[52] Sorgenfrei C, Smolnik S. The Effectiveness of E‐Learning Systems: A Review of the Empirical Literature on Learner Control. Decision Sciences Journal of Innovative Education. 2016; 14(2): 154-184.

[53] Premlatha K R, Geetha T V. Learning content design and learner adaptation for adaptive e-learning environment: a survey. Artificial Intelligence Review. 2015; 44(4): 443-465.

[54] Ping T A. Students' Interaction in the Online Learning Management Systems: A Comparative Study of Undergraduate and Postgraduate Courses. Asian Association of Open Universities Journal. 2011; 6(1): 59-73.

[55] Hasan M K, Tonmon T T, Kabir H, Masud S B, Hasan M A, Das B C, Akter M, Hawlader M D H, Mitra D K. Availability and use of technology for e-learning: To what extent do these impact Bangladeshi university students? A cross-sectional study. F1000 Research. 2022; 10(1285): 1-16.

[56] Chaveesuk S, Chaiyasoonthorn W. COVID-19 in Emerging Countries and Students’ Intention to Use Cloud Classroom: Evidence from Thailand. Education Research International. 2022; 2022(3): 1-13.

[57] Dahleez K A, El-Saleh A A, Al Alawi A M’ Abdelfattah F A. Higher education student engagement in times of pandemic: the role of e-learning system usability and teacher behavior. The International Journal of Educational Management (IJEM). 2021; 35(6): 1312-13-29.

[58] Erenler H H T. Heuristic Evaluation of E-Learning. International Journal of Organizational Leadership. 2018; 7(2): 195-210.

[59] Rupere T, Jakovljevic M. Usability and user evaluation of an integrated multimedia e-learning management system. Knowledge Management & E-Learning. 2021; 13(3): 334-366.

[60] Younas A, Faisal C N, Habib N A, Ashraf R, Ahmad M. Role of Design Attributes to Determine the Intention to Use Online Learning via Cognitive Beliefs. IEEE Access. 2021; 9(1): 94181-94202.

[61] Hasibuan D P, Santoso H B, Ariana Yunita, Rahmah A. An Indonesian Adaptation of the E-Learning Usability Scale. Journal of Physics: Conference Series. 2020; 1566(1): 1-7.

[62] Cheng F F, Chiu C C, Wu C S, Tsaih D C. The influence of learning style on satisfaction and learning effectiveness in asynchronous web-based learning system. Library Hi Tech. 2017; 35(6): 473-489.

[63] Fishbein M, Ajzen I. Belief, attitude, intention and behaviour: An introduction to theory and research. Reading, MA: Addison-Wesley; 1975.

[64] Davis F D. A technology acceptance model for empirically testing new end-user information systems: theory and results [dissertation]. Massachusetts institute of technology: Wayne State University; 1980.

[65] Davis F D, Bagozzi R, Warshaw P R. User Acceptance of Computer Technology: A Comparison of Two Theoretical Models. Management Science. 1989; 35(8): 982-1003.

[66] Davis F D. User acceptance of information technology System characteristics, user perceptions and behavioral impacts. International Journal of Man Machine Studies. 1993; 38(3): 475-487.

[67] Ajzen I. From Intentions to Actions: A Theory of Planned Behavior. In: Kuhl J, Beckmann J. (eds.) Action Control from Cognition to Behavior. Berlin, Heidelberg: Springer; 1985. p.11-39.

[68] Bandura A. Social foundations of thought and action: A social cognitive theory. New Jersey: Prentice Hall; 1986.

[69] Davis F D, Bagozzi R P, Warshaw P R. Extrinsic and Intrinsic Motivation to Use Computers in the Workplace. Journal of Applied Social Psychology. 1992; 22(14): 1111-1132.

[70] Kuleshova V V, Kutsak L V, Liulchak S, Tsoi T V, Ivanenko I V. Implementation of Modern Distance Learning Platforms in the Educational Process of HEI and their Effectiveness. International Journal of Higher Education. 2020; 7 (1): 217-229.

[71] Binyamin S S, Rutter M, Smith S. Extending the Technology Acceptance Model to Understand Students’ use of Learning Management Systems in Saudi Higher Education. International Journal of Emerging Technologies in Learning (iJET). 2019; 14(3): 4-21.

[72] Bagci K, Celik H E. Examination of Factors Affecting Continuance Intention to use Web-Based Distance Learning System via Structural Equation Modelling. Eurasian Journal of Educational Research. 2018; 18(78): 43-66.

[73] Alharbi H, Sandhu K, Brown T. Factors for the acceptance of recommender systems in E-Learning for Saudi Universities: A proposed framework. Recent Patents on Computer Science. 2015; 8(2): 90-99.

[74] Sandelowski M, Barroso J. Handbook for Synthesizing Qualitative Research. New York: Springer Publishing Company; 2007.

[75] Batanero J, Rueda M, Cerero J. Use of Augmented Reality for Students with Educational Needs: A Systematic Review (2016-2021). Societies. 2022; 12(2): 1-11.

[76] Morera B, Garcia I, Casanova B. Systematic Review of Inclusive Musical Practices in Non-Formal Educational Contexts. Education Sciences. 2023; 13(1): 1-16.

[77] Chen X, Zou D, Xie H, Wang F. Past, present, and future of smart learning: a topic-based bibliometric analysis. International Journal of Educational Technology in Higher Education. 2021; 18(2): 1-29.

[78] Egger M, Smith G D, O'Rourke K. Rationale, potentials, and promise of systematic reviews. In: Egger M, Smith G D, Altman, D G (eds.) Systematic Reviews in Health Care: Meta-Analysis in Context. London: BMJ Publishing Group; 2001. p. 3-19.

[79] Tricco A C, Tetzlaff J, Moher D. The art and science of knowledge synthesis. Journal of Clinical Epidemiology. 2011; 64(1): 11-20.

[80] Khan K S, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. Journal of the Royal Society of Medicine. 2003; 96(3): 118-121.

[81] Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart L. A, the PRISMA-P Group. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015; 4 (1).

[82] Alghafis A, Alrasheed A, Abdulghany A. A Study on the Usability of Moodle and Blackboard – Saudi Students Perspectives. International Journal of Interactive Mobile Technologies (iJIM). 2020; 14(10): 159-165.

[83] Poelmans S, Wessa P. A constructivist approach in a blended e-learning environment for statistics. Interactive Learning Environments. 2015; 23(3): 385-401.

[84] Tang P, Yao Z, Luan J, Xiao J. How information presentation formats influence usage behaviour of course management systems flow diagram navigation versus menu navigation. Behaviour & Information Technology. 2020; 41(2): 383-400.

[85] Chu A, Biancarelli D, Drainoni M L, Liu J H, Schneider J I, Sullivan R, Sheng A Y. Usability of Learning Moment: Features of an E-learning Tool That Maximize Adoption by Students. The Western Journal of Emergency Medicine. 2019; 21(1): 78-84.

[86] Al-Hunaiyyan A, Alhajri R, Alghannam B, Al-Shaher, A. Student Information System: Investigating User Experience (UX). International Journal of Advanced Computer Science and Applications (IJACSA). 2021; 12(2): 80-87.

[87] Kusumastuti D L, Sasmoko S, Yossy E, Rabiha S, Indrianti Y. Usability Efficiency Analysis on E-Learning Websites. International Journal of Engineering & Technology. 2018; 7(3.30): 215-218.

[88] Revythi A, Tselios N. Extension of technology acceptance model by using system usability scale to assess behavioral intention to use e-learning. Education and Information Technologies. 2019; 24(1): 2341-2355.

[89] Ramadhan A, Hidayanto A N, Salsabila G A, Wulandari I, Jaury J A, Anjani N N. The effect of usability on the intention to use the e-learning system in a sustainable way: A case study at Universitas Indonesia. Education and Information Technologies. 2022; 27(2): 1489-1522.

[90] Kumar J A, Bervell B, Osman S. Google classroom: insights from Malaysian higher education students’ and instructors’ experiences. Education and Information Technologies. 2020; 25(1): 4175-4195.

[91] Barri M A. What makes web-enhanced learning successful: four key elements. International Journal of Technology Enhanced Learning. 2020; 12(4): 426-446.

[92] Graham M, Hannigan K, Curran P. Imagine: Visual Design in First-Year Composition. Journal of Visual Literacy. 2005; 25(1): 21-0.

[93] Petter S, Delone W, McLean E. Measuring information systems success: Models, dimensions, measures, and interrelationships. European Journal of Information Systems. 2008; 17(3): 236-263.

[94] Ameen N, Willis R, Abdullah M, Shah M H. Towards the successful integration of e‐learning systems in higher education in Iraq: A student perspective. British Journal of Educational Technology. 2019; 50(3): 1434-1446.

[95] Aparicio M, Bação F, Oliveira T. Grit in the path to e-learning success. Computers in Human Behavior. 2017; 66(1): 388-399.

[96] Nielsen J, Loranger H. Prioritizing Web Usability. Berkeley: New Riders; 2006.

[97] Hinderks A, Schrepp M, Mayo F J D, Escalona M J, Thomaschewski J. Developing a UX KPI based on the user experience questionnaire. Computer Standards & Interfaces. 2019; 65(1): 38-44.

[98] Abascal J, Arrue M, Fajardo I, Garay-Vitoria N. The use of guidelines to automatically verify Web accessibility. Universal Access in the Information Society. 2004; 3(1): 71-79.

[99] Kitanov S, Davcev D. Mobile Cloud Computing Environment as a Support for Mobile Learning. CLOUD COMPUTING 2012: The Third International Conference on Cloud Computing, GRIDs, and Virtualization: 2012 July 99-105: Nice: France.

[100] Little B. Issues in mobile learning technology. Human Resource Management International Digest. 2013; 21(3): 26-29.

[101] Aljawarneh S, Al-Rousan T, Maatuk A M, Akour M A. Usage of data validation techniques in online banking: A perspective and case study. Security Journal. 2014; 27(1): 27-35.

[102] Zaharias P, Poulymenakou A. Developing a Usability Evaluation Method for e-Learning Applications: Beyond Functional Usability. International Journal of Human-Computer Interaction. 2009; 25(1): 75-98.

[103] Rubin J, Chisnell D, Spool J. 2008. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. s l: Wiley.