Abstract

Improving the educational process through evaluation requires adopting new concepts and approaches. In addition to the traditional methodologies applied in this field, such as observation or surveys, business methods are increasingly being implemented, including the Deming cycle or selected elements of project management methodologies. Their application allows for a holistic and unbiased view of the quality of teaching and learning.

This article aims to illustrate and evaluate the usefulness of project management methodologies in identifying Critical Success Factors (CSFs) and risks in creating and evaluating academic courses. The subject of the research was the course "Creativity and Decision-making", and the main method of the research, which was conducted in four cycles from 2020 to 2023, was the Deming cycle (PDCA). Data collection was based on data triangulation, including unstructured interviews with students and experts, as well as participant observation. Content analysis with a categorisation key was used to analyse the data, with the results presenting lists of extracted critical success and risk factors determining the quality of the course.

The implemented procedure represents an innovative approach to the evaluation of academic courses, at the core of which is the creation and updating of two substantively opposing lists: the determinants of learning successes and failures constituting of a map of strategic control points, and a basis for the improvement of the educational process.

Keywords: critical success factors (CSF), risk management, project-based learning (PBL), project management methodology (PMM), Deming cycle (PDCA)

References

• Aggarwal, A. K. (2020). Using Deming's Cycle for improvement in a course: A case study. International Journal of Web-Based Learning and Teaching Technologies, 15(3), 31-45. https://doi.org/10.4018/IJWLTT.2020070103
• Ahtee, T., & Poranen, T. (2009). Risks in students' software projects. 22nd Conference on Software Engineering Education and Training (pp. 154-157). https://doi.org/10.1109/CSEET.2009.31
• Alhabeeb, A., & Rowley, J. (2018). E-learning critical success factors: Comparing perspectives from academic staff and students. Computers & Education, 127, 1-12. https://doi.org/10.1016/j.compedu.2018.08.007
• Alias, Z., Zawawi, E. M. A., Yusof, K., & Aris, N. M. (2014). Determining critical success factors of project management practice: A conceptual framework. Procedia - Social and Behavioral Sciences, 153, 61-69. https://doi.org/10.1016/j.sbspro.2014.10.041
• Anh, V. T. K. (2018). Evaluation models in educational program: strengths and weaknesses. Journal of Foreign Studies, 34(2), 140-150. https://doi.org/doi:10.25073/2525-2445/vnufs.4252
• Arksey, H., & Knight, P. (1999). Interviewing for social scientists: An introductory resource with examples. Sage.
• Berg, H. P. (2010). Risk management: Procedures, methods and experiences. Risk Management, 17(1), 79-95.
• Boehm, B. W., Port, D., & Brown, A.W. (2002). Balancing Plan-Driven and Agile Methods in Software Engineering Project Courses. Computer Science Education, 12(3), 187-195. https://doi.org/10.1076/csed.12.3.187.8617
• Cagliano, A. C., Grimaldi, S., & Rafele, C. (2015). Choosing project risk management techniques. A theoretical framework. Journal of Risk Research, 18(2), 232-248. https://doi.org/10.1080/13669877.2014.896398
• Charvat, J. (2003). Project management methodologies: Selecting, implementing, and supporting methodologies and processes for projects. John Wiley and Son, Inc.
• Cheawjindakarn, B. , Suwannatthachote, P., & Theeraroungchaisri, A. (2012). Critical success factors for online distance learning in higher education: A review of the literature. Creative Education, 3(8B), 61-66. https://doi.org/10.4236/ce.2012.38B014
• Chen, S. H. (2012). The establishment of a quality management system for the higher education industry. Quality and Quantity, 46, 1279-1296. https://doi.org/10.1007/s11135-011-9441-1
• Chiocchio, F. (2007). Project team performance: A study of electronic task and coordination communication. Project Management Journal, 38(1), 97-109. https://doi.org/10.1177/875697280703800110
• Crawford, L., & Pollack, J. (2007). How generic are project management knowledge and practice? Project Management Journal, 38(1), 87-96. https://doi.org/10.1177/875697280703800109
• Crispin, G. (2020). The essence of risk identification in project risk management: An overview. International Journal of Science and Research, 9(2), 1553-1557.
• Dam, M., Janssen, F. J. J. M., & van Driel, J. H. (2020). Making sense of student data in teacher professional development. Professional Development in Education, 46(2), 256-273. https://doi.org/10.1080/19415257.2018.1550104
• Dandage, V. D., Mantha, S. S., & Rane, S. B. (2017). Exploring the critical success factors for project management and project risk management. Journal of Advances in Science and Technology, 13(1), 348-352.
• DeWalt, K. M., & DeWalt, B. R. (2002). Participant observation: a guide for fieldworkers. AltaMira Press.
• Divjak, B., & Redjep, N. (2015). Strategic decision making cycle in higher education: Case study of e-learning. Proceedings of the International Conference on E-learning (pp. 19-26). https://files.eric.ed.gov/fulltext/ED562492.pdf
• Eckardt, P. N., Craig, M., & Kraemer, L. (2020). The impact of project-based learning on student content knowledge in an undergraduate, teacher preparation, foundations of education course. Journal for Leadership and Instruction, 19(1), 38-42.
• El-Hassan, K., & Al-Hroub, A. (2013). Strengths and Weaknesses of the Evaluation System of Education in Libya. UNESCO Regional Bureau for Education in the Arab States. https://doi.org/10.13140/RG.2.1.3997.4645
• Fernandes, S., Dinis-Carvalho, J., & Ferreira-Oliveira, A. T. (2021). Improving the performance of student teams in project-based learning with Scrum. Education Sciences, 11(8), 444. https://doi.org/10.3390/educsci11080444
• Flick, U. (2018). The SAGE handbook of qualitative data collection. SAGE Publications Ltd. https://doi.org/10.4135/9781526416070
• Gunarathna, C., Yang, R., Wijeratne Mudiyanselage, P., Amarasinghe, G., Samarasinghalage, T., Weerasinghe, R. P. N., Zhao, H., Zhang, C., Liu, C., Wang, K., & Dev Sureshkumar Jayakumari, S. (2023). Project-based learning for proactive skills development of postgraduate students in solar energy building design digitalization. Smart and Sustainable Built Environment (ahead-of-print). https://doi.org/10.1108/SASBE-08-2022-0173
• Hakim, L., Rosadi, K. I., Widdah, M. E., Us, K. A., Shalahudin, & My, M. (2020.) Did PDCA Cycle, service quality and innovation capability influence private universities performance? Systematic Reviews in Pharmacy, 11(10), 874-883.
• Helsloot, I., & Jong, W. (2006). Risk management in higher education and research in the Netherlands. Journal of Contingencies and Crisis Management, 14(3), 142-159. https://doi.org/10.1111/j.1468-5973.2006.00490.x
• Hounsell, D. (2002). The evaluation of teaching. In: S. Marshall, H. Fry, & S. Ketteridge (Eds.), A handbook for teaching and learning in higher education: Enhancing academic practice (2nd revised ed.) (pp. 200-212). Kogan Page.
• Ifeoma, E. F. (2022). The role of evaluation in teaching and learning process in education. International Journal of Advanced Academic and Educational Research, 13(5), 120-129. https://arcnjournals.org/images/2726145223713511.pdf
• Johnsen, M. M. W., Sjolie, E., & Johansen, V. (2023). Learning to collaborate in a project-based graduate course: A multilevel study of student outcomes. Research in Higher Education, 1-24. https://doi.org/10.1007/s11162-023-09754-7
• Jones, T. K., & Le Fevre, D. M. (2021). Increasing teacher engagement in innovative learning environments: Understanding the effects of perceptions of risk. In: W. Imms, & T. Kvan, (Eds.), Teacher transition into innovative learning environments (pp. 73-84). Springer. https://doi.org/10.1007/978-981-15-7497-9_7
• Kawulich, B. B. (2005). Participant observation as a data collection method. Forum: Qualitative Social Research, 6(2). https://doi.org/10.17169/fqs-6.2.466
• Kirk, D, Luxton-Reilly, A., & Tempero, E. (2022). Refining a risk framework for student group projects. Proceedings of the 22nd Koli Calling International Conference on Computing Education Research (pp. 1-11). https://doi.org/10.1145/3564721.3564730
• Kokotsaki, D., Menzies, V., & Wiggins, A. (2016). Project-based learning: A review of the literature. Improving Schools, 19(3), 267-277. https://doi.org/10.1177/1365480216659733
• Krajcik, J., & Shin, N. (2014). Project-based learning. In: R. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences (2nd ed.) (pp. 275-297). Cambridge University Press.
• Lavanya, N., & Malarvizhi, T. (2008). Risk analysis and management: a vital key to effective project management. Paper presented at PMIR Global Congress 2008. Project Management Institute. https://www.pmi.org/learning/library/risk-analysis-project-management-7070
• Liegel, K. M. (2004). Project-based learning and the future of project management. Paper presented at PMIR Global Congress 2004. Project Management Institute. https://www.pmi.org/learning/library/project-based-learning-future-project-management-8273
• Marchewka, J. T. (2010). A framework for identifying and understanding risks in information technology projects. Journal of International Technology and Information Management, 19(1), 61-74. https://doi.org/10.58729/1941-6679.1057
• McLean, M., & Ashwin, P. (2016). The quality of learning, teaching and the curriculum. In P. Scott, J. Gallacher, & G. Parry (Eds.), New languages and landscapes of higher education (pp. 84-102). Oxford University Press. https://doi.org/doi: 10.1093/ACPROF:OSO/9780198787082.003.0004
• Melton, T. (2011). Project management toolkit: The basics for project success. Elsevier.
• Mergen, S., Kepler, F., Silva da Silva J., & Cera, M. (2014). Using PDCA as a General Framework for Teaching and Evaluating the Learning of Software Engineering Disciplines. iSys - Brazilian Journal of Information Systems, 7(2), 5-24.
• Miller, R. I. (Ed.) (1991). Applying the Deming method to higher education, for more effective human resource management. College and Personnel Association.
• Min, W., & Yu, Z. (2023). A systematic review of critical success factors in blended learning. Education Sciences, 13(5), 469. https://doi.org/10.3390/educsci13050469
• Morgan, S., & Stewart, A. (2017). Continuous improvement of team assignments: Using a web-based tool and the plan-do-check-act cycle in design and redesign: Continuous Improvement of team assignments. Decision Sciences Journal of Innovative Education, 15(3), 303-324. https://doi.org/10.1111/dsji.12132
• Nadim, Z. S., & Al-Hinai, A. H. (2016). Critical success factors of TQM in higher education institutions context. International Journal of Applied Sciences and Management, 1(2), 147-156.
• Netshifhefhe, L., Nobongoza, V., & Maphosa, C. (2016). Quality assuring teaching and learning processes in higher education: A critical appraisal. Journal of Communication, 7, 65-78.
• Neumann, M., & Baumann, L. (2021). Agile methods in higher education: Adapting and using eduScrum with real world projects. 2021 IEEE Frontiers in Education Conference (FIE) (pp. 1-8). https://doi.org/10.1109/FIE49875.2021.9637344
• O'Mahony, K., & Garavan, T. N. (2012). Implementing a quality management framework in a higher education organization: A case study. Quality Assurance in Education, 20(2), 184-200. https://doi.org/10.1108/09684881211219767
• Peraza, A. V., & Furumura, Y. (2022). Project-based learning to develop intercultural communicative competence in virtual exchange contexts. International Journal of Computer-Assisted Language Learning and Teaching, 12(3), 1-17. http://doi.org/10.4018/IJCALLT.307059
• Pietrzak, M., & Paliszkiewicz, J. (2015). Framework of strategic learning: The PDCA Cycle. Management, 10(2), 149-161.
• Project Management Institute. (2004). A guide to the project management body of knowledge (PMBOK Guide) (3rd ed.). https://skat.ihmc.us/rid=1240963222395_1554076313_2451/PMBOK-%203rd%20edition.pdf
• Puri, G. (2012). Critical success Factors in e-Learning - An empirical study. International Journal of Multidisciplinary Research, 2(1), 149-161.
• Raza, M. A., Bilal, M., Rasheed, M. R., Chandio, B. A., Ahmad, N., & Sawand, F. A. (2015). Quality assessment in higher education. International Letters of Social and Humanistic Sciences, 50, 162-171.
• Richards, J. C. (2013). Curriculum approaches in language teaching: Forward, central, and backward design. RELC Journal, 44(1), 5-33. https://doi.org/10.1177/0033688212473293
• Ruzic-Dimitrijevic, L., & Dakic, J. (2014). The risk management in higher education institutions. Online Journal of Applied Knowledge Management, 2(1), 137-152.
• Saleh, A. A., Mohammed, A. H., & Abdullah, M. N. (2015). Critical success factors for sustainable university: A framework from the energy management view. Procedia - Social and Behavioral Sciences, 172, 503-510. https://doi.org/10.1016/j.sbspro.2015.01.392
• Schieg, M. (2006). Risk management in construction project management. Journal of Business Economics and Management, 7(2), 77-83.
• Soong, M. H. B, Chan, H. Ch., Chua, B. Ch., & Loh, K. F. (2001). Critical success factors for online course resources. Computers & Education, 36(2), 101-120. https://doi.org/10.1016/S0360-1315(00)00044-0
• Sumarni, W. (2015). The strengths and weaknesses of the implementation of project based learning: A review. International Journal of Science and Research, 4(3), 478-484.
• Toufaili, B. (2018). Factors influencing quality teaching in higher education. International Journal of Research, 5(21), 553-557.
• van Rooij, S. W. (2009). Scaffolding project-based learning with the project management body of knowledge (PMBOKR). Computers and Education, 52(1), 210-219. https://doi.org/10.1016/j.compedu.2008.07.012
• van Rooij, W. S. (2007). WebMail versus WebApp: Comparing problem-based learning methods in a business research course. Journal of Interactive Learning Research, 18(4), 555-569.
• White, M. D., & Marsh, E. E. (2006). Content analysis: a flexible methodology. Library Trends, 55(1), 22-45. https://doi.org/10.1353/lib.2006.0053
• Wideman, R. M. (1992). Project and program risk management: A guide to managing project risks and opportunities. Project Management Institute.
• Yambi, T. (2018). Assessment and evaluation in education. https://www.academia.edu/35685843/ASSESSMENT_AND_EVALUATION_IN_EDUCATION
• Zajda, J. (2021). Globalisation and education reforms. Creating effective learning environments. Springer.
• Zhang, Y., & Wildemuth, B. M. (2005). Qualitative analysis of content. Human Brain Mapping, 30(7), 2197-2206.
• Zhang, Y., & Wildemuth, B. M. (2009). Unstructured interviews. In: B. Wildemuth (Ed.), Applications of social research methods to questions in information and library science (pp. 222-231). Libraries Unlimited.
• Zuluaga-Ortiz, R., Delahoz-Dominguez, E., Perinan-Luna, A., Escorcia, J., Moreira-Villegas, F., & Arteta, A. (2022). Evaluation of Educational Quality Under a Six Sigma Approach to Engineering Degrees in Colombia. In: K. Saeed, J. Dvorský (Eds.), Computer Information Systems and Industrial Management. CISIM 2022. Lecture Notes in Computer Science, 13293 (pp. 172-188). Springer. https://doi.org/10.1007/978-3-031-10539-5_13
• Zwikael, O., & Globerson, S. (2006). From critical success factors to critical success processes. International Journal of Production Research, 44(17), 3433-3449. https://doi.org/10.1080/00207540500536921