USTW Vs. STW: A Comparative Analysis for Exam Question Classification based on Bloom’s Taxonomy

Mohammed Osman Gani; Ramesh Kumar Ayyasamy; Anbuselvan Sangodiah; Yong Tien Fui

doi:10.13164/mendel.2022.2.025

Mohammed Osman Gani Faculty of Information and Communication Technology, Universiti Tunku Abdul Rahman, Kampar, Malaysia https://orcid.org/0000-0002-9918-8589
Ramesh Kumar Ayyasamy, PhD Department of Information Systems, Faculty of Information and Communication Technology, Universiti Tunku Abdul Rahman, Kampar, Malaysia https://orcid.org/0000-0002-3538-3887
Anbuselvan Sangodiah, PhD School of Computing, Faculty of Computing and Engineering, Quest International University Perak, Ipoh, Malaysia https://orcid.org/0000-0002-3277-4006
Yong Tien Fui Department of Information Systems, Faculty of Information and Communication Technology, Universiti Tunku Abdul Rahman, Kampar, Malaysia https://orcid.org/0000-0001-6752-5985

DOI: https://doi.org/10.13164/mendel.2022.2.025

Keywords: Bloom’s Taxonomy, Exam Question Classification, Supervised, Term Weighting, TF-IDF, TF-ICF, Unsupervised

Abstract

Bloom’s Taxonomy (BT) is widely used in educational institutions to produce high-quality exam papers to evaluate students’ knowledge at different cognitive levels. However, manual question labeling takes a long time, and not all evaluators are familiar with BT. The researchers worked to automate the exam question classification process based on BT as a solution. Enhancement in term weighting is one of the ways to increase classification accuracy while working with text data. However, all the past work on the term weighting in exam question classification focused on unsupervised term weighting (USTW) schemes. The supervised term weighting (STW) schemes showed effectiveness in text classification but were not addressed in past studies of exam question classification. As a result, this study focused on the effectiveness of STW in classifying exam questions using BT. Hence, this research performed a comparative analysis between the USTW schemes and STW for exam question classification. The STW schemes used in this study are TF-ICF, TF-IDF-ICF, and TF-IDF-ICSDF, whereas the USTW schemes used for comparison are TF-IDF, ETF-IDF, and TFPOS-IDF. This study used Support Vector Machines (SVM), Na¨ıve Bayes (NB), and Multilayer Perceptron (MLP) to train the model. Accuracy and F1 score were used in this study to evaluate the classification result. The experiment result showed that overall, the STW scheme TF-ICF outperformed all the other schemes, followed by the USTW scheme ETF-IDF. Both the ETF-IDF and TFPOS-IDF outperformed standard TFIDF. The outcome of this study indicates the future research direction where the combination of STW and USTW schemes may increase the Accuracy of BT-based exam question classification.

References

Abduljabbar, D. A., and Omar, N. Exam questions classification based on bloom’s taxonomy cognitive level using classifiers combination. Journal of Theoretical and Applied Information Technology 78, 3 (2015), 447.

Alsaeedi, A. A survey of term weighting schemes for text classification. International Journal of Data Mining, Modelling and Management 12, 2 (2020), 237–254.

Aninditya, A., Hasibuan, M. A., and Sutoyo, E. Text mining approach using tf-idf and naive bayes for classification of exam questions based on cognitive level of bloom’s taxonomy. In 2019 IEEE International Conference on Internet of Things and Intelligence System (Io-TaIS) (2019), IEEE, pp. 112–117.

Chen, K., Zhang, Z., Long, J., and Zhang, H. Turning from tf-idf to tf-igm for term weighting in text classification. Expert Systems with Applications 66 (2016), 245–260.

Chen, L., Jiang, L., and Li, C. Modified dfs-based term weighting scheme for text classification. Expert Systems with Applications 168 (2021), 114438.

Chen, L., Jiang, L., and Li, C. Using modified term frequency to improve term weighting for text classification. Engineering Applications of Artificial Intelligence 101 (2021), 104215.

Debole, F., and Sebastiani, F. Supervised term weighting for automated text categorization.In Proceedings of the 2003 ACM symposium on Applied computing (2003), pp. 784–788.

Dogan, T., and Uysal, A. K. Improved inverse gravity moment term weighting for text classification. Expert Systems with Applications 130 (2019), 45–59.

Domeniconi, G., Moro, G., Pasolini, R., and Sartori, C. A study on term weighting for text categorization: A novel supervised variant of tf. idf. In DATA (2015), pp. 26–37.

Gu, Y., and Gu, X. A supervised term weighting scheme for multi-class text categorization. In International Conference on Intelligent Computing (2017), Springer, pp. 436–447.

Jayakodi, K., Bandara, M., Perera, I., and Meedeniya, D. Wordnet and cosine similarity based classifier of exam questions using bloom’s taxonomy. International Journal of Emerging Technologies in Learning 11, 4 (2016).

Joachims, T. Text categorization with support vector machines: Learning with many relevant features. In European conference on machine learning (1998), Springer, pp. 137–142.

Kavadi, D. P., Ravikumar, P., and Srinivasa Rao, K. A new supervised term weight measure for text classification. International Journal of Advanced Science and Technology 29, 6 (2020), 3115–3128.

Lan, M., Tan, C. L., Su, J., and Lu, Y. Supervised and traditional term weighting methods for automatic text categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 4 (2008), 721–735.

Mazyad, A., Teytaud, F., and Fonlupt, C. A comparative study on term weighting schemes for text classification. In International Workshop on Machine Learning, Optimization, and Big Data (2017), Springer, pp. 100–108.

Mohammed, M., and Omar, N. Question classification based on bloom’s taxonomy using enhanced tf-idf. International Journal on Advanced Science, Engineering and Information Technology 8 (2018), 1679–1685.

Mohammed, M., and Omar, N. Question classification based on bloom’s taxonomy cognitive domain using modified tf-idf and word2vec. PLOS ONE 15, 3 (2020), e0230442.

Omar, N., Haris, S. S., Hassan, R., Arshad, H., Rahmat, M., Zainal, N. F. A., and Zulkifli, R. Automated analysis of exam questions according to bloom’s taxonomy. Procedia-Social and Behavioral Sciences 59 (2012), 297–303.

Osman, A., and Yahya, A. Classifications of exam questions using linguistically-motivated features: a case study based on bloom’s taxonomy. In The Sixth International Arab Conference on Quality Assurance in Higher Education (IACQA’2016) (2016), vol. 467, p. 474.

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. Scikit-learn: Machine learning in python. Journal of Machine Learning Research 12 (2011), 2825–2830.

Quan, X., Wenyin, L., and Qiu, B. Term weighting schemes for question categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 5 (2010), 1009–1021.

Ren, F., and Sohrab, M. G. Class-indexingbased term weighting for automatic text classification. Information Sciences 236 (2013), 109–125.

Sangodiah, A., Ahmad, R., and WAN AHMAD, W. F. Taxonomy based features in question classification using support vector machine. Journal of Theoretical & Applied Information Technology 95, 12 (2017).

Sangodiah, A., Fui, Y. T., Heng, L. E., Jalil, N. A., Ayyasamy, R. K., and Meian, K. H. A comparative analysis on term weighting in exam question classification. In 2021 5th International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT) (2021), IEEE, pp. 199–206.

Sangodiah, A., San, T. J., Fui, Y. T., Heng, L. E., Ayyasamy, R. K., and Jalil, N. A. Identifying optimal baseline variant of unsupervised term weighting in question classification based on bloom taxonomy. MENDEL 28, 1 (2022), 8–22.

Wang, D., and Zhang, H. Inversecategory-frequency based supervised term weighting scheme for text categorization. Journal of Information Science and Engineering 29 (2013), 209–225.

Yahya, A. A., and Osman, A. Automatic classification of questions into bloom’s cognitive levels using support vector machines. In The International Arab Conference on Information Technology (2011).

Yahya, A. A., Osman, A., Taleb, A., and Alattab, A. A. Analyzing the cognitive level of classroom questions using machine learning techniques. Procedia-Social and Behavioral Sciences 97 (2013), 587–595.

Yahya, A. A., Toukal, Z., and Osman, A. Bloom’s taxonomy–based classification for item bank questions using support vector machines. In Modern Advances in Intelligent Systems and Tools. Springer, 2012, pp. 135–140.

Yusof, N., and Hui, C. J. Determination of bloom’s cognitive level of question items using artificial neural network. In 2010 10th International Conference on Intelligent Systems Design and Applications (2010), IEEE, pp. 866–870.