Ability Estimation Using the Classical Test Theory and Three-Parameter Item Response Theory Model

Bayah  Amiruddin,; Mercedita  Langamin,

Ability Estimation Using the Classical Test Theory and Three-Parameter Item Response Theory Model

Discipline: Education

Abstract:

This study aimed to estimate and compare the academic abilities of the students using the Classical Test Theory (CTT) and the Item-Response Theory (IRT) models. Specifically, the Bayesian modal or maximum a posteriori (MAP) under the three- parameter logistic model of IRT was used as ability estimator. The study involved a total of 52 Grade 11 students in a descriptive-quantitative approach using Microsoft Excel, R, and SPSS software in the data calculation. The findings revealed that 30 (57.69%) students passed and 22 (42.31%) failed under the CTT approach. However, under the IRT approach, 34 (65.38%) passed while the remaining 18 failed. Moreover, when the CTT and IRT results were compared, it produced significant results indicating that transmuted grades of the students were higher under IRT than CTT. Furthermore, the results of the two approaches showed significant relationship further providing a better picture of the comparison. In this study, IRT appeared to provide more details/information about student performances/abilities than the CTT and thus the use of IRT in the ability estimation is recommended. In addition, Bayesian modal or MAP appeared to work well as ability estimator and thus itsemployment is likewise encouraged.

References:

Brouwer, D., Meijer, R. & Zevalkink, J. (2013). Measuring individual significant change on the Beck depression inventory-II through IRT-based statistics. Psychotherapy Research, 23, 489-501.
Brownlee, J. (2018). A gentle introduction to Maximum a Posteriori (MAP) for machine learning. Machine learning mastery. https://machinelearningmastery.com/maximum-a-posteriori-estimation/
Cassady, J.& Johnson, R. (2002). Cognitive test anxiety and cognitive performance.
Contemporary Educational Psychology, 27(2), 270-295. https://doi.org/10.1006/ceps.2001.1094
Cuervo, E. & Andrade, J. (2004). Modelling abilities in 3-IRT models. Revista Colombiana de
Estadistica, 27(1), 27- 41.
De Ayala, R. (2009). The theory and practice of item response theory. New York, NY: The Guilford Press.
Demars, C. (2010). Item response theory. Oxford: Oxford University Press.
Embretson, S. & Reise, S. (2000). Item response theory for psychologists. New York, NY. Psychology Press. https://doi.org/10.4324/9781410605269
Falmagne, J. (1989). A latent trait theory via a stochastic learning theory for a knowledge space. Psychometrika, 54, 283–303. https://doi.org/10.1007/BF02294521
Fan, X. (1998). Item response theory and classical test theory: An empirical comparison of their item/person statistics. Educational and Psychological Measurement, 58(3), 357-382. https://doi.org/10.1177/0013164498058003001
Hambleton, R., Swaminathan, H., & Rogers, H. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage Press.
Hambleton, R. & Swaminathan, H. (1985). Item response theory principles and applications. Boston, MA: Kluwer Nijhoff Publishing.
Harris, D. (1989). Comparison of 1-, 2-, and 3-parameter IRT models. Educational Measurement: Issues and Practice, 8(1), 35-41. https://doi.org/10.1111/j.1745-3992.1989.tb00313.x
Jabrayilov, R., Emons, W. & Sijtsma, K. (2016). Comparison of classical test theory and item response theory in individual change assessment. Applied Psychological Measurement, 40(8), 559-572. https://doi.org/10.1177/0146621616664046
Janssen, G., Meier, V. & Trace, J. (2014). Classical test theory and itemresponse theory: Two understandings of one high-stakes performance exam. Columbian Applied Linguistics Journal, 6(2), 167-184. https://doi.org/10.14483/udistrital.jour.calj.2014.2.a03
Johnson, N. (2004). From classical test theory to item response theory: An introduction to a desirable transition. In O.A. Afemikhe and J.G. Adewale (Eds.), Issues in educational measurement and evaluation in Nigeria. 371-384. University of Ibadan.
Leonard, K. (Ed.). (2005). Encyclopedia of Social Measurement. Elsevier. https://doi.org/10.1016/B0-12-369398-5/00198-5
Magno, C. (2009). Demonstrating the difference between classical test theory and item response theory using derived data. The International Journal of Educational and Psychological Assessment, 1(1), 1-11.
National Council on Measurement Education. (2017). Glossary of important assessment and measurement terms. https://web.archive/20170722194028/
Prieler, J. (2007). So wrong for so long: Changing our approach to change. The Psychologist, 20, 730-732.
Reise, S., & Waller, N. (2009). Item response theory and clinical measurement. Annual Review of Clinical Psychology, 5, 27-48. https://doi.org/10.1146/annurev.clinpsy.032408.153553
Rubio, D. (2005). Alpha reliability. In K. Leonard (Ed.), Encyclopedia of Social Measurement. 59-63. Elsevier.
Sallil, U. (2017). Estimating examinees’ ability in a computerized adaptive testing and non-adaptive testing using three- parameter logistic IRT model [Unpublished master’s thesis). Mindanao State University - Iligan Institute of Technology, Iligan City, Philippines.
Sebille, V., et al. (2010). Methodological issues regarding power of classical test theory and IRT-based approaches for the comparison of patient-reported outcome measures – A simulation study. Medical Research Methods, 10(1), 1-10. https://doi.org/10.1186/1471-2288-10-24
Steyers, R. (2001). Classical test theory. In N. Smelser and P. Baltes (Eds.), International Encyclopedia of the Social and Behavioral Sciences. 1955-1962. Elsevier.
Swaminathan, H., & Gifford, J. (1986). Bayesian estimation in the three parameter logistic model. Psychometrika, 51, 589–601. https://doi.org/10.1007/BF02295598
Thomas, M. (2010). The value of item response theory in clinical assessment: A review. Sage, 18(3), 291-307. Doi 110.1177/1073191110374797
Thompson, N. (2021). Classical test theory vs. item response theory. Smarter Assessment. https://assess.com/classical-test-theory-vs-item-response-theory/
Thompson, N. (2018). What is the three parameter IRT model (3PL)? Smarter Assessment. Https://assess.com/three-parameter-irt-3pl-model/
Tsun, A. (2022). Maximum a posteriori estimation. Stanford University News https://web.standford.edu/class/archive/cs/cs109/cs109.1218/files/student_drive/7.5.pdf