The effects of item characteristics and item Response Theory Model on the distribution and effectiveness of Person-Fit Indices

Abstract

The purpose of this study was to examine the effects of item difficulty, item discrimination and type of item response theory (IRT) model on the distribution and effectiveness of the standardized maximum likelihood index (LZ) and the fourth standardized extended caution index (ECIZ4). The study was based on Item response theory. Data was generated in eighteen combinations of item difficulty, item discrimination and type of IRT model to simulate the responses of 2000 non-aberrant examinees' response patterns to a 60-item test according to one-parameter (IPLM), two-parameter (2PLM), and three-parameter (3PLM) models. The 30% spuriously low aberrant response patterns were simulated by randomly modifying 18 items from correct to incorrect and the 30% spuriously high aberrant response patterns were simulated by randomly modifying 18 items from incorrect to correct of the first 500 response patterns. Item discrimination parameters (a) were represented by two conditions of low and high. Low a parameter ranged from .4 to 1.0 while high a parameter ranged from .7 to 1.3 in a uniform distribution. Item difficulty parameters (b) were represented by three conditions of high, moderate and low variances. The high b parameter ranged from -1.2 to +3.0 while the moderate b parameter ranged from -3 to +1.2, and the low b parameter ranged from -3.0 to +1.2 in a uniform distribution. The pseudo guessing parameter (c) ranged from 0.05 to .20 in a uniform distribution and the ability estimates (9) were generated from the normal (0,1) distribution. The examinee ability estimates, item parameters and response vectors were generated using the computer program, WinGen2 by Han, (2006) and the LZ and the ECIZ4 person-fit indices were computed using WPERFIT computer program developed by Urbano, (2010). MANOVA and ANOVA along with Sheffe's Post hoc tests were used to determine the relationship between independent variables and dependent variables. Using Pillai's statistic, the overall MANOVAs for the descriptive statistics and the percentile estimates were found to be significant at 0.05 level of significance. Further, the results showed that the mean, standard deviation, skewness, kurtosis, and the percentile estimates of LZ and ECIZ4 indices were affected by the variations of item difficulty and type of IRT model; but were robust against item discrimination parameter. The LZ index was slightly skewed to the left whereas the ECIZ4 index was slightly skewed to the right an indication that each index is skewed towards the direction of it's aberrance. In addition, the distribution of LZ and ECIZ4 indices were found to approximate a normal distribution under the 2PLM and the 3PLM models. The 30%' spuriously low treatment samples were more detectable by LZ and ECIZ4 indices than the 30% spuriously high treatment samples. Further, spuriously low treatment samples were more detectable under high item difficulty and spuriously high treatment samples were more detectable under low item difficulty parameters. LZ performed better than ECIZ4 under high item difficulty whereas ECIZ4 performed better than LZ under low and moderate item difficulties. For practical purposes, the recommendation was that LZ and ECIZ4 indices be used under 2PLM and 3PLM models.