Application of Confirmatory Factor Analysis to the Validity Study of a Performance Assessment: A Multitrait-multimethod Structure and its Invariance across Gender and Grade

Abstract

This study investigated construct validity and factorial invariance of a performance

assessment of reading comprehension and writing proficiency, through a

multitrait-multimethod structure (MTMM), using the confirmatory analysis technique. First,

interrater reliability was examined for each measured variable using three different

generalizability coefficients. Although all of the measures were found to be highly reliable,

exploratory factor analysis indicated that trait and method effects were confounded in the

measured variables. Consequently, confirmatory factor analysis was used to disentangle

multidimensionality and examine the convergent and discriminant validity of the latent

variables according to the Campbell-Fiske criteria. These analyses indicated that a model with

three correlated trait factors and three correlated method factors provided the best fit to the

data. Finally, a factorial invariance across gender and grade was examined. While this

MTMM factorstructure was fitted to the data in each subgroup (fifth grade boys. fifth grade

girls, sixth grade girls), the factorial invariance across gender and grade was supported only

in a particular set of parameters. Methodological and practical implications of the use of

confirmatory factor analysis in multitrait-multimethod analyses are also discussed for

construct validation in performance assessment across different groups.