Construct Validity in Human Scoring and Criterion: What Criterion would(not) measure

Abstract

This is a pilot study which aims to examine the reliability of automated essay scoring (AES) and to investigate validity of construct that Criterion would/would not measure. Criterion assessed iBT TOEFL independent writing tasks by comparing human raters evaluation. In particular, the current study explored which essay features were most closely related to each of the six different analytic dimensions for e-rater (Criterion). Five types of prompts were employed to assign a writing test to fifty college students in Seoul. The result showed that the agreement between human-rater and Criterion is moderate. In addition, three essay features (development, organization, and grammar) were crucial factors to predict the holistic score in human rating. Grammar, however, was a powerful predictor to tell the whole score in AES, which reflects that development and organization were not evaluated appropriately in Criterion. This result suggests that the feature dimensions in e-rater need to be refined/revised in the development and organization construct dimensions. The findings have some implications in teaching students process writing and using AES.