Investigating Complex Interaction Effects Among Facet Elements in an ESL Writing Test Consisting of Integrated and Independent Tasks

Abstract

The main purposes of the current study are to: (a) examine the interactional effects among test-takers, tasks, and raters, as well as the main effects of these facets, in an ESL writing test consisting of both integrated and independent writing tasks and (b) thereby identify additional sources of score variability and error in the rating of test-taker responses. A total of 162 test-takers with 29 different L1 backgrounds participated in the study, each of whom took the same six writing tasks, which included 3 Listening-Writing (LW), 2 Reading-Writing (RW), and 1 Independent Writing (IW) tasks. Each of the essays was rated by each of the 6 trained raters to obtain a completely-crossed data matrix for the test-takers, tasks, and raters. A computer program, FACETS (Linacre, 1998), was used to calibrate the test-takers, tasks, and raters and conduct interaction analysis on 970 essays. Results of the analyses revealed that raters seemed to be having slight difficulty in maintaining a consistent level of severity across all of the 6 tasks and that a close inspection of the rating patterns of selected test-takers demonstrated the usefulness of interaction analysis in pinpointing particular combinations of facet elements with unusual interaction patterns. The implications of these findings for writing assessment are discussed along with the avenues for further investigation.