S-Space Language Education Institute (언어교육원) Language Research (어학연구) Language Research (어학연구) Volume 51 Number 1/3 (2015)
Investigating Complex Interaction Effects Among Facet Elements in an ESL Writing Test Consisting of Integrated and Independent Tasks
- Lee, Yong-Won; Kantor, Robert
- Issue Date
- 서울대학교 언어교육원
- 어학연구, Vol.51 No.3, pp. 653-678
- ESL writing assessment; integrated tasks; independent tasks; main effects; interaction effects; Many-faceted Rasch Measurement; Facets
- The main purposes of the current study are to: (a) examine the interactional effects among test-takers, tasks, and raters, as well as the main effects of these facets, in an ESL writing test consisting of both integrated and independent writing tasks and (b) thereby identify additional sources of score variability and error in the rating of test-taker responses. A total of 162 test-takers with 29 different L1 backgrounds participated in the study, each of whom took the same six writing tasks, which included 3 Listening-Writing (LW), 2 Reading-Writing (RW), and 1 Independent Writing (IW) tasks. Each of the essays was rated by each of the 6 trained raters to obtain a completely-crossed data matrix for the test-takers, tasks, and raters. A computer program, FACETS (Linacre, 1998), was used to calibrate the test-takers, tasks, and raters and conduct interaction analysis on 970 essays. Results of the analyses revealed that raters seemed to be having slight difficulty in maintaining a consistent level of severity across all of the 6 tasks and that a close inspection of the rating patterns of selected test-takers demonstrated the usefulness of interaction analysis in pinpointing particular combinations of facet elements with unusual interaction patterns. The implications of these findings for writing assessment are discussed along with the avenues for further investigation.