Evidence Supporting a Validity Argument for an English Listening Comprehension Test: Two Prototyping Studies

Abstract

Before a new test can be implemented for operational use, extensive research is required to collect validity evidence that would support the use of scores from the new test. This paper describes two prototyping studies that were conducted as part of a test development project for a new general English listening comprehension test. The purpose of the prototyping studies was to make decisions about the selection of item types to be included in the test as well as to collect supporting evidence for a validity argument for the use of scores from the new listening test. The first prototyping study gathered evidence that would support the domain description inference, that is, whether the test tasks appropriately sample test takers general English listening comprehension ability. The second prototyping study also gathered evidence for the domain description inference but with a narrower focus on the specific configuration of one test task. For both studies, test takers' performance on shortened versions of the test and test takers' post-test questionnaire responses were examined. Results from the prototyping studies are presented, along with a description of how they informed decisions about items on the test as well as plans for further research that would produce additional validity evidence for the test.