Does the experimental set affect TOEFL iBT reading performance?

Abstract

This study investigates whether including an experimental set and test takers being aware of it influences TOEFL iBT reading performance. Before going into the experiment, the existence and the purpose of the experimental set is established based on several grounds. To discover a certain item or test functioning differently (DIF) in favor of some test takers or test layout that is meant to be comparable, the concepts of the Mantel-Haenszel procedure (Mantel and Haenszel, 1959) is borrowed. Test scores of participants who are designated into the reference group, focal group 1, and focal group 2 were compared but statistically revealed little regarding the effect of the experimental set. The qualitative approach taken after the test, however, exposed some critical issues related to fairness, performance, and ethics. Along with discussions, some implicit and improvable limitations finally conclude the paper.