This eye-tracking study aimed at investigating test-wiseness strategies used by L2 learners when answering 4-option multiple-choice items and 4-option multiple true-false items in an L2 listening test. In particular, the study explored (a) the extent to which L2 learners performed differently on the two items types, (b) the extent to which L2 learners used test-wiseness strategies for the two item types, and (c) the extent to which the use of test-wiseness strategies introduced construct-irrelevant variance for the two item types. To address these goals, three types of data—namely, test score data, eye-tracking data, and verbal report data—were gathered from 40 English-as-a-second-language learners at a large public university in the Pacific region. The data were analyzed both quantitatively and qualitatively using scanpath analysis. The findings revealed that (a) multiple true-false items were more difficult than multiple-choice items, (b) test-wiseness strategies were used less frequently for answering multiple true-false items than for answering multiple-choice items, and (c) the use of test-wiseness strategies had a statistically significant effect on the observed scores for both item types. The study has implications for test item design and highlights the importance of gathering validity evidence based on response processes.
Ruslan Suvorov, University of Western Ontario, Canada