Resistance to coaching in forced-choice testing

In forced-choice tests (FCTs), examinees are typically presented with questions with two equally plausible answer alternatives, of which only one is correct. The rationale underlying this test is that guilty examinees tend to avoid relevant crime information, producing a nonrandom response pattern. The validity of FCTs is reduced when examinees are informed about this underlying rationale, with coached guilty examinees refraining from avoiding the correct information but trying to provide a random mix of correct and incorrect answers. To detect such intentional randomization, a "runs" test-looking at the distribution of the number of alternations between correct and incorrect answers-has been suggested but with limited success. We designed a runs test based on distinguishing between patterns that look random and patterns that are random. Specifically, we alternated the horizontal presentation (i.e., presentation left or right on the screen) of the correct answer alternative between each trial. As a consequence, guilty examinees were faced with having to choose to randomize either between correct and incorrect answers-leading to chance performance-or between answers presented on the left or right, producing a pattern that "looks" random. As innocent examinees are unaware of the correct answers, they can only randomize between horizontal positions. Results showed that the number of correct items selected distinguished guilty from innocent examinees only when they were not informed about the underlying rationale. In contrast, alternations between correct and incorrect answers did distinguish informed guilty from innocent examinees. Incremental validity of the alternation criterion and theoretical implications are discussed.