Eye Tracking Reading Comprehension Tests

The “jingle fallacy” refers to the flawed assumption that tests sharing a similar name must measure the same underlying construct. In their recent research paper, “The jingle fallacy in comprehension tests for reading,” Lee, Godwin, and Drieghe (2024) provide an example of the fallacy in the context of reading comprehension. The study, published in PLOS ONE, highlights significant discrepancies between two commonly used comprehension tests, the Nelson Denny Reading Test (NDRT) and the Wechsler Individual Achievement Test (WIAT-II), particularly in their ability to predict skilled adult readers’ eye movement patterns under varying comprehension demands.

Eye Tracking Reading Comprehension

A critical element of the research was the use of an SR Research EyeLink 1000, which provided invaluable insights into real-time reading strategies. By recording participants’ eye movements during the comprehension tasks, the researchers could observe nuanced changes in reading behavior, such as the number of fixations, average fixation duration, and forward saccade length. These objective measures allowed for a deeper understanding of how readers adapt their strategies when faced with higher or lower comprehension demands, and how individual differences in comprehension ability manifest in reading patterns.

The study involved 91 university students who read passages while their eye movements were recorded. Participants completed two experimental blocks—one with higher and one with lower comprehension demands.

Nelson Denny Reading Test Predicts Eye Movement Patterns

The key finding revealed that while both comprehension tests aim to measure reading comprehension, only the NDRT was a reliable predictor of the observed eye movement patterns. Specifically, high scorers on the NDRT consistently increased their pace of reading over time, making fewer fixations and longer saccades, adapting their strategy efficiently across both high and low comprehension demands. In contrast, low scorers on the NDRT reached a ceiling in their reading speed and ability to reduce fixations, especially in low comprehension demand conditions.

Crucially, the WIAT-II comprehension test did not show similar predictive power regarding eye movement patterns. This disparity underscores the jingle fallacy: despite both tests being labeled “reading comprehension,” they appear to tap into different underlying skills or are affected by different confounding factors. 

The detailed eye movement data provided a granular view of reading processing that traditional accuracy scores alone could not. For instance, the eye-tracking data revealed that readers generally adapted a more “careful” strategy (more fixations, longer durations, shorter saccades) when comprehension demands were high, and a more “risky” strategy (fewer fixations, shorter durations, longer saccades) when demands were low. These subtle shifts in reading behavior, quantified through eye movement metrics, provided direct evidence of how readers adjust their cognitive effort in real-time.

This research highlights the critical importance of carefully selecting comprehension tests for research, particularly when investigating skilled adult reading and eye movement behaviors. The precise and objective data provided by eye tracking technology was fundamental in uncovering these discrepancies, demonstrating its essential role in advancing our understanding of reading comprehension and individual differences.

For information regarding how eye tracking can help your research, check out our solutions and product pages or contact us. We are happy to help!