Rethinking rule diversity in figural matrices: A log file analysis on the role of task switching and implications from a validation study

Abstract

Figural matrices typically consist of multiple distinct logical rules, requiring test-takers to disengage from one rule before applying the next. Prior studies have consistently shown moderate associations between matrix performance and task-switching ability. However, these findings are largely based on correlational data. The present two-study article aimed to (a) determine, on a theoretical level, whether task-switching ability is func tionally involved in matrix processing, and (b) assess, from a diagnostic perspective, whether relaxing the constraint of distinct rules within a single matrix threatens psychometric validity. To this end, we manipulated matrices to include both distinct-rule and identical-rule transitions, enabling experimental within-subject com parisons of matrix processing in both conditions based on log file analyses. In study 1 (N = 209), task-switching ability exerted a functional influence only during distinct-rule transitions. However, the correlation between task-switching ability and matrix performance remained comparably strong even during identical-rule transi tions. This dual pattern supports both a switch-dependency hypothesis (i.e., that task-switching is functionally involved in matrix processing) and a shared-resource hypothesis (i.e., that task-switching and matrix processing draw on a common cognitive resource). In study 2 (N = 258), we evaluated the convergent validity of the newly designed mixed-rule format against the traditional distinct-rule format. Test scores were highly correlated (r = 0.87), and test characteristics (e.g., reliability, IRT and TIF parameters) and external validity were very similar. Taken together, these findings suggest that although task-switching demands can vary depending on matrix design, relaxing rule constraints does not compromise psychometric validity. This flexibility in item development may be particularly useful in large-scale assessments or student selection tests that require continuous item renewal.