Most test data is tossed as a single number with an attached label, having delivered about the minimum amount of information that the test data could have delivered. Well-constructed cognitive test batteries produce profiles of strengths and weaknesses of different cognitive abilities. It is the pattern of scores on the subtests that usually contains the most useful information, including the gaps between them.
Separating the domains that get collapsed into one number
Scores from different ability subtests usually are averaged together to produce a single score, known as a composite score. However, scores from different subtests often correlate at only modest levels. In many cases, the pathways to producing the same score value are different for different people. Thus, scores from different subtests need to be reported.
- Logical reasoning, the capacity to hold premises, apply rules, and reach a conclusion that does not depend on prior knowledge of the content.
- Pattern recognition, the extraction of regularity from incomplete information, which drives performance on matrix and series tasks.
- Working memory, the amount of information that can be held and manipulated at once, distinct from long-term recall.
- Processing speed, the rate at which simple, well-understood operations are executed accurately.
- Problem solving, the coordination of the above when the method is not supplied.
Why the gaps are the signal
A strong Reasoning score paired with very low Working Memory scores would indicate someone who can solve very difficult problems on paper but whose accuracy suffers dramatically in meetings, verbal briefs, etc. Where information is not written down. On the other hand, a high Working Memory span paired with very slow Reasoning would indicate someone who appears reliable in completing procedural type tasks but whose reliability would quickly fail when those procedures no longer apply.
Join The European Business Briefing
New subscribers this quarter are entered into a draw to win a Rolex Submariner. Join 40,000+ founders, investors and executives who read EBM every day.
SubscribeWhat each domain predicts, and what it does not
Processing speed is probably the most over-read measure in all applied settings. It’s a great predictor for doing a lot of highly structured work over time, but it’s a very poor predictor of problem solving on novel problems, and it can be greatly affected by sleep, by medications, by mood, and by practice. Therefore, single session measures of processing speed are not very reliable or useful.
Like working memory, pattern recognition is a relatively stable measure but also of limited scope. In addition to indicating efficient abstraction, high scores on pattern recognition do not indicate whether the recognized pattern is correct or not. Rather, this is a function of executive control.
Reading problem-solving tasks properly
Strategies are most frequently observed on open-ended problem-solving tasks. Include in the report of test results a description of how the candidate tackled the problem, whether there was any indication of revising an approach that was not working, and whether there were any indicators that the candidate was willing to abandon a sunk cost approach to problem-solving. As noted above, such indicators transfer better to work than does the test score itself.
Matching the instrument to the question
The nature of the decisions one is trying to support should also determine the medium (e.g. Paper and pencil, digital, etc.) through which the assessment is to be given. It is worth comparing the established options for cognitive testing before commissioning anything, since interesting but unusable data is typically generated by those who settle on a format without first deciding what will best suit their purposes.
| Format | Best suited to | Main limitation |
| Unproctored online screening | High-volume sifting, early-stage filtering | Identity and environment are unverified, and speed data is unreliable |
| Proctored timed battery | Comparative selection against a norm group | Anxiety and test familiarity inflate variance |
| Full clinical neuropsychological assessment | Diagnostic questions, suspected decline, injury follow-up | Cost and duration, plus the need for qualified interpretation |
| Repeated brief testing | Tracking change over time, treatment response | Practice effects must be modeled, not ignored |
Norms do more work than the test does
A raw score from a cognitive assessment means very little. Just as is the case for measuring height, one needs a comparison group. How a score will compare with that of others will depend on a variety of factors including the age, level of education, and even primary language of others. This is a rather obvious point but one that is so frequently not taken into account in assessment. Using a general normative sample on a specialized population is a gross source of error that typically goes in one direction.
Interpretation errors that survive because they sound reasonable
Despite their questionable merit, several practices have become entrenched in organizational life and are used to buttress givens.
- Treating a difference of a few points between subtests as meaningful without reference to the standard error of measurement.
- Reading a low score as a ceiling rather than as one observation taken under specific conditions on a specific day.
- Using cognitive data to explain interpersonal behavior, which it does not measure.
- Ignoring effort and engagement indicators, which change the meaning of every other number in the profile.
When we have generated hypotheses based upon the performance data from the cognitive assessment, then test them against appropriate independent data (e.g., work samples, structured interviews, observed behavioral samples), then we can accept that the assessment has provided us with something useful (i.e., support for our generated hypotheses). And, of course, the test data can also refute our hypotheses, which tells us that we need to investigate conditions rather than capacity.
Where the value actually sits
Cognitive testing forms specific hypotheses about information use, which can be verified against other evidence. Therefore, rather than be used as conclusions in themselves, results from cognitive testing should form the basis of specific, testable questions about an individual’s information use. Results from such tests are the quickest and most reliable way of reducing many possible explanations for poor performance to a smaller number.
