The Science Behind Game-Based Assessment

Traditional tests – think multiple-choice exams or lengthy questionnaires – have long been the standard for evaluating students and job candidates. But in recent years, game-based assessments (GBAs) have emerged as an innovative alternative. These assessments embed test content within game-like activities, aiming to measure skills and traits in a more interactive and engaging way. How do game-based assessments work, and are they as valid as traditional methods? This article delves into the science behind GBAs from an educational psychology and psychometrics perspective.

How Game-Based Assessments Work (and What They Measure)

At their core, game-based assessments are designed to stealthily collect data on a person's abilities or personality through gameplay. Instead of directly asking questions ("Are you a good problem-solver?"), a GBA might present a puzzle or scenario and infer problem-solving ability from the individual's actions. A well-known concept in this field is stealth assessment, coined by researchers like Valerie Shute. In a stealth assessment, the test is woven into the fabric of the game – often invisible to the player (Shute & Moore, 2017).

For example, consider a game where you navigate through challenges to save a virtual world. Behind the scenes, every decision, response time, strategy, and outcome is logged. Using an approach called evidence-centered design, test developers define which in-game behaviors indicate proficiency in the target skill. As the person plays, a statistical model (often Bayesian) continuously updates their score based on the evidence gathered.

Game-based assessments can measure a range of constructs: cognitive abilities (like memory, attention, problem-solving), knowledge in specific domains, and even personality traits or soft skills. Research indicates that these game indicators correlate with traditional cognitive measures – one meta-analysis found an observed correlation around r = 0.30 (which rose to r ≈ 0.45 after correcting for reliability) between game-based metrics and conventional cognitive test scores (Stoet & Graaf, 2023).

For personality and behavioral traits, GBAs often use situational gameplay to elicit natural behavior. Unlike self-report questionnaires that ask candidates to rate themselves (which can introduce social desirability bias), a game can prompt spontaneous reactions. Proponents argue this leads to more authentic data: candidates may "forget" they are being assessed and just play, potentially reducing faking (Equalture, 2022).

Comparison to Traditional Testing: Engagement, Anxiety, and Fairness

One big motivation for game-based assessment is to address some drawbacks of traditional tests. Engagement is a prime example. Traditional tests can be dry and stress-inducing; bored or anxious test-takers may not perform their best, which can undermine the validity of the results. GBAs, on the other hand, strive to be fun and immersive. Studies have found that embedding assessment in a game can increase test-taker engagement and even enjoyment (Equalture, 2022).

Test anxiety is another critical factor. High-stakes exams often spike anxiety, which can impair performance. Preliminary research suggests game-based assessments can mitigate test anxiety. In one case study, students who took an exam through a game interface reported lower nervousness and performed without the usual stress-related drop in scores (Bonefont et al., 2022). The interactive, low-pressure atmosphere of a game can cultivate a feeling of confidence and calm.

However, we must consider fairness and biases. One concern is the gaming familiarity factor: will people who are avid gamers perform better regardless of the target construct? Researchers have been examining this. Encouragingly, a study on cognitive GBAs found that while gaming experience had some effect, the assessments still showed criterion-related validity in predicting job performance (Tondello & Gutierrez, 2021). Cultural bias can potentially be reduced in GBAs by relying less on language and presenting challenges in a more universal way through visuals and interactions.

Validity and Reliability: Do GBAs Measure Up?

The ultimate question for any assessment is: Is it valid? That is, does it actually measure what it claims to measure, and can it predict outcomes of interest? Game-based assessments pose a challenge here because they are complex and often assess multiple things at once. Researchers approach validation through multiple avenues:

• Content validity: Ensuring the game's tasks truly reflect the skills to be measured. Using evidence-centered design documentation, developers explicitly link each game element to the competencies of interest.
• Construct and criterion validity: Statistical analysis comparing game-based scores to other measures. Meta-analyses have found moderate correlations between GBA scores and traditional test scores for cognitive abilities.
• Reliability: Ensuring that if someone took the game assessment twice, they'd get a similar score. By using robust design and evidence accumulation, one can achieve reliability on par with traditional assessments.

It's worth noting that not all game-based assessments automatically have strong validity – it depends on the quality of design and research behind them. Therefore, creators of GBAs typically go through extensive pilot testing and iteration, similar to traditional test development.

Conclusion

Game-based assessments represent a significant evolution in the approach to testing. The science behind them – from stealth assessment techniques to evidence of validity – is growing and largely encouraging. Studies have shown that GBAs can achieve reliability and measure targeted constructs, with moderate to strong correlations to established tests. They also bring added benefits of engagement, reduced anxiety, and richer data on performance processes.

The key takeaway from the current state of research is balance: leverage the strengths of game-based assessment while mitigating their weaknesses. By doing so, educators and employers can harness games not just for play, but for meaningful insight.