A recent study has provided valuable insights into the psychometric properties and potential biases of using machine learning algorithms to evaluate asynchronous video interviews (AVIs) for personality traits and interview performance. As companies increasingly adopt AVIs to streamline hiring processes, understanding the efficacy and fairness of these technologies has become crucial. The findings were published in the journal Computers in Human Behavior.
Researchers from Tilburg University and Vrije Universiteit Amsterdam analysed data from 710 participants who completed mock AVIs for the study. Each participant responded to eight personality-focused questions designed to assess traits like Extraversion and Conscientiousness. The researchers collected self-reports, observer reports of personality, and interview performance ratings. They then used machine learning algorithms to predict personality traits and interview performance based on extracted audio, visual, and verbal features from the videos.
The study revealed that the algorithm could predict observer-reported Extraversion and Conscientiousness with substantial accuracy, explaining 32% of the variance for these traits. In contrast, self-reported personality traits showed lower predictability, with the algorithm explaining only 12% of the variance. For interview performance, the algorithm explained 44% of the variance, indicating a strong potential for AVIs in evaluating candidates.
Consistent with Trait Activation Theory, the algorithm’s accuracy improved when participants answered trait-relevant questions. This finding underscores the importance of designing interview questions that effectively elicit specific personality traits. The algorithm also demonstrated moderate test-retest reliability over seven months, which, although lower than desired for personnel selection, suggests some level of stability in the assessments.
Algorithmic bias is a significant issue that the study highlights. While the algorithm did not introduce new biases in most cases, it did amplify existing gender biases in favour of women. Higher interview performance scores for women served as proof of this, and the AI-based assessments further widened the gap. Such biases pose potential legal and ethical issues, especially since gender is a protected characteristic in many jurisdictions.
The study also examined biases related to age and attractiveness. Interestingly, while human raters exhibited a bias favouring more attractive participants, the algorithm attenuated this effect, suggesting a potential area where AI can reduce human biases. However, biases introduced by video meta-information, such as professional attire and audio quality, still pose challenges, indicating the need for careful consideration of these factors in AVIs.
The findings of this study have important implications for both researchers and practitioners. For companies, the results suggest that AVIs, when paired with machine learning algorithms, can effectively assess interview performance and certain personality traits. However, the presence of algorithmic biases highlights the need for ongoing monitoring and adjustment of these systems to ensure fairness and compliance with anti-discrimination laws.
For researchers, the study underscores the necessity of further exploration into the sources of algorithmic bias and the development of more robust models. Future studies could focus on enhancing the reliability of these assessments and expanding the range of traits evaluated by the algorithms.
