AI screening tools are filtering out candidates you would have hired. Here's what we found when we tested five of them.

We ran an internal experiment over a six-week period. The setup: we took fifty real CVs from a senior operations role we'd recently worked on — a brief where we already knew which candidate the client had hired and how the placement turned out. We ran the same fifty CVs through five widely-used AI screening tools. We then compared each tool's ranking against the outcome we already knew.

We're not naming the tools. They have lawyers. We will share the pattern, because the pattern matters more than the specific products.

Three findings, in order of importance

The first finding is that the tools mostly agreed with each other on the top ten candidates and mostly agreed on the bottom ten. The candidates who were obviously well-qualified and the candidates who were obviously off-brief got ranked consistently across all five tools. If you're using AI screening as a first-pass filter on a high-volume role, this is genuine value. The tools are removing clearly-irrelevant CVs at scale, which they should.

The second finding is that the tools disagreed sharply in the middle thirty candidates. This is the band where most actual hires come from — the candidates who aren't obvious yes or no, where human judgement does most of the work. In our test, one tool ranked the candidate who actually got hired in position 27. Another tool ranked her in position 4. A third tool ranked her in position 19. There's no consensus middle ground in the tools' models, which means the tools are guessing in the band that matters most.

The third finding, and the one we found most concerning, was the specific reasons the tools downgraded the candidate who got hired. She had a fourteen-month career gap eight years ago — a maternity break that she'd been transparent about and that hadn't affected her career trajectory. Three of the five tools penalised her for the gap. She'd also spent two years at a smaller company that none of the tools had seen many CVs from before. Four of the five tools downgraded her for this. The fifth, which used a slightly different model, didn't. By the time the algorithmic penalties stacked, her ranking was low enough that some hiring managers wouldn't have looked.

That candidate is a year into the role now and is, by the hiring manager's own description, the strongest senior hire the company has made in three years.

What we take from this

AI screening tools are useful for the volume part of recruitment. They are dangerous as the primary filter on senior or specialist roles, because the structural biases in the models are systematic, not random. Career gaps get penalised. Non-traditional career paths get penalised. Smaller employers and less-common job titles get penalised. The candidates who get through cleanly are the candidates whose CVs look like the CVs the model was trained on, which means companies relying heavily on AI screening end up with a less diverse, less interesting hiring pool than they think they have.

A small admission. We use AI in our own workflow — we said as much in an earlier post. We don't use it as the primary filter. We use it to surface candidates we might have missed by running broader pattern matches against a brief, and then we run human screening on everything it surfaces. The order matters. AI as a candidate-surfacing tool is useful. AI as a candidate-rejecting tool is risky.

If you're a hiring manager wondering whether the AI screening your ATS does is filtering out people you'd have hired, here's the answer. Yes, almost certainly, at the margin. The question is whether the volume efficiency is worth the cost. For the right role, it might be. For the senior role where one good hire is worth more than fifty mediocre interviews, it probably isn't.

— Kern & Monique