Most interviews for AI-adjacent engineering roles still open with a coding exercise. It is a fair test of whether someone can write code. It is a poor test of the thing that actually goes wrong once a system is in front of a client.
What actually breaks
The failures worth worrying about are not syntax or speed. They are a model producing confident output that nobody checked. A pipeline that cannot show where a number came from. An evaluation written after the build to justify it, rather than before it to constrain it.
None of that is caught by watching someone solve a puzzle quickly.
Code generation is now the cheapest part of the job. Which means the differentiator has moved to the person deciding whether the generated thing should ship — and whether they can explain that decision to someone who was not in the room.
Test review, not build
The most useful exercise is not a build task. It is a review task.
Hand the candidate a piece of AI output that is plausible and quietly wrong: a reconciliation that balances but misclassifies, a summary that drops the exception, a workflow with a step no human ever sees. Then ask what they would do about it.
Four things surface fast:
whether they spot it at all
whether they can say why it is wrong, rather than just that it looks off
whether their fix addresses the cause or the symptom
whether they say "I'd need to check" at the point where checking is the right answer
The last one carries the most signal. A candidate who never reaches for uncertainty is not confident — they are unpractised at being accountable for output.
Ask them to define good
Describe a workflow and ask how they would know it had worked. Not how they would build it. How they would tell, afterwards, whether the thing was any good.
Most people jump straight to architecture. The ones worth hiring ask what the output is for, who is accountable for it, and what a wrong answer costs. Those three questions are evaluation design in plain clothes, and someone who reaches for them unprompted will not need to be taught the discipline later.
If a candidate cannot define good before building, they cannot tell you afterwards whether the system works. They can only tell you it runs.
Ask what they refused to ship
A short question with a long tail: what have you built that you decided not to release, and why?
Weak answers describe things that failed technically. Strong answers describe things that worked fine and were held back anyway — because the data was thinner than it looked, or the failure mode was invisible to the user, or nobody could explain the output to someone entitled to ask.
That tells you whether the person has ever carried the consequence of a decision, which is not the same as having shipped a lot.
Why this is the bar
We hold the same standard internally: vet before adopting, disclose what the model did, verify before anything reaches a client. An interview should test whether someone can work that way under mild pressure — not whether they can recite it.
Hiring for AI work is still being run as hiring for software work with extra libraries attached. It is not the same job. The scarce competence is the willingness to be accountable for output a machine produced, and that is testable — as soon as you stop testing typing speed.
If you are building a bench for AI work and the vetting bar is the part you are unsure about, that is a conversation we have often — [email protected]