Some of us at dontapplyai developed CAPTCHAs in our prior lives, useful if you're creating a product like this right?
You have almost certainly used them (some of our engineers worked at FANG on anti-bot designs that run the Internet and some others were ML engineers). When we tried to vibe code our interview CAPTCHAs here's what we learnt.
They all sucked (all 10,000 of them).
It didn't matter which model we chose, Astra, Fable 5.
The ones that passed our test bench (vision models and advanced OCR system) were well.. unsurprisingly unreadable to Humans. See the problem here? It's kinda obvious now.
LLM don't "know" what good CAPTCHA is because it just can't perceive things the way a Human does and we're using the very systems to "see" that we were trying to break. We were caught in a chicken and egg situation.
It also taught us Vision models are good and there's no point trying to beat them in the conventional sense. Instead we have developed and grow new techniques that fool Vision models in novel ways which we benchmark daily and remove poor designs from our pool as new models are released, which are then served to our users.
The rest of this post explains what an interview CAPTCHA has to do, why an AI model can't judge one, and how our team makes them instead.
What an interview CAPTCHA is for
AI interview assistants read the interview question from the candidate's screen. They take a screenshot, turn the image into text and write an answer for the candidate to read out. An interview CAPTCHA shows the question in a way that a person can read at a glance and that these tools have much more trouble reading.
Interview CAPTCHAs are similar to the CAPTCHAs on sign-up forms, with two differences. They have to work on a screen that is shared during a live conversation, and they can't slow down an honest candidate who is reading the question for the first time.
What a design has to do
Each design has to work for two readers. The candidate has to be able to read it quickly and easily, and an AI tool should have trouble reading it or should read it wrong.
Most new ideas fail one of these tests. Plain text is easy for both people and machines to read. Heavy distortion stops machines, but it also stops candidates. Only a small number of designs are easy for people and hard for models, and most of the design work is spent finding new ones.
Why AI models can't judge designs
An AI model can produce many design ideas quickly, but it can't tell you which ones work. To judge a design, someone has to look at it and decide whether a person can read it easily and whether a machine has trouble with it. A model can only judge a design by reading the image itself, and it doesn't see images the way people do.
A vision model splits an image into small squares and turns each square into a list of numbers. It looks at one still frame at a time. People recognize words by their shape, fill in parts of letters that are missing and use the rest of a sentence to work out a smudged word. When something on the screen moves, people combine what they saw over a fraction of a second.
If a model can read a design, the design has failed. If a model can't read a design, the model has no way of knowing whether a person could. In both cases, the answer has to come from a person looking at the screen.
Designs made from a prompt
Vibe coding means describing what you want to an AI model and using what it produces. When we asked models for text that AI can't read, we mostly got the ideas that appear most often in their training data, such as wavy letters, background noise, rotated characters and low contrast. Other models have seen the same ideas many times, and current readers handle them well. Machine learning models learned to read the distorted text on sign-up forms years ago, and most sites have stopped relying on it.
In our tests, current vision models read through most of these techniques, including heavy noise drawn over the text.
Where people do better
People can find gaps that a model can't see. A useful place to look is a misread, where a model reads a word wrong, skips a line or answers a question that isn't on the screen. Each misread shows something about how that model sees, and a person can use it to build the next design. A model can't do this for itself, because it doesn't know what it failed to see.
People also notice problems that a model doesn't report. A design can be readable but tiring by the end of a long interview, or hard to read for someone who is color blind. These problems matter in a real interview as much as whether an AI tool can read the question.
How our designs are made
Our CAPTCHA designs are built by a team of five machine learning and AI engineers. The team regularly revises the designs in use and releases new ones. Every day the designs in our pool are tested against vision models and OCR systems, and the ones that do poorly are removed.
Each design is assigned to one group of customers, and other groups use different designs. If someone studies a design used at one company, or builds a tool to read it, that work doesn't help at companies in other groups.
What this means for your interviews
Your interviews use the designs assigned to your account, and the designs change as the team releases new ones. A candidate who has practiced against the designs used at another company is unlikely to have seen yours.
An interview CAPTCHA doesn't prove anything about a candidate. It makes AI help slower and harder to use during the interview, which makes it easier for you to notice.
