AI engineering interview questions: retrieval, evaluation and everything that breaks

The field is young enough that nobody has ten years of experience, so interviewers weight engineering judgement heavily over tool familiarity. The questions are less about what you have used and more about how you would know it was working.

Almost every candidate has now built something with a language model. That means the demo no longer differentiates anyone, and interviews have moved to the parts that are hard: making retrieval actually relevant, knowing whether a change made things worse, and handling text you did not write.

Retrieval, past the diagram

Everyone can draw the pipeline: embed the documents, embed the question, find the nearest, put them in the context. Interviewers have seen it. The questions are all about what makes it work badly.

• Chunking. Too small and a chunk loses the context that made it meaningful. Too large and the relevant sentence is diluted by everything around it. There is no universal size, and saying so with a reason is better than naming one.
• Semantic search missing exact matches. Nearest-neighbour search is poor at identifiers, error codes and product names, which is why hybrid retrieval with a keyword component so often beats pure embeddings in practice.
• Retrieving the wrong thing confidently. Similarity is not relevance. Something always comes back, whether or not anything useful exists.
• Stale content. The index is a copy. What happens when the source changes is an operational question people forget to answer.

Users say the answers are often wrong. How would you debug that?

Junior answer: I would improve the instructions to the model, try a larger model, and adjust the temperature to make it more accurate and less prone to making things up.

Senior answer: First I would find out whether retrieval or generation is failing, because they need opposite fixes and guessing wastes a week. I would take a sample of the bad answers and look at what was actually retrieved. If the right passage was never retrieved, no amount of instruction tuning will help and the work is in chunking or hybrid search. If the right passage was retrieved and the answer still contradicted it, that is a generation and grounding problem. Before changing anything I would build a small set of question and expected-answer pairs, so I can tell whether the next change helped or just moved the failures somewhere I was not looking.

Splitting retrieval failure from generation failure is the single most useful diagnostic in this area, and most candidates skip straight to the model.

Evaluation, which is the real interview

If there is one topic that separates people who have shipped this from people who have prototyped it, this is it. Non-deterministic output means you cannot rely on the usual assertion-based safety net, and the temptation is to test by trying a few things and feeling good about them.

What a strong answer describes is a fixed evaluation set built from real failures, run on every change, with results compared rather