Your goal: say how you would know an LLM pipeline works, read messy results without being fooled
by a headline, and make a ship call you can defend.
How this works (about 30 minutes)
- Felix asks which role you are interviewing for (product, engineering or other) and pitches his
follow-ups to it. The question stays the same.
- One question, taken deep. He holds a fact sheet and, later, a results sheet. Ask for what you
want to know; he hands it over a line at a time. The numbers are made up for the exercise.
- Many follow-ups, one curveball, then a trade-off turn: ship, hold or change?
- If you want to sketch, say "can I use the notepad?" and Felix puts it up.
- No hints and no scores during the call. Your report comes after.
What scores well
- A definition of good tied to the user and what a wrong answer costs, before any metric
- A test set from real traffic with hard and risky cases added, and a held-out slice
- Graders chosen on purpose, and how you would know the model judge can be trusted
- Reading results by slice with an interval, not only the average
- A gate with a bar and a line on the risky cases, a staged live test, and failures fed back
- Answers that say where they are going and land in a minute or two
Tips
- Take Gen AI Coach · Evaluation first.
- Say what decision the measure will drive before you say which measure.