Skip to content
Sayam Jain
CV
All posts

Evals before vibes

Published · 9 min read

Short answer

An eval is a fixed set of inputs and expected outcomes that you can rerun after every prompt or model change. I compare two prompts on 24 synthetic WhatsApp cases, grade them in code, and check an AI judge against labels I wrote by hand.

Here's how a lot of prompt changes ship. You tweak the prompt, try it on three examples you remember, the answers look better, and you ship it. That's vibes.

I've done it too. Then I spent a few months building checks for my WhatsApp assistant, and the most useful thing they taught me was where my own checks were weaker than I thought. This post is what I built, what it caught, and a small demo that shows the difference between trying a prompt on a few examples and running it against a fixed set.

There's also a practical reason this is timely. OpenAI's hosted Evals platform became read only for existing users on 31 October 2026, and its dashboard and API are scheduled to shut down on 30 November. If your test cases and graders live there, this is a good time to move them somewhere you can version and run yourself. OpenAI's deprecation notes have the current dates.

What I built for my WhatsApp assistant

Every finding

  1. Model replyquote, chat, message
  2. Quote checkplain code
  3. Focusverified only

Around it

  1. 58 test filesparsers, evidence, Focus
  2. Eval runnerfast checks could only pass
  3. AI judgesaw only the reply
  4. Six labelled scenariosnever wired in
  5. Seven day gatekept the new brief off
  6. !correctreasons nothing reads yet
Fig. 1 The checks around my WhatsApp bot

Every finding goes from the model's reply through a quote check in plain code, and only verified findings reach Focus. Around that sit 58 test files, an evaluation runner, an AI judge, six labelled scenarios, a gate of seven reviewed days and the !correct command.

Message Yourself OS reads my WhatsApp and tells me who's waiting on me and what I promised. Being wrong is expensive there: a bot that says "you promised this" had better be right. So most of the work went into checks.

Every finding carries its evidence. The model has to return the exact quote, the chat and the message it came from. Plain code then fetches that message and checks that the quote is really in it and the time matches. A finding ends up in one of a few states: verified, missing, not found, quote mismatch or time mismatch. Only verified findings reach the Focus list at the top of !today. Anything else appears under "Needs verification", never as fact.

58 test files pin the parsers, the evidence check and the rule that unverified items stay out of Focus.

A local evaluation runner has a fast tier that needs no WhatsApp connection or network, and a full tier that runs commands through a local server and tracks latency and regressions.

An AI judge scores replies. If the judgement itself fails, the result is reported as "not scored" and is never shown as a number.

A gate on real use. The newer, ranked !today stays behind a switch until it passes seven reviewed days of real use, with no errors and nothing missed.

!correct records why an extraction was wrong, in my own words.

What those checks caught, and what they couldn't

The evidence check earned its keep straight away. It's the reason a made up quote can't turn into a reminder. It's also the part I trust most, because it's plain code checking plain facts.

The rest taught me more by showing me their limits:

  • The gate did its job by saying no. When I last checked in August 2026, the ranked brief had nine runs over three days and almost none had been reviewed. So it stayed off. Without that gate, I might have switched it on because the replies looked good.
  • My fast checks had a blind spot. They confirmed that all 44 commands were wired up, but the heading checks compared against text I had typed into the test, not the actual output. Useful as a wiring check, not an evaluation of what the command returned.
  • My first AI judge only saw the reply. It never saw the messages the reply was about, so it could judge tone and format, not whether the answer was true. I wrote in my own docs that its scores were advisory for exactly that reason.
  • I wrote six labelled scenarios and never wired them in. An explicit promise, an implied one, a resolved one, group noise, several people and a dead conversation. They sat in a file that no code read.
  • !correct records reasons, but nothing reads them back. Every correction should become a test case. Mine became a line in a file, but the runner did not use those lines.

Pattern dikh raha hai? Everything that worked compared the output against something fixed: a real message, a test or a calendar of reviewed days. Everything that didn't was either checking itself or never run.

The same lesson, in one small demo

Vibes

  1. 3 easy casespicked by hand
  2. v1 3/3, v2 3/3every run
  3. Looks fineso ship v1?

Eval

  1. 24 labelled casesanswers written down
  2. v1: 16 to 18 of 246 to 8 false positives
  3. v2: 22 to 23 of 241 false positive
  4. Cases that movednamed, not felt
Fig. 2 Three examples against twenty four

On the three hand picked cases both prompts score 3 of 3 in every run, so v1 looks fine. On all 24 labelled cases v1 passes 16 to 18 with 6 to 8 false positives, and v2 passes 22 to 23 with one false positive, the rhetorical question.

To show the idea without using my private chats, I built a small eval with 24 synthetic cases. Each case contains one to three chats and the expected findings written down by hand. The set has 17 expected findings in total, and nine cases should return nothing. It covers plain questions, requests without question marks, Hinglish questions and promises, group messages addressed to someone else, requests already answered, promises already kept, rhetorical questions and messages that should not count.

The task is the same as in my last post: find what's waiting on me and what I promised, with exact quotes. The code, cases, prompts and results are in the companion demo on GitHub.

I compared two prompts:

  • v1 is the original prompt from the last post. It asks for what is waiting on me and what I promised, but leaves much of the meaning implicit.
  • v2 spells out the rules: a question or request to me that I haven't answered, a promise in any language including Hinglish, and no statements, rhetorical questions, questions addressed to other people or things already done.

First, the vibes check. I ran both prompts on three easy cases, the kind you'd try by hand. Both got all three right in every run. If that's all you look at, v1 is fine.

Then I ran both prompts on all 24 cases using openai/gpt-oss-120b through Groq. I repeated the run three times. Each sequence in the table is in run order: one, two, then three.

Measure v1, runs 1 / 2 / 3 v2, runs 1 / 2 / 3
Three easy cases passed 3/3, 3/3, 3/3 3/3, 3/3, 3/3
Cases passed, out of 24 18, 17, 16 22, 23, 23
False positives 7, 6, 8 1, 1, 1
Missed findings, out of 17 0, 1, 1 1, 0, 0

A case only passes when the expected findings are present, nothing extra is returned, the finding has the right kind and its quote passes the evidence check. One bad finding can make the whole case fail.

The difference is in the cases you wouldn't think to try. v1 said a client's "Can you send the invoice?" was still waiting on me, two messages after I'd sent it. It flagged "Why is Bengaluru traffic like this every single day?" as a question for me. And it marked Arjun's "Bro the API docs link is broken" as waiting on me, the same mistake I found by eye in my last post. This time a test case catches it every run.

v2 isn't perfect either. The rhetorical question fooled it in every run, and in one run it missed a Hinglish promise that v1 got right. That's the point of a fixed set: you see exactly which cases moved, in which direction, instead of relying on a feeling that it's better.

The judge agreed with the mistakes

AI judge

  1. Every findingfrom v1 and v2
  2. Pass or failwith a reason
  3. Passed 6 wrong findingsv1, every run

My labels

  1. Right answerswritten by hand
  2. Judge against labelscase by case
  3. Wrong one way onlyit approved mistakes
Fig. 3 The judge against my labels

An AI judge gave every finding a pass or fail with a reason. Compared with labels written by hand, it never failed a correct finding, but it passed six of v1's wrong findings in every run, so its mistakes all pointed the same way.

I also ran an AI judge over the findings the prompts returned. It saw each finding together with the chat it came from, and had to give a pass or fail: was this really something waiting on Sayam, or really a promise he made? I then compared those judgements with the labels I had written by hand.

In every v1 run, the judge passed six findings that my labels said were wrong. It never failed one of the correct findings it judged. The examples included the broken link, the rhetorical question and the invoice I had already sent. Its explanations sounded reasonable: a broken link "requires Sayam to respond or fix"; a rhetorical question "legitimately expects a reply".

There is an important limit here: the judge only sees findings that the prompt returned. It cannot catch a missed promise, because there is no finding for it to judge. Its agreement is not a second score for the prompt and it is not a measure of recall.

This also reminded me of the limitation in my first production judge, which only saw the generated reply and not the messages behind it. The setups are different, but the lesson is similar: a plausible explanation is not evidence.

This is why I wouldn't trust an LLM judge just because it returns a confident explanation. Anthropic recommends calibrating judges against human experts, and the 2023 LLM as a judge study documented position, verbosity and self enhancement biases. A judge can be useful at scale, but its disagreements need to be checked against labels for your own task.

How I'd start evals for an AI feature tomorrow

  1. Start with 20 to 50 cases from real failures. That's Anthropic's recommendation for a useful first set. My public demo uses synthetic messages for privacy, so it demonstrates the method rather than measuring the assistant on production conversations. When building your own set, turn real failures into cases after removing or replacing private details.
  2. Write the expected answer down by hand. Your labels define what success means. If the right answer is unclear to you, the model cannot reliably infer it.
  3. Prefer pass or fail for concrete checks. Hamel Husain and Shreya Shankar argue that binary judgements make the criteria clearer and more consistent than a score out of ten. A seven out of ten doesn't tell you what to fix.
  4. Grade with code first. Is the quote in the message? Is the sender right? Is the JSON valid? Code is cheap, fast and consistent for facts that can be checked deterministically.
  5. Add a judge only when code cannot judge the quality. Give it a clear rubric and compare its decisions with your labels. Check the disagreements instead of trusting its score.
  6. Keep a holdout set. Use some cases while changing the prompt, but keep others out of that loop so you can see whether the improvement generalises.
  7. Run the set after every prompt or model change. Keep old failures in it, so a fix never quietly brings an old bug back.

You don't need an evaluation platform to start. My demo is a JSONL file, two graders and a loop. The TypeScript runner has no package dependencies; a real model run still needs a provider API key and has to respect that provider's limits.

If you do want a tool, OpenAI's migration guide points to Promptfoo. In March 2026 Promptfoo announced it had agreed to be acquired by OpenAI and said it would remain open source. Inspect AI is another open source option, developed by the UK AI Security Institute and Meridian Labs.

And this isn't just my concern. In LangChain's State of Agent Engineering survey, 52.4% of organisations surveyed reported running offline evaluations on test sets. It's a useful signal about the teams in that survey, not a measurement of every company building AI.

What this demo doesn't prove

These are 24 synthetic cases, labelled by me, run three times on one model. They show how two prompts behaved on this particular set. They don't prove that v2 is better on real conversations, on another model or on a different task. My labels are judgement calls too: you could argue that a broken link really is waiting on me, and the judge did.

The judge results have a narrower meaning still. It only judges returned findings, so it cannot say anything about missed items. This is a small check of where the judge agrees with my labels, not a measure of recall or a production quality score.

The point isn't that v2 is the best prompt. It's that with a fixed set I can see which cases it fixed, which one it still gets wrong and which one it broke once. Then I can argue about specific cases instead of a feeling that a prompt is better.

Vibes are how you find what to test. Evals are how you know what changed.

Sources

ai · evals · llm · testing

Written by Sayam Jain, AI Product Engineer in Bengaluru. More about me

Need something like this built?Work with me

Related posts

  1. Not every AI feature needs an agent

    Use a workflow when you already know the steps. Let code handle selection, validation and execution, and let the model handle the parts that need language understanding. Use an agent when the next step genuinely depends on what it discovers.

    ai · agents · llm · architecture

Have an interesting product, engineering challenge, or collaboration in mind?

I'd love to hear about it.

contact@sayamjain.com
Work with me: packages and prices