A pull request the gate stopped
Could not load data/runs.json.
What it can't catch
- Small drops. With about 50 cases only big drops are detectable. Even if nothing improved, at least 5 cases have to flip from right to wrong before the gate can fail.
- A different model. It gates with a small local Ollama model, not the model the app runs in production. It tests the prompt, the schema notes, retrieval and the pipeline, not the production model's quality.
- Noisy runners. Runner CPUs differ and can flip a couple of cases even at temperature 0. Runs of the same code against itself have shown b = c = 2.
- A guess after an abstention. In docs-rag, an answer that says the documents don't cover it and then guesses in the next sentence still passes the rule-based check. Catching that needs an LLM judge, and there is no judge yet because there are no human labels to calibrate it against.
Production traffic
About 25% of real questions to the live apps are sampled, redacted and sent to Langfuse. A weekly job queues the flagged ones in a private repo, where I label them before any becomes a new eval case. Nothing a visitor typed is published here.