Voice Agent Regression Harness: the thinking.
A proposed replay harness that runs recorded calls against a new agent version and blocks the release when behaviour regresses.
The starting point.
What made this worth building.
A prompt or model change that improves one call can quietly break another. Without a way to replay past conversations against a candidate version, regressions are found by callers rather than before release.
The direction I chose.
The approach I am proposing.
A library of recorded calls with the expected behaviour attached to each. Every candidate version is replayed against the library, differences are compared per check rather than word for word, and the release fails when a regression appears in a protected check.
How the pieces connect.
Each stage exists because the one before it cannot be trusted on its own.
- 01Recorded call library
- 02Expected behaviour per call
- 03Replay against candidate version
- 04Behaviour diff against baseline
- 05Regression classification
- 06Release gate
The decisions behind it.
The constraints I set deliberately, and would defend.
- Compare against expected behaviour, not transcripts word for word.
- Mark a subset of checks as protected, so a regression there fails the release outright.
- Keep the gate deterministic — a model may explain a difference, but never decides pass or fail.
- Turn every confirmed production failure into a new case in the library.
What has to hold true.
How I would know it works — and how I would know it does not.
- Verify a known-bad version is actually blocked.
- Test flaky checks, so noise cannot fail a release on its own.
- Test versions that are genuinely better and confirm they are not flagged as regressions.
- Confirm the library grows from reviewed real failures rather than synthetic cases.
Where this stands today.
Stated plainly, with nothing implied.
This is a planned concept. Everything above is design reasoning — no part of it has been built or verified yet.
Keep going