Reliability engineering
PlannedVoice Agent Regression Harness
A proposed replay harness that runs recorded calls against a new agent version and blocks the release when behaviour regresses.
01 / THE PROBLEM
Why this matters.
A prompt or model change that improves one call can quietly break another. Without a way to replay past conversations against a candidate version, regressions are found by callers rather than before release.
02 / THE DIRECTION
The proposed solution.
A library of recorded calls with the expected behaviour attached to each. Every candidate version is replayed against the library, differences are compared per check rather than word for word, and the release fails when a regression appears in a protected check.
03 / ARCHITECTURE
How the pieces connect.
Conceptual flow — proposed architecture
- 01Recorded call library
- 02Expected behaviour per call
- 03Replay against candidate version
- 04Behaviour diff against baseline
- 05Regression classification
- 06Release gate
04 / ENGINEERING DECISIONS
Designed with intent.
- Compare against expected behaviour, not transcripts word for word.
- Mark a subset of checks as protected, so a regression there fails the release outright.
- Keep the gate deterministic — a model may explain a difference, but never decides pass or fail.
- Turn every confirmed production failure into a new case in the library.
05 / EVALUATION APPROACH
What needs to hold true.
- Verify a known-bad version is actually blocked.
- Test flaky checks, so noise cannot fail a release on its own.
- Test versions that are genuinely better and confirm they are not flagged as regressions.
- Confirm the library grows from reviewed real failures rather than synthetic cases.