Your agent test passes on your laptop and fails in CI. The laptop has a token for the issue tracker’s MCP server; CI doesn’t, and shouldn’t. Even with a token, the tracker listed three open bugs yesterday and lists four today, so the agent’s summary changes and the assertion on it breaks.

Web developers dealt with this years ago. VCR (Ruby), Polly.js, Betamax and go-vcr record real HTTP responses into a file the first time a test runs, then serve them from that file on every run after. The file is called a cassette. MCP needs the same thing: a proxy that sits between an agent and its MCP servers, writes every exchange into one cassette, and plays it back later with no server, no credentials and no network. mcprec below is an illustrative name for a tool you’d have to build.

An API can pass every unit test in its suite and still break every client that calls it, because unit tests check that functions do what the code says they do, not that the API does what the contract says it does. That gap is where most production API incidents actually come from: a field renamed, a status code changed from 200 to 204, a required parameter quietly made optional. Testing an API well means testing at several different levels, because each one catches a different class of failure.

Unit tests verify that individual functions behave correctly in isolation. They run fast, catch regressions close to where they are introduced, and document expected behavior at the code level. They do not verify that the API endpoints exposed to consumers behave according to the documented contract, that changes in one service do not break the consumers that depend on it, or that the system behaves correctly under the conditions that production traffic creates.