Developers report malformed tool calls that end agent runs and poison conversation history. aimock now makes these bad outputs available on demand, so you can test your recovery code.
Quality is the biggest barrier to putting agents into production, according to LangChain's survey of 1,340 professionals. It was the top answer, cited by 32% of respondents. That finding covers agent quality broadly. Developer reports show one concrete part of the problem: a single unusable tool call can derail the rest of a run.
In a Kimi CLI report, truncated tool arguments failed JSON parsing but remained in the conversation history. Subsequent requests replayed the malformed string. The provider repeatedly rejected them with HTTP 400, and the reporter said recovery required clearing or restarting the session. Rejecting the original arguments was necessary. It did not, by itself, restore a usable conversation.
An OpenClaw report showed a different consequence: HTTP 200 responses followed by failed agent turns. A commenter reported that an unattended run terminated despite a configured fallback chain. The report did not establish whether upstream generation or local stream assembly corrupted the arguments. It did establish the testing question: what does your application do after a successful request delivers an unusable tool call?
Model misbehavior faults are now available in aimock, CopilotKit's open-source mock server for LLM APIs. They give your tests controlled bad model outputs through supported API formats, so you can exercise the parser, validation, and recovery code in your application.
A tool call crosses several boundaries. Your client reads the response. Your application parses the arguments, checks their schema, selects a tool, and decides whether to execute it. Recovery also has to manage retries, conversation history, and what the user sees.
Each boundary needs its own assertions. Invalid arguments must not execute a tool. A retry must have a limit. Conversation history must remain usable. An empty answer must not leave the interface waiting forever.
A timeout test cannot answer those questions. A successful fixture does not exercise those branches. Waiting for a live model to repeat the same failure makes a poor regression test.
With aimock, your test selects the adverse output. Your real client and application process it. The test then asserts the behavior you require. For the Kimi-style cascade, aimock can supply the initial malformed arguments; your test must check history handling. This feature does not automatically simulate a provider's later HTTP 400 rejection or repair the agent.
A single bad response followed by a normal one is a useful first recovery test.
This setup fragment configures a weather fixture and corrupts its tool arguments once per test ID:
import { LLMock } from "@copilotkit/aimock";
const mock = new LLMock({
port: 0,
enableMisbehavior: true,
misbehavior: {
seed: 42,
faults: [{
fault: "tool-args-invalid-json",
style: "truncated",
tool: "weather",
providers: ["openai-chat"],
rate: 1,
times: 1,
}],
},
});
mock.onMessage("weather", {
toolCalls: [{
name: "weather",
arguments: '{"city":"Paris"}',
}],
});The surrounding test starts the server and points its OpenAI Chat client at ${mock.url}/v1. Both attempts send the same X-Test-Id. The test stops the server during cleanup.
On the first matching request, the tool argument string is truncated. On the next matching request, the fixture returns its original arguments because the fault has spent its one firing.
For an application that retries invalid arguments once, assert three things: the first response does not execute the tool, the retry succeeds, and the tool runs exactly once with city: "Paris".
aimock supplies the responses. The application supplies the retry and the assertions. A transport retry alone does not implement this recovery.
The fault engine uses request context where the fault requires it. A schema-violation fault targets a provable constraint in the declared tool schema. An unknown-tool fault produces a name absent from the request's tool declarations.
The library also covers duplicate tool-call IDs, length stops during tool calls, empty responses, refusals, content filtering, and reasoning-only output. Together with invalid JSON, schema violations, and unknown tools, that gives you nine fault types for different application checks.
You can configure faults at the server, for a test ID, on a fixture, or through a request header. The highest-priority configuration wins as a whole. At most one fault applies to each response, so a test can isolate one failure at a time.
Rates control how often a fault fires. Limits control how many successful firings it gets. A numeric seed makes the rate decisions repeatable with the same test ID, configuration source, fault entry, and evaluation sequence. Concurrent requests can change that sequence, so a seed does not make concurrent scheduling deterministic.
The request journal records which fault aimock prepared, where its configuration came from, and why other candidates skipped. It also records prepared tool names, argument strings, and emitted IDs.
Your test can inspect that evidence alongside the response and the application's effects, so a fault that never fired cannot masquerade as successful recovery.
A journal entry proves that aimock prepared the fault. Client-side assertions remain necessary to prove what the application received and did.
The feature covers supported modes across OpenAI, Anthropic, Bedrock, Gemini, Cohere, and Ollama API formats. Azure OpenAI and OpenRouter use the OpenAI Chat format. Availability varies by fault, API, and streaming mode.
Unsupported fixture or request-header faults fail explicitly. Broad server and runtime configurations skip unsupported formats and record the reason. The support matrix gives the exact boundaries.
Some outputs are modeled contracts whose exact native occurrence remains unverified; the reference identifies them. A local failure test demonstrates application behavior under that response, not how often a provider produces it.
Start with a tool call that matters to your application. Choose one failure, define the expected recovery, and assert that the tool executes only with acceptable arguments.
The model misbehavior guide includes configuration, retry examples, SDK behavior, and journal inspection. The aimock repository contains the source and setup instructions.
The next malformed tool call does not have to be the first time your application exercises its recovery path.
Install the latest version of aimock to use model misbehavior faults:
npm install @copilotkit/aimockLearn more:
We’ll keep shipping updates to aimock. Have an idea or found a bug? Open an issue or a PR on GitHub, and follow CopilotKit on Twitter for what’s next.
If you get stuck, reach out in the CopilotKit or AG-UI communities.
Subscribe to our blog and get updates on CopilotKit in your inbox.
