Evaluating CodeCall
Covers FrontMCP 1.10.10 as of
Choose CodeCall after measuring the workload you need to support. This evaluation uses the published 47-tool CRM, the same synthetic data and the same deterministic task through direct calls and a server-side script. It checks that both paths return the same answer. It does not measure model tokens, hosted latency, cost or a model's ability to write the script.
Run the fixture
Download the evaluation project, extract it, and follow its README. Dependencies and the fixture are pinned. In the site checkout, run:
node scripts/evaluate-codecall.mjs --runs=5The harness removes only the plugin registration for the direct baseline. Both servers use the same tool implementations. A scripted client supplies the same selection logic a model would need to produce. The small-catalog control performs a single direct lookup; it avoids pretending every task benefits from script execution.
Read the output
| Field | What it measures | What it cannot establish |
|---|---|---|
| catalog bytes | UTF-8 bytes of JSON tool definitions | Model tokenizer counts or billing |
| direct response bytes | Sum of result envelopes from the underlying calls | Which content a particular client includes in its prompt |
| discovery response bytes | Search and schema response envelopes | Free or zero-overhead discovery |
| CodeCall response bytes | The execution result envelope | Total context use without discovery and request sizes |
| direct calls / execution calls | Requests made by the scripted fixture | Number of model reasoning steps |
| directMs / scriptMs | In-process time after server startup | Cold deployment or hosted-model latency |
Runs are sequential after startup; initial and later calls can differ because of cache state. The output records each sample instead of reducing variability to one flattering number. Compare results before aggregating them. A failed assertion exits unsuccessfully; do not exclude incorrect results from an evaluation.
Add a model evaluation deliberately
Record the exact prompt, model/version, settings, catalog, client behavior, execution limits, and a fixed task set. Include simple lookups, joins, ambiguous requests and refused operations. Measure actual input/output tokens, tool requests, answer correctness and wall time for repeated cold and warm runs. Keep API price arithmetic dated and separate from those observations.
Publish failures and limits alongside results. Do not extrapolate a fixture's byte reduction to a percentage token saving, security certification or universal speedup. Tool authorization still applies in both paths; read CodeCall's limits before enabling scripts.
Next step
Inspect the complete CRM, then replace its synthetic data with your own read-only workload. If direct calls already produce a small answer, keep that simpler path.