Uploaded April 2026 | Updated November 2026, 3 days ago
MCP vs. CLI: which is actually better for agent tool use? Laurie Voss, Head of Developer Relations at Arize AI runs 500 evals to find out.
The setup: one model (Claude Opus 4.6), one GitHub repo, 25 tasks across 4 difficulty tiers, and three tooling strategies — the official GitHub MCP server, a long encyclopedic gh skill, and a short opinionated gh skill. Five passes each, all traced and scored in Arize with LLM-as-judge.
What the data showed:
➡️ Correctness was a tie at ~82% across all arms
➡️ On the hardest analytical tasks, MCP cost 6× more and ran 5× longer than the CLI arms
➡️ One MCP run hit 71 tool calls, 8 minutes, and $2 — and 68 of those calls were bash, because the agent gave up on the API
➡️ On tasks that mapped cleanly to a single MCP tool (create branch + PR), MCP won
➡️ A baseline arm with no skill and no MCP nearly matched the skills, because gh is so well-represented in training data
➡️ Short opinionated skills beat long encyclopedic ones
The real conclusion isn't MCP vs. CLI — it's MCP + CLI. Use the shell when the model knows the tool, auth is local, and you need to compose. Use MCP when the tool is remote, needs OAuth, hides an agent, or gets shipped to non-developer users who won't paste an API key into a config file.
MCP vs. CLI: which is actually better for agent tool use? Laurie Voss, Head of Developer Relations at Arize AI runs 500 evals to find out.
The setup: one model (Claude Opus 4.6), one GitHub repo, 25 tasks across 4 difficulty tiers, and three tooling strategies — the official GitHub MCP server, a long encyclopedic gh skill, and a short opinionated gh skill. Five passes each, all traced and scored in Arize with LLM-as-judge.
What the data showed:
➡️ Correctness was a tie at ~82% across all arms
➡️ On the hardest analytical tasks, MCP cost 6× more and ran 5× longer than the CLI arms
➡️ One MCP run hit 71 tool calls, 8 minutes, and $2 — and 68 of those calls were bash, because the agent gave up on the API
➡️ On tasks that mapped cleanly to a single MCP tool (create branch + PR), MCP won
➡️ A baseline arm with no skill and no MCP nearly matched the skills, because gh is so well-represented in training data
➡️ Short opinionated skills beat long encyclopedic ones
The real conclusion isn't MCP vs. CLI — it's MCP + CLI. Use the shell when the model knows the tool, auth is local, and you need to compose. Use MCP when the tool is remote, needs OAuth, hides an agent, or gets shipped to non-developer users who won't paste an API key into a config file.










