Can Agents Build Real Commission Plans?

We gave an AI agent a business requirements document, 43 MCP tools, and zero domain knowledge. It built a complete, validated commission plan in 92 seconds. Here's the evaluation framework that tells us whether we can trust it.

Commission plans are hard. Not because the math is complicated — it usually isn't — but because the modeling is. You need to understand business rules, map them to platform primitives, configure data sources, define conditions and tiers, and then validate the output against expected payouts. A new implementation engineer spends days on their first plan. An experienced one still needs hours.

We wanted to know: can an AI agent do this? Not "can it help" — can it actually build a working plan end-to-end, with no human in the loop?

The answer is yes. But the interesting part isn't that it works. It's how we know it works, and how we got there.

Key results:

The benchmark: Build Your First Plan

Every new implementation engineer at CaptivateIQ goes through the same exercise: Build Your First Plan (BYFP). You get a business requirements document describing a sales compensation structure. You build it in the platform. You validate the payouts match.

BYFP is a near-perfect AI benchmark for three reasons:

If an agent can pass BYFP, it can navigate the platform well enough to handle real customer work. If it can't, we know exactly where it fails.

What the agent does

The agent starts with nothing — no pre-loaded knowledge of the platform, no cached examples, no fine-tuning. It has:

The agent's workflow follows this sequence: Read BRD → Discover Platform → Create Org Structure → Configure Data Sources → Build Plan Components → Define Formulas & Conditions → Run Calculation → Check if Payouts Match. If they don't match, it loops back to debug and retry.

The agent reads the BRD, discovers what tools are available, creates the organizational structure, configures data ingestion, builds the plan, runs the calculation, and checks the results against the expected payouts. If the numbers don't match, it debugs and retries.

The efficiency curve

We didn't get here in one shot. Over 22 iterations, we improved both the tools and the agent's environment until the numbers converged:

AttemptTool CallsDurationErrorsWhat Changed00455~8 min1First true zero-knowledge pass007323m 45s0First zero-error run (Opus)01123~170s0Bulk APIs + inline column creation015242m 15s0Fastest cold-start0191892s0Record best (Opus, reasoning effort 50)02225~3 min1Sonnet self-corrects formula bug mid-build

Every failed run generates an infrastructure audit. The fixes compound. This is the flywheel.

Each iteration didn't just tweak the prompt. It improved the tools. When the agent struggled to create formula components one at a time, we built a bulk creation API — tool calls dropped from 55 to 23. When it couldn't discover formula syntax from the UI, we added runtime reference docs that the agent reads on demand. When it wasted tool calls on unnecessary GET requests, we enriched response payloads to include the data it would ask for next.

The improvement loop is: agent fails, we study the failure, we fix the infrastructure, agent gets faster. The agent is a stress test for the platform's API surface. Every run makes the APIs better for humans too.

Cross-model results

We ran the eval across Claude model sizes to understand the capability threshold:

ModelPass RateAvg Tool CallsCost/RunNotesOpus 4.68/10 (80%)24$2.02Most reliable. Handles reduced reasoning effort.Sonnet 4.64/5 (80%)23-25$1.02Best cost/performance. Self-corrects errors mid-build.Haiku 4.50/3 (0%)94-117$0.82-1.27Builds the right structure, fails on complex override logic.

The gap between models isn't just accuracy — it's recovery. Sonnet catches its own mistakes and fixes them (attempt 022: spotted a formula bug, corrected it, passed). Haiku builds correct org structures and data sources but can't debug a failing formula when the override logic gets complex. Opus is the most reliable but at twice the cost — on the BYFP benchmark, Sonnet matches Opus's pass rate at half the price.

The 3 failed Opus runs traced back to infrastructure bugs (MCP server config, timezone handling, feature flag state) — not model capability. All three were subsequently fixed.

What makes this possible

Three things had to exist before the agent could work:

Full CRUD tool coverage

43 MCP tools covering every platform operation. Not "read-only access to the API" — full create, read, update, delete across organizations, plans, components, formulas, conditions, data sources, and payouts. Partial coverage produces partial plans. The agent needs to go end-to-end.

Runtime reference documentation

The agent doesn't know CaptivateIQ's formula syntax from training data. It discovers it at runtime by calling get_formula_reference and get_system_reference — tools that return the available functions, operators, and syntax rules. This is critical: it means the agent works with the current platform, not a snapshot frozen in training data. When we ship new formula functions, the agent learns them immediately.

Bulk APIs

Early iterations burned 50+ tool calls creating components one by one. Bulk endpoints cut this to 23. One call creates an entire formula with all its components. The efficiency curve in the table above is mostly a story about bulk API adoption — and about CSV upload replacing row-by-row data entry.

We spent more time improving the tools than improving the prompt. The agent is a generic reasoner — Claude with MCP tools. The intelligence is in the tool design, not the agent configuration. Good tools make dumb prompts work. Bad tools make smart prompts fail.

What's next

BYFP is a controlled benchmark. Real customer plans are messier: ambiguous requirements, legacy data formats, edge cases that aren't in any BRD. We're working toward multi-plan complexity (manager overrides, team rollups), real customer BRDs with unstructured requirements, and production readiness with human-in-the-loop approval before changes go live.

The point isn't to replace implementation engineers. It's to give them a first draft that's already 80% right, so they can focus on the 20% that requires judgment. The BYFP eval tells us how close we are to that bar — and right now, on the standardized benchmark, the agent meets it.

Drafted with Claude, edited by Ray Zeller. All eval data and timings are from production runs.