How this is scored
Everything below is the actual rubric, not a summary of one. If a report tells you something you disagree with, this page should let you find the exact assumption you disagree with and say so.
This product is the estimator. ARC Control is the measurement. They are the only two surfaces that talk. The map — Assess → Measure → Control, four grains on one trace, how a figure is promoted from heuristic to measured — is docs/08-how-ark-works.md in the ARK repository. Connect production at Control /start.
The provenance ladder
Four rungs. Every figure in every report carries one.
The central design commitment of this system is that a guess and a measurement must never look the same on screen. Confidence in a user interface is a claim, and an unlabelled number is a claim made without evidence.
A rule of thumb from the rubric. Defensible, arguable, and not observed anywhere. Most cost figures start here.
A published figure — a model's list price, a vendor's stated context window, a third-party benchmark. True of the world, not necessarily of you.
Derived from measurements, but not itself a measurement: either observed on systems like yours, or your own observed call shape priced at list rates. Useful, and still a prior.
Computed from telemetry emitted by your own running workload, above the sample floor. This is the only rung that can contradict the rubric, and it is allowed to.
A prior is promoted only above a minimum sample of 30 traces for that pattern. Below the floor the number stays heuristic and says so, because a mean drawn from four traces is noise wearing a decimal point.
The seven suitability dimensions
Weighted. The weakest dimension usually decides the verdict, which is deliberate.
| Dimension | Weight | What moves it |
|---|---|---|
| Task fit | 0.22 | The shapes you selected. Calculate and Look up alone score 5 and raise a hard blocker: the verdict is not-ai whatever else is true. A mix of deterministic and linguistic shapes scores 55; purely linguistic or judgement work scores 85. |
| Tolerance for variation | 0.16 | Whether one answer is correct. "Exact" scores 15, the strongest negative signal after task fit, because a probabilistic system is a poor fit for a verifiable question. "Tolerant" scores 80, "subjective" 90. |
| Cost of being wrong | 0.16 | Your stated error tolerance: none 20, low 45, medium 78, high 92. Zero tolerance combined with bounded or full autonomy is a hard blocker, and zero tolerance caps the verdict at assisted. |
| Data readiness | 0.14 | Whether the job needs knowledge outside the request, and how many systems of record supply it. Self-contained input scores 85; one to three named sources 72; more than three 55; needed but unnamed 30. |
| Volume economics | 0.14 | Units per month against the cost of a build: 10,000 or more scores 95; 2,000 or more, 82; 400 or more, 62; 50 or more, 38; fewer, 15. Under 50 a month is a hard blocker for a team build: buy, or do it by hand. |
| Value per unit | 0.10 | Minutes a person spends per unit today: 20 or more scores 95; 5 or more, 80; 2 or more, 60; under 2, 35. Not supplied scores a neutral 50 and is listed as the most useful number to go and collect. |
| Risk load | 0.08 | Starts at 90 and loses 9 per sensitive data class, 7 per write action, 20 per irreversible action, 20 for full autonomy and 8 for bounded. Irreversible actions under autonomy, or credentials in the data flow, are hard blockers. |
This table is the engine's own rubric, rendered from the same constant the scoring code reads, so it cannot describe a different rubric from the one that scored your report. A weighted average alone would let a high volume score paper over a workload that must be exactly right. It does not: a small number of conditions are hard blockers that cap the verdict regardless of the total, and the report lists them by name. Totals below 40 are not-ai, below 58 not yet, below 72 assisted.
Assumed call shapes
Per architecture pattern. These are the figures ARC Control replaces with measurement.
| Pattern | Turns / outcome | Context growth / turn | Failure rate | Retries | Cache hit |
|---|---|---|---|---|---|
| Deterministic automation (no model)deterministic-automation | 0 | 0 | 0% | 0 | 0% |
| Small-model classifierclassifier | 1 | 0 | 3% | 1 | 30% |
| Single-shot LLM callllm-single-shot | 1 | 0 | 5% | 1 | 40% |
| Retrieval-augmented generationrag | 1 | 0 | 8% | 1 | 20% |
| Deterministic workflow with LLM nodesllm-workflow | 3 | 300 | 10% | 1 | 35% |
| Bounded agent with tool allow-listbounded-agent | 7 | 900 | 15% | 1.5 | 45% |
| Autonomous agentautonomous-agent | 14 | 1200 | 22% | 2 | 50% |
| Human-in-the-loop hybridhybrid-human-loop | 1 | 0 | 4% | 1 | 35% |
Turns per outcome and failure rate are where cost estimates go wrong by an order of magnitude. An agent that averages seven turns in a benchmark and nineteen in production is not a rounding error — it is a bill that arrives at nearly three times the forecast, and no amount of care with the price-per-token column will catch it.
How cost is computed
Cost per successful outcome, not cost per call.
One attempt costs the input tokens plus the accumulated context of every prior turn, priced at the model's input rate, plus the output tokens at the output rate, less the share of input served from cache at the cached-read rate. Context is accumulated per turn rather than assumed constant — the thing that makes agents expensive is not the number of calls, it is that call seven carries the transcript of calls one through six.
An outcome costs one attempt, plus the failure rate multiplied by the retries per failure, multiplied by the cost of an attempt. Divide by the success rate and you get the cost of a result you can actually use. That is the number this report quotes.
Every tool that quotes a price per thousand tokens and stops there is quoting the cost of the attempt, not the cost of the outcome. For a single-shot classifier the gap is a few per cent. For a bounded agent it is routinely forty.
Where the report refuses to answer
The parts that are deliberately blank.
- ·ROI without a baseline. If you do not tell it what the work costs today, it reports payback as unknowable rather than inventing a denominator. A fabricated ROI is worse than no ROI, because it survives being repeated.
- ·Accuracy. This tool cannot tell you how accurate a model will be on your data. Nobody can, before the golden set exists. What it gives you instead is the threshold your error tolerance implies and the size of the set you need to measure it honestly.
- ·The human baseline. If you did not supply a human error rate, the report says so rather than treating your current process as perfect. “93% accurate” is a result or a regression depending entirely on a number most teams have never measured.
- ·Displacement. AI removing 60% of the human minutes on a task is an assumption, stated as one and adjustable. Teams routinely model it as 100% and are routinely wrong.
- ·A model for a not-ai job. When the verdict is not-ai the report names no model, prices no tokens, sizes no golden set and offers no sample. It sizes the deterministic build and its tests instead. Recommending a model in the same breath as refusing one would make the refusal decorative.
Use it from CI
The same engine, without the form. Score a backlog on every change to it.
POST /api/assess takes one Workload, or an array of up to 20, as JSON. The body may be at most 256 KB. A larger batch is refused with 413 rather than truncated; split it. GET /api/assess returns a complete example Workload.
curl -sS https://<this-host>/api/assess \
-H 'content-type: application/json' \
-d '{"id":"wl_triage","name":"Triage support tickets",
"task":["classify","summarize"],
"volume":{"unitsPerMonth":4000,"unitLabel":"ticket"},
"input":{"avgTokens":1200},"output":{"avgTokens":250},
"determinism":"tolerant","errorTolerance":"medium",
"dataClasses":["internal","pii"],"autonomy":"suggest",
"current":{"minutesPerUnit":6,"fullyLoadedHourlyUsd":62}}' \
| jq '{verdict, modelId, costPerUnit: .costPerUnit.value, basis: .costPerUnit.basis}'Each result carries verdict, blockers, pattern, the cost and payback estimates with their basis, the risk level, the trust block and the full assessment. When the verdict is not-ai, modelId and assessment.model are null, cost is zero with a deterministic source, and assessment.evaluation.kind is deterministic. A useful gate: fail the build when a workload someone wants to fund comes back not-ai or not-yet.
A body that is not a Workload is a 422 with the schema issues. Nothing is stored: the response is the only copy.
Known limitations
Written down so they can be argued with rather than discovered.
- Token estimates come from your description of the workload. If your real prompts are twice the size you think they are — and they usually are, once system prompts, few-shot examples and retrieved context are counted — every cost figure scales with them.
- Model prices are captured from published rates on a known date and carry an expiry. A promotional rate that lapses is the most common way a correct forecast becomes wrong without anyone changing anything.
- Build cost is a rubric estimate against a loaded engineering rate. It has no knowledge of your integration surface, your procurement process, or your security review queue, and those are frequently the long pole.
- The evaluation plan tells you what to measure and what threshold to hold. It cannot tell you whether you will hit it.
- Calibration draws on your own telemetry once Control is running, which means it is accurate about the workloads you already operate and silent about the ones you do not.
- This tool assesses one workload at a time. Portfolio effects — shared retrieval infrastructure, a platform team amortised across six projects, a volume commitment — are real and are not modelled here.