The Cost of Useful Tokens

Date: 2026-07-05

Status: Working paper v0.2, full draft

Track: VKR AI Research

Abstract

AI inference is usually priced in tokens, but economic usefulness is not. A token is a billing unit, a sampling unit, and sometimes a throughput unit. It is not a unit of work completed.

This paper argues that AI systems should be compared at the level of accepted work rather than generated text. The economically relevant question is not "How cheap are tokens?" but "How much does an accepted unit of useful work cost, including latency, retries, routing, tool calls, verification, and rejected outputs?" As fixed-capability inference prices fall, reasoning models consume hidden or intermediate tokens, and agentic systems make multiple calls per task, raw price-per-token becomes a weaker proxy for deployed value.

We propose a measurement framework centered on Cost Per Accepted Task, Cost Per Useful Output Unit, Latency To Accepted Answer, Verification Cost Per Accepted Task, and Waste Ratio. The framework is designed for small, repeatable task suites rather than broad leaderboard claims. Its purpose is to turn AI usage into economic accounting: for a given workflow, which model or routing policy produces accepted work at the lowest all-in cost under a stated latency and verification standard?

The immediate artifact is a protocol for a first VKR AI Research benchmark. The benchmark should publish prompts, model versions, pricing snapshots, attempt logs, acceptance criteria, verification notes, and raw cost data. It should not claim to settle model quality. It should show how to price useful AI work.

One-Sentence Thesis

The cost of AI work should be measured in accepted task outputs, not generated tokens.

1. Motivation

The AI economy is increasingly organized around tokens. Providers price input tokens, output tokens, cached tokens, reasoning tokens, audio tokens, image tokens, and context tokens. Benchmarks report performance, latency, output speed, and sometimes cost. Buyers then try to infer whether a system is worth deploying.

That inference is harder than it looks.

The token is a convenient engineering unit, but it is not a unit of economic output. A million tokens can represent a correct code patch, a useless research draft, ten failed tool attempts, or a chain of hidden reasoning that improves the final answer. A low token price can be erased by a low accepted output share. A high token price can be justified if it reduces retries or review time. A model can look cheap on a dashboard and expensive inside a workflow.

The difference matters because frontier capability is moving from single-response chat toward multi-step work. Agentic systems retrieve files, write code, call tools, inspect state, retry after failure, ask verifier models to judge outputs, and escalate uncertain cases to humans. The bill is not just the final answer. It is the entire path from request to accepted artifact.

This paper introduces the "useful token" frame as a deliberately imperfect shorthand. Useful tokens are not directly observable. What is observable is the cost, latency, and verification burden required to produce accepted outputs. The right denominator is not output tokens. It is accepted work.

2. What the Public Data Already Shows

Public measurement work points in two directions at once.

First, inference is becoming cheaper at fixed capability levels. Epoch AI's 2026 trends page reports that the cost to inference a model at a fixed level of performance has fallen rapidly, with roughly 40x/year declines and a two-month halving time for fixed-performance LLM inference prices, while emphasizing unevenness across tasks.^{[epoch-trends]} In a related data insight, Epoch measured fixed-capability inference price declines across six benchmarks and found rates ranging from 9x to 900x per year, with the GPT-4-level price on one PhD-level science benchmark falling about 40x/year.^{[epoch-inference]}

Second, model cost is already becoming more granular than a single per-token price. Artificial Analysis separates input price, output price, blended price, cost per task, time-to-first-token, time-to-first-answer-token, output speed, end-to-end response time, and average reasoning tokens.^{[aa-methodology]} Its methodology notes that models with identical per-token prices can have different task costs when one produces longer answers or more reasoning tokens. Its Intelligence Index methodology goes further by reporting costs from provider token counts, including cached input, reasoning, and output tokens, across a weighted benchmark suite.^{[aa-intelligence]}

Third, the benchmark landscape shows why task-level accounting matters. SWE-bench reports the percentage of real-world software issues resolved, and SWE-bench Verified is a human-filtered subset of 500 instances.^[swebench] METR estimates model capability using task-completion time horizons: the human-expert task duration at which an AI agent is predicted to succeed at a given reliability level.^[metr-time] Stanford HAI's 2026 AI Index reports rapid progress on benchmarks such as SWE-bench Verified while also noting measurement gaps and uneven, jagged capabilities.^{[stanford-index]}

Together, these sources imply the core problem. Capability is rising, price is falling, and benchmark reporting is becoming more sophisticated. But a buyer still cannot infer deployed economics from price-per-token alone. A 90% success rate is not a cost number until the 10% failures, retries, fallback models, human review, latency misses, and downstream remediation are priced.

Measurement Gap

Existing public dashboards already do part of the job. Epoch tracks fixed-capability price declines. Artificial Analysis reports cost per task, reasoning tokens, and latency components. SWE-bench reports resolved software issues. METR reports task-completion time horizons. These are valuable measurements.

The missing layer is deployment accounting. A deployment does not choose a model in isolation. It chooses a route: prompt, context policy, model, fallback model, tool stack, verifier, reviewer, retry logic, timeout, and escalation rule. Two systems with the same model can therefore have different cost per accepted task. Two models with the same benchmark score can also produce different all-in costs once verification and latency are included.

The useful-token framework is designed to sit above model benchmarking and below enterprise financial planning. It asks a narrower question than "Which model is best?" and a more operational question than "Which model is cheap?" For a specified task family and acceptance standard, it asks which route produces accepted work at the lowest all-in cost.

3. Definitions

3.1 Token

A token is a model-specific representation of text or other content. It is a useful billing and throughput unit, but it is not semantically stable across providers, tokenizers, modalities, or model families.

Artificial Analysis distinguishes standardized token counts from native provider token counts for this reason.^{[aa-methodology]} Standardized token counts help compare speed on the same text. Native token counts are required to compute actual provider bills.

3.2 Attempt

An attempt is one end-to-end model route applied to one task. It may include one model call or many model calls. It may include retrieval, tool calls, judge calls, retries, or fallback routes. The attempt ends when the system submits an answer, times out, errors, or escalates.

3.3 Accepted Task

An accepted task is an attempt whose submitted output passes the pre-declared acceptance criteria for that task. Acceptance can be deterministic, test-based, rubric-based, expert-reviewed, outcome-based, or some combination of these.

3.4 Useful Output Unit

A useful output unit is the smallest accepted unit of work that the task definition treats as economically meaningful.

Examples:

• one accepted extracted row;
• one correct spreadsheet cell;
• one cited claim that survives audit;
• one patch that passes tests and review;
• one customer-support response that resolves a case without escalation;
• one legal issue/rule/application/conclusion criterion that passes rubric review.

3.5 Useful Token

A useful token is not directly observable. In this paper, "cost of useful tokens" means the share of total model spend that contributes to accepted work after failed attempts, retries, routing overhead, hidden reasoning, tool-call scaffolding, verifier calls, and rejected outputs are included.

The useful-token question is therefore not:

How much did each generated token cost?

It is:

How much total system cost was required to produce accepted work?

4. Economic Model

For a task family \(T\), model route \(r\), and task instance \(i\), define:

C_model(i, r) = direct model API cost
C_tool(i, r) = tool, retrieval, sandbox, or execution cost
C_verify(i, r) = verifier model cost + human or expert review cost
C_remediate(i, r) = cost of fixing downstream errors discovered after acceptance
L(i, r) = latency from request to accepted answer, timeout, or escalation
A(i, r) = 1 if accepted, 0 otherwise

The all-in cost of an attempt is:

C_attempt(i, r) =
  C_model(i, r)
+ C_tool(i, r)
+ C_verify(i, r)
+ C_remediate(i, r)

For a task set \(S\), the primary metric is:

Cost Per Accepted Task(r, S) =
  sum_i C_attempt(i, r) / sum_i A(i, r)

This metric intentionally penalizes failed attempts. If a system is cheap when it succeeds but fails often, its cost per accepted task rises. If a system is expensive per attempt but succeeds reliably and reduces verification, its cost per accepted task can be lower.

The model also supports a useful-output-unit version:

Cost Per Useful Output Unit(r, S) =
  sum_i C_attempt(i, r) / sum_i accepted_units(i, r)

This matters when outputs vary in size. A research synthesis task may produce ten cited claims. A structured extraction task may produce two hundred rows. A support task may produce one accepted resolution. Output count should follow the task's economics, not the model's tokenization.

5. Metrics

5.1 Cost Per Accepted Task

CPAT = total all-in attempt cost / accepted task count

This is the anchor metric. It includes failed attempts, retries, fallback models, routing overhead, verification calls, tool costs, and human review cost where measured.

5.2 Cost Per Useful Output Unit

CPUOU = total all-in attempt cost / accepted output units

This is useful for extraction, research, data QA, document review, and other tasks where one accepted task can contain multiple economically meaningful subunits.

5.3 Latency To Accepted Answer

LTAA = time from request to first accepted answer

This differs from time-to-first-token, time-to-first-answer-token, output speed, and end-to-end response time. Those are important system metrics, but LTAA is the workflow metric. A system can stream quickly and still take a long time to produce accepted work.

5.4 Verification Cost Per Accepted Task

VCPAT = verification labor + verifier model cost / accepted task count

Verification includes human review time, expert review time, test execution, judge-model calls, source audit, rubric scoring, and downstream QA. In many valuable domains, verification cost can exceed generation cost.

5.5 Waste Ratio

Waste Ratio = total generated or intermediate tokens / tokens in accepted output

Waste is not always bad. Reasoning-heavy systems may spend many hidden or intermediate tokens to improve acceptance. The purpose of the metric is to make the trade visible.

5.6 Escalation Rate

Escalation Rate = escalated tasks / total tasks

An escalation is not a failure if the workflow expects human review for uncertain cases. It is a cost and latency event that must be priced.

5.7 Acceptance-Adjusted Throughput

Accepted Tasks Per Hour =
  accepted task count / total wall-clock time

This matters for queue-based systems, customer operations, compliance review, and batch research work. Cheap accepted tasks are less useful if throughput is too low for the workflow.

6. Why Price Per Token Fails

6.1 It ignores output length

Two systems with identical per-token prices and similar quality can produce different bills if one writes longer answers. Length can be useful when it contains evidence, but verbosity is not value by itself.

6.2 It ignores reasoning overhead

Reasoning models can consume substantial intermediate tokens before producing the visible answer. This may improve acceptance, but the economic object is the accepted artifact, not the final message alone.

6.3 It ignores retries

If a model fails 30% of the time and the fallback path uses a larger model or human review, advertised per-token price understates deployed cost.

6.4 It ignores routing

Modern systems increasingly use routers: small models for easy cases, larger models for hard cases, verifier-gated fallback, ensemble voting, or judge reranking. The relevant price is the route cost distribution, not any one model's unit price.

6.5 It ignores verification

For high-stakes tasks, generation can be cheap relative to checking. A low-cost answer that requires expensive expert review can be less attractive than a higher-cost answer that reduces review burden.

6.6 It ignores latency windows

Some tasks are bounded by human experience or workflow deadlines. A cheaper system that misses the latency window can be economically worse than a more expensive system that completes inside it.

7. Benchmark Design

VKR's first benchmark should be small, repeatable, and intentionally boring. The goal is not to crown the best model. The goal is to demonstrate a methodology that turns model usage into cost accounting.

7.1 Task Selection Principles

Tasks should satisfy five constraints:

1. The task has a clear input artifact.
2. The output can be accepted or rejected by a pre-declared rule, test, rubric, or reviewer process.
3. Token usage, model calls, latency, and verification time can be logged.
4. The task is economically legible: an accepted output maps to a unit of work.
5. The task is small enough for repeated measurement but realistic enough to expose retries, ambiguity, or verification burden.

7.2 Initial Task Families

7.3 Model Routes

The benchmark should compare routes, not only models:

7.4 Logging Schema

Each attempt should produce a row with at least:

run_id
task_id
task_family
route_id
model_provider
model_name
model_version_or_snapshot
prompt_version
input_artifact_hash
pricing_snapshot_date
input_tokens_native
cached_input_tokens_native
output_tokens_native
reasoning_tokens_native_or_proxy
tool_call_count
verifier_model_calls
human_review_minutes
expert_review_minutes
first_token_latency_ms
first_answer_token_latency_ms
end_to_end_latency_ms
latency_to_accepted_answer_ms
attempt_status
accepted
accepted_output_units
acceptance_method
failure_mode
model_cost_usd
tool_cost_usd
verification_cost_usd
remediation_cost_usd
total_attempt_cost_usd
reviewer_notes

The benchmark should publish a data dictionary so readers know exactly how each field was collected.

7.5 Output Tables

Minimum aggregate table:

Minimum failure table:

8. Experimental Protocol

8.1 Pre-registration

Before running the benchmark, VKR should freeze:

• task list;
• prompt versions;
• model routes;
• acceptance criteria;
• verifier instructions;
• human-review procedure;
• pricing snapshot date;
• latency budget;
• exclusion rules;
• aggregation formulas.

This prevents post-hoc metric selection.

8.2 Repeats

For low-variance deterministic routes, one attempt per task may be enough for a first public artifact. For stochastic or agentic routes, repeated attempts are needed. The paper should report whether results are pass@1, best-of-n, or averaged across repeats.

Artificial Analysis uses pass@1 across many evaluations and documents its temperature, retry, and scoring choices.^{[aa-intelligence]} VKR should follow the same spirit of disclosure even with a smaller benchmark.

8.3 Pricing

Costs should be computed from published provider prices on a stated date. If a provider exposes cached tokens, reasoning tokens, or tool-call charges, these should be logged separately. If a field is unavailable, the benchmark should say so and avoid pretending precision.

8.4 Human Review Cost

Human review should be priced using a declared hourly rate, not left implicit. The benchmark can publish sensitivity tables at multiple reviewer rates:

The point is not to choose one universal rate. The point is to show how deployment economics change when verification is cheap or expensive.

8.5 Acceptance Criteria

Every task should include:

• pass condition;
• fail condition;
• partial-credit rule, if any;
• accepted output units;
• source of truth;
• reviewer role;
• examples of common failure modes.

If acceptance is subjective, the paper should say so. Subjectivity is not a reason to avoid measurement. It is a reason to disclose the review process.

8.6 Sensitivity Analysis

Every published result should include at least three sensitivity checks:

1. Reviewer-rate sensitivity: recompute costs at multiple human or expert hourly rates.
2. Acceptance-stringency sensitivity: show how costs change when partial credit is removed or stricter review is applied.
3. Pricing sensitivity: recompute model cost under at least one alternate pricing snapshot or provider route where available.

The purpose is to separate robust conclusions from artifacts of one price sheet, one reviewer cost assumption, or one acceptance threshold.

9. Expected Patterns

These are hypotheses, not results.

Hypothesis 1: The cheapest model per token will rarely be cheapest per accepted task.

Low per-token price matters, but accepted-output share, answer length, retries, and review burden can erase the advantage.

Hypothesis 2: Verification dominates generation for ambiguous work.

For tasks where correctness cannot be checked automatically, model cost may be small relative to human or expert review.

Hypothesis 3: Routing beats monolithic inference in mixed workloads.

If tasks vary in difficulty, static model choice wastes money. A route that escalates only uncertain cases should improve cost per accepted task.

Hypothesis 4: Latency constraints change the frontier.

The best model for offline research may be a poor model for interactive customer support or live analyst workflows.

Hypothesis 5: Reasoning-token spend has convex value.

Some reasoning overhead is valuable. Beyond a task-specific point, extra reasoning tokens may produce little acceptance improvement.

Hypothesis 6: Verifier quality determines whether small-model routes scale.

Small models become economically attractive when the system can cheaply know when they are wrong. If verification is weak, fallback logic becomes either too aggressive or too trusting.

10. Interpretation

The useful-token frame changes how model comparisons should be read.

A model with higher raw capability is not automatically more economical. It must be judged on accepted work under task-specific constraints. Conversely, a cheap model is not automatically efficient. It may spend less per attempt and more per accepted result.

The framework also changes how AI infrastructure should be evaluated. The valuable stack is not simply the stack that serves tokens cheaply. It is the stack that routes, verifies, retries, and escalates in ways that minimize all-in cost per accepted workflow.

This is especially important for AI agents. Agents can make many hidden economic decisions: how much context to retrieve, how long to reason, whether to call tools, when to stop, when to ask for review, and when to escalate. Those decisions are invisible in price-per-token comparisons but visible in attempt-level accounting.

11. What Would Change the Claim

The useful-token frame would be less important if three conditions held at once:

• per-token price strongly predicted cost per accepted task across domains;
• verification time was negligible relative to generation time;
• retry and fallback behavior did not materially change total spend.

That is unlikely for high-value knowledge work, but it should be tested rather than asserted. The first benchmark should publish enough raw attempt data for readers to recompute every cost metric under different price and verification assumptions.

12. Limitations

This paper is a measurement proposal, not an empirical benchmark result.

Important limitations:

• Published provider prices may not reflect enterprise discounts.
• Hidden reasoning-token accounting differs across providers.
• Tokenizer differences make some token comparisons unstable.
• Acceptance criteria can encode subjective judgments.
• Human verification time varies by reviewer expertise.
• Short benchmark tasks may not predict production workflows.
• Cost per accepted task can be gamed by weakening acceptance criteria.
• A narrow benchmark may overfit to tasks that are easy to measure.
• Remediation cost is difficult to observe unless failures are tracked after deployment.

The remedy is not to abandon task-level economics. It is to make acceptance criteria, verifier type, task distribution, pricing assumptions, and review cost explicit.

13. Minimum Publishable Benchmark

The first VKR benchmark should publish:

• 30 to 60 total tasks;
• 3 task families;
• 3 to 5 model routes;
• public task prompts where licensing permits;
• model/provider versions;
• prompt versions;
• pricing snapshot date;
• acceptance criteria;
• manual-review notes;
• CSV of attempts and outcomes;
• aggregate tables;
• clear limitations.

The release should be framed as a method demonstration. The claim is not "this model is best." The claim is "this is how to account for inference as useful work."

Reporting Standard

The benchmark should be easy to disagree with productively. A reader should be able to change the reviewer hourly rate, swap in a different pricing snapshot, remove a disputed task, or recompute cost per accepted task under a stricter acceptance rule.

Minimum release package:

• paper or technical memo;
• task cards;
• prompt files;
• model route definitions;
• verifier instructions;
• raw attempts CSV;
• aggregate tables;
• pricing snapshot;
• reviewer notes or anonymized review labels;
• exclusions and failed runs;
• license terms for task artifacts;
• reproducibility notes.

The strongest version of this work is not a static essay. It is a small public measurement kit that other researchers can rerun, critique, and extend.

14. Research Agenda

The full useful-token program has four follow-on questions:

1. Which task families show the largest gap between price-per-token and cost per accepted task?
2. When does verifier-gated fallback beat single-model inference?
3. How sensitive are deployment economics to reviewer hourly cost?
4. How much reasoning-token spend is economically justified before marginal acceptance gains flatten?

These questions connect AI economics, evaluation, deployment risk, and workflow design. That is the right surface for VKR AI Research: not model cheerleading, but measurement of the infrastructure and economics of useful AI work.

Appendix A: Example Task Card

task_id: research_citation_014
family: short_research_synthesis
input: two source documents + one question
output: 5 to 8 bullet synthesis with citations
accepted_output_unit: cited claim
pass_condition:
  - each cited claim is supported by the cited source
  - no source is used to support a stronger claim than it contains
  - no material contradiction between bullets
fail_condition:
  - unsupported factual claim
  - wrong citation
  - missing limitation where source uncertainty is material
verification:
  - reviewer audits every cited claim
  - reviewer records accepted_claim_count and rejected_claim_count
cost_fields:
  - model cost
  - reviewer minutes
  - verifier model calls, if any

Appendix B: Example Failure Taxonomy

References

^{[epoch-trends]} Epoch AI, "Trends in Artificial Intelligence", updated February 5, 2026.

^{[epoch-inference]} Ben Cottier, Ben Snodin, David Owen, and Tom Adamczewski, "LLM inference prices have fallen rapidly but unequally across tasks", Epoch AI, March 12, 2025.

^{[aa-methodology]} Artificial Analysis, "Language Model Benchmarking Methodology", accessed July 5, 2026.

^{[aa-intelligence]} Artificial Analysis, "Artificial Analysis Intelligence Benchmarking Methodology", accessed July 5, 2026.

^[swebench] SWE-bench, "Official Leaderboards", accessed July 5, 2026.

^[metr-time] METR, "Task-Completion Time Horizons of Frontier AI Models", last updated May 8, 2026.

^{[stanford-index]} Stanford HAI, "The 2026 AI Index Report", accessed July 5, 2026.