# The Cost of Useful Tokens Date: 2026-07-05 Status: Working paper v0.2, full draft Track: VKR AI Research ## Abstract AI inference is usually priced in tokens, but economic usefulness is not. A token is a billing unit, a sampling unit, and sometimes a throughput unit. It is not a unit of work completed. This paper argues that AI systems should be compared at the level of accepted work rather than generated text. The economically relevant question is not "How cheap are tokens?" but "How much does an accepted unit of useful work cost, including latency, retries, routing, tool calls, verification, and rejected outputs?" As fixed-capability inference prices fall, reasoning models consume hidden or intermediate tokens, and agentic systems make multiple calls per task, raw price-per-token becomes a weaker proxy for deployed value. We propose a measurement framework centered on Cost Per Accepted Task, Cost Per Useful Output Unit, Latency To Accepted Answer, Verification Cost Per Accepted Task, and Waste Ratio. The framework is designed for small, repeatable task suites rather than broad leaderboard claims. Its purpose is to turn AI usage into economic accounting: for a given workflow, which model or routing policy produces accepted work at the lowest all-in cost under a stated latency and verification standard? The immediate artifact is a protocol for a first VKR AI Research benchmark. The benchmark should publish prompts, model versions, pricing snapshots, attempt logs, acceptance criteria, verification notes, and raw cost data. It should not claim to settle model quality. It should show how to price useful AI work. ## One-Sentence Thesis The cost of AI work should be measured in accepted task outputs, not generated tokens. ## 1. Motivation The AI economy is increasingly organized around tokens. Providers price input tokens, output tokens, cached tokens, reasoning tokens, audio tokens, image tokens, and context tokens. Benchmarks report performance, latency, output speed, and sometimes cost. Buyers then try to infer whether a system is worth deploying. That inference is harder than it looks. The token is a convenient engineering unit, but it is not a unit of economic output. A million tokens can represent a correct code patch, a useless research draft, ten failed tool attempts, or a chain of hidden reasoning that improves the final answer. A low token price can be erased by a low accepted output share. A high token price can be justified if it reduces retries or review time. A model can look cheap on a dashboard and expensive inside a workflow. The difference matters because frontier capability is moving from single-response chat toward multi-step work. Agentic systems retrieve files, write code, call tools, inspect state, retry after failure, ask verifier models to judge outputs, and escalate uncertain cases to humans. The bill is not just the final answer. It is the entire path from request to accepted artifact. This paper introduces the "useful token" frame as a deliberately imperfect shorthand. Useful tokens are not directly observable. What is observable is the cost, latency, and verification burden required to produce accepted outputs. The right denominator is not output tokens. It is accepted work. ## 2. What the Public Data Already Shows Public measurement work points in two directions at once. First, inference is becoming cheaper at fixed capability levels. Epoch AI's 2026 trends page reports that the cost to inference a model at a fixed level of performance has fallen rapidly, with roughly 40x/year declines and a two-month halving time for fixed-performance LLM inference prices, while emphasizing unevenness across tasks.[^epoch-trends] In a related data insight, Epoch measured fixed-capability inference price declines across six benchmarks and found rates ranging from 9x to 900x per year, with the GPT-4-level price on one PhD-level science benchmark falling about 40x/year.[^epoch-inference] Second, model cost is already becoming more granular than a single per-token price. Artificial Analysis separates input price, output price, blended price, cost per task, time-to-first-token, time-to-first-answer-token, output speed, end-to-end response time, and average reasoning tokens.[^aa-methodology] Its methodology notes that models with identical per-token prices can have different task costs when one produces longer answers or more reasoning tokens. Its Intelligence Index methodology goes further by reporting costs from provider token counts, including cached input, reasoning, and output tokens, across a weighted benchmark suite.[^aa-intelligence] Third, the benchmark landscape shows why task-level accounting matters. SWE-bench reports the percentage of real-world software issues resolved, and SWE-bench Verified is a human-filtered subset of 500 instances.[^swebench] METR estimates model capability using task-completion time horizons: the human-expert task duration at which an AI agent is predicted to succeed at a given reliability level.[^metr-time] Stanford HAI's 2026 AI Index reports rapid progress on benchmarks such as SWE-bench Verified while also noting measurement gaps and uneven, jagged capabilities.[^stanford-index] Together, these sources imply the core problem. Capability is rising, price is falling, and benchmark reporting is becoming more sophisticated. But a buyer still cannot infer deployed economics from price-per-token alone. A 90% success rate is not a cost number until the 10% failures, retries, fallback models, human review, latency misses, and downstream remediation are priced. ## Measurement Gap Existing public dashboards already do part of the job. Epoch tracks fixed-capability price declines. Artificial Analysis reports cost per task, reasoning tokens, and latency components. SWE-bench reports resolved software issues. METR reports task-completion time horizons. These are valuable measurements. The missing layer is deployment accounting. A deployment does not choose a model in isolation. It chooses a route: prompt, context policy, model, fallback model, tool stack, verifier, reviewer, retry logic, timeout, and escalation rule. Two systems with the same model can therefore have different cost per accepted task. Two models with the same benchmark score can also produce different all-in costs once verification and latency are included. The useful-token framework is designed to sit above model benchmarking and below enterprise financial planning. It asks a narrower question than "Which model is best?" and a more operational question than "Which model is cheap?" For a specified task family and acceptance standard, it asks which route produces accepted work at the lowest all-in cost. ## 3. Definitions ### 3.1 Token A token is a model-specific representation of text or other content. It is a useful billing and throughput unit, but it is not semantically stable across providers, tokenizers, modalities, or model families. Artificial Analysis distinguishes standardized token counts from native provider token counts for this reason.[^aa-methodology] Standardized token counts help compare speed on the same text. Native token counts are required to compute actual provider bills. ### 3.2 Attempt An attempt is one end-to-end model route applied to one task. It may include one model call or many model calls. It may include retrieval, tool calls, judge calls, retries, or fallback routes. The attempt ends when the system submits an answer, times out, errors, or escalates. ### 3.3 Accepted Task An accepted task is an attempt whose submitted output passes the pre-declared acceptance criteria for that task. Acceptance can be deterministic, test-based, rubric-based, expert-reviewed, outcome-based, or some combination of these. ### 3.4 Useful Output Unit A useful output unit is the smallest accepted unit of work that the task definition treats as economically meaningful. Examples: - one accepted extracted row; - one correct spreadsheet cell; - one cited claim that survives audit; - one patch that passes tests and review; - one customer-support response that resolves a case without escalation; - one legal issue/rule/application/conclusion criterion that passes rubric review. ### 3.5 Useful Token A useful token is not directly observable. In this paper, "cost of useful tokens" means the share of total model spend that contributes to accepted work after failed attempts, retries, routing overhead, hidden reasoning, tool-call scaffolding, verifier calls, and rejected outputs are included. The useful-token question is therefore not: ```text How much did each generated token cost? ``` It is: ```text How much total system cost was required to produce accepted work? ``` ## 4. Economic Model For a task family \(T\), model route \(r\), and task instance \(i\), define: ```text C_model(i, r) = direct model API cost C_tool(i, r) = tool, retrieval, sandbox, or execution cost C_verify(i, r) = verifier model cost + human or expert review cost C_remediate(i, r) = cost of fixing downstream errors discovered after acceptance L(i, r) = latency from request to accepted answer, timeout, or escalation A(i, r) = 1 if accepted, 0 otherwise ``` The all-in cost of an attempt is: ```text C_attempt(i, r) = C_model(i, r) + C_tool(i, r) + C_verify(i, r) + C_remediate(i, r) ``` For a task set \(S\), the primary metric is: ```text Cost Per Accepted Task(r, S) = sum_i C_attempt(i, r) / sum_i A(i, r) ``` This metric intentionally penalizes failed attempts. If a system is cheap when it succeeds but fails often, its cost per accepted task rises. If a system is expensive per attempt but succeeds reliably and reduces verification, its cost per accepted task can be lower. The model also supports a useful-output-unit version: ```text Cost Per Useful Output Unit(r, S) = sum_i C_attempt(i, r) / sum_i accepted_units(i, r) ``` This matters when outputs vary in size. A research synthesis task may produce ten cited claims. A structured extraction task may produce two hundred rows. A support task may produce one accepted resolution. Output count should follow the task's economics, not the model's tokenization. ## 5. Metrics ### 5.1 Cost Per Accepted Task ```text CPAT = total all-in attempt cost / accepted task count ``` This is the anchor metric. It includes failed attempts, retries, fallback models, routing overhead, verification calls, tool costs, and human review cost where measured. ### 5.2 Cost Per Useful Output Unit ```text CPUOU = total all-in attempt cost / accepted output units ``` This is useful for extraction, research, data QA, document review, and other tasks where one accepted task can contain multiple economically meaningful subunits. ### 5.3 Latency To Accepted Answer ```text LTAA = time from request to first accepted answer ``` This differs from time-to-first-token, time-to-first-answer-token, output speed, and end-to-end response time. Those are important system metrics, but LTAA is the workflow metric. A system can stream quickly and still take a long time to produce accepted work. ### 5.4 Verification Cost Per Accepted Task ```text VCPAT = verification labor + verifier model cost / accepted task count ``` Verification includes human review time, expert review time, test execution, judge-model calls, source audit, rubric scoring, and downstream QA. In many valuable domains, verification cost can exceed generation cost. ### 5.5 Waste Ratio ```text Waste Ratio = total generated or intermediate tokens / tokens in accepted output ``` Waste is not always bad. Reasoning-heavy systems may spend many hidden or intermediate tokens to improve acceptance. The purpose of the metric is to make the trade visible. ### 5.6 Escalation Rate ```text Escalation Rate = escalated tasks / total tasks ``` An escalation is not a failure if the workflow expects human review for uncertain cases. It is a cost and latency event that must be priced. ### 5.7 Acceptance-Adjusted Throughput ```text Accepted Tasks Per Hour = accepted task count / total wall-clock time ``` This matters for queue-based systems, customer operations, compliance review, and batch research work. Cheap accepted tasks are less useful if throughput is too low for the workflow. ## 6. Why Price Per Token Fails ### 6.1 It ignores output length Two systems with identical per-token prices and similar quality can produce different bills if one writes longer answers. Length can be useful when it contains evidence, but verbosity is not value by itself. ### 6.2 It ignores reasoning overhead Reasoning models can consume substantial intermediate tokens before producing the visible answer. This may improve acceptance, but the economic object is the accepted artifact, not the final message alone. ### 6.3 It ignores retries If a model fails 30% of the time and the fallback path uses a larger model or human review, advertised per-token price understates deployed cost. ### 6.4 It ignores routing Modern systems increasingly use routers: small models for easy cases, larger models for hard cases, verifier-gated fallback, ensemble voting, or judge reranking. The relevant price is the route cost distribution, not any one model's unit price. ### 6.5 It ignores verification For high-stakes tasks, generation can be cheap relative to checking. A low-cost answer that requires expensive expert review can be less attractive than a higher-cost answer that reduces review burden. ### 6.6 It ignores latency windows Some tasks are bounded by human experience or workflow deadlines. A cheaper system that misses the latency window can be economically worse than a more expensive system that completes inside it. ## 7. Benchmark Design VKR's first benchmark should be small, repeatable, and intentionally boring. The goal is not to crown the best model. The goal is to demonstrate a methodology that turns model usage into cost accounting. ### 7.1 Task Selection Principles Tasks should satisfy five constraints: 1. The task has a clear input artifact. 2. The output can be accepted or rejected by a pre-declared rule, test, rubric, or reviewer process. 3. Token usage, model calls, latency, and verification time can be logged. 4. The task is economically legible: an accepted output maps to a unit of work. 5. The task is small enough for repeated measurement but realistic enough to expose retries, ambiguity, or verification burden. ### 7.2 Initial Task Families | Family | Acceptance Method | Output Unit | Why It Matters | | --- | --- | --- | --- | | Structured extraction | schema checks + source spot review | accepted row | Common enterprise workflow; easy to price per row. | | Code repair | unit tests + patch review | accepted patch | Strong verification surface; comparable to software-agent benchmarks. | | Short research synthesis | citation audit | accepted cited claim | Tests hallucination, evidence use, and review burden. | | Financial data QA | spreadsheet checks | accepted answer cell | Natural fit for VKR Markets and AI Research overlap. | | Policy/compliance memo | rubric review | accepted criterion | Exposes expert-like verification without claiming legal advice. | | Customer support response | scenario rubric | accepted response | Measures resolution quality under latency constraints. | ### 7.3 Model Routes The benchmark should compare routes, not only models: | Route | Description | Expected Strength | Expected Weakness | | --- | --- | --- | --- | | Small model | One low-cost model attempt. | Low cost and latency. | Lower acceptance on ambiguous tasks. | | Frontier general model | One high-capability model attempt. | Higher acceptance. | Higher per-attempt cost. | | Small with fallback | Small model first; large model after failure. | Lower cost on easy cases. | Failure detection can be hard. | | Verifier-gated fallback | Small model plus verifier; escalate uncertain outputs. | Better cost-quality tradeoff. | Verifier adds cost and can be wrong. | | Judge-rerank | Multiple candidates; judge selects. | Higher acceptance where diversity helps. | More tokens and more latency. | ### 7.4 Logging Schema Each attempt should produce a row with at least: ```text run_id task_id task_family route_id model_provider model_name model_version_or_snapshot prompt_version input_artifact_hash pricing_snapshot_date input_tokens_native cached_input_tokens_native output_tokens_native reasoning_tokens_native_or_proxy tool_call_count verifier_model_calls human_review_minutes expert_review_minutes first_token_latency_ms first_answer_token_latency_ms end_to_end_latency_ms latency_to_accepted_answer_ms attempt_status accepted accepted_output_units acceptance_method failure_mode model_cost_usd tool_cost_usd verification_cost_usd remediation_cost_usd total_attempt_cost_usd reviewer_notes ``` The benchmark should publish a data dictionary so readers know exactly how each field was collected. ### 7.5 Output Tables Minimum aggregate table: | Route | Accepted Share | Mean Cost / Attempt | CPAT | Median LTAA | VCPAT | Escalation Rate | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | Small model | to be measured | to be measured | to be measured | to be measured | to be measured | to be measured | | Frontier general model | to be measured | to be measured | to be measured | to be measured | to be measured | to be measured | | Verifier-gated route | to be measured | to be measured | to be measured | to be measured | to be measured | to be measured | Minimum failure table: | Failure Mode | Definition | Count | Share Of Failures | Typical Cost Impact | | --- | --- | ---: | ---: | --- | | Schema failure | Output cannot be parsed. | to be measured | to be measured | retry or repair | | Incorrect answer | Output parses but fails acceptance. | to be measured | to be measured | review + fallback | | Unsupported claim | Claim lacks valid evidence. | to be measured | to be measured | citation audit | | Tool failure | Tool call, sandbox, or retrieval error. | to be measured | to be measured | rerun or manual handling | | Timeout | Route exceeds latency budget. | to be measured | to be measured | escalation | ## 8. Experimental Protocol ### 8.1 Pre-registration Before running the benchmark, VKR should freeze: - task list; - prompt versions; - model routes; - acceptance criteria; - verifier instructions; - human-review procedure; - pricing snapshot date; - latency budget; - exclusion rules; - aggregation formulas. This prevents post-hoc metric selection. ### 8.2 Repeats For low-variance deterministic routes, one attempt per task may be enough for a first public artifact. For stochastic or agentic routes, repeated attempts are needed. The paper should report whether results are pass@1, best-of-n, or averaged across repeats. Artificial Analysis uses pass@1 across many evaluations and documents its temperature, retry, and scoring choices.[^aa-intelligence] VKR should follow the same spirit of disclosure even with a smaller benchmark. ### 8.3 Pricing Costs should be computed from published provider prices on a stated date. If a provider exposes cached tokens, reasoning tokens, or tool-call charges, these should be logged separately. If a field is unavailable, the benchmark should say so and avoid pretending precision. ### 8.4 Human Review Cost Human review should be priced using a declared hourly rate, not left implicit. The benchmark can publish sensitivity tables at multiple reviewer rates: | Reviewer Rate | CPAT Small Model | CPAT Frontier Model | CPAT Verifier-Gated | | ---: | ---: | ---: | ---: | | $25/hour | to be measured | to be measured | to be measured | | $75/hour | to be measured | to be measured | to be measured | | $200/hour | to be measured | to be measured | to be measured | The point is not to choose one universal rate. The point is to show how deployment economics change when verification is cheap or expensive. ### 8.5 Acceptance Criteria Every task should include: - pass condition; - fail condition; - partial-credit rule, if any; - accepted output units; - source of truth; - reviewer role; - examples of common failure modes. If acceptance is subjective, the paper should say so. Subjectivity is not a reason to avoid measurement. It is a reason to disclose the review process. ### 8.6 Sensitivity Analysis Every published result should include at least three sensitivity checks: 1. Reviewer-rate sensitivity: recompute costs at multiple human or expert hourly rates. 2. Acceptance-stringency sensitivity: show how costs change when partial credit is removed or stricter review is applied. 3. Pricing sensitivity: recompute model cost under at least one alternate pricing snapshot or provider route where available. The purpose is to separate robust conclusions from artifacts of one price sheet, one reviewer cost assumption, or one acceptance threshold. ## 9. Expected Patterns These are hypotheses, not results. ### Hypothesis 1: The cheapest model per token will rarely be cheapest per accepted task. Low per-token price matters, but accepted-output share, answer length, retries, and review burden can erase the advantage. ### Hypothesis 2: Verification dominates generation for ambiguous work. For tasks where correctness cannot be checked automatically, model cost may be small relative to human or expert review. ### Hypothesis 3: Routing beats monolithic inference in mixed workloads. If tasks vary in difficulty, static model choice wastes money. A route that escalates only uncertain cases should improve cost per accepted task. ### Hypothesis 4: Latency constraints change the frontier. The best model for offline research may be a poor model for interactive customer support or live analyst workflows. ### Hypothesis 5: Reasoning-token spend has convex value. Some reasoning overhead is valuable. Beyond a task-specific point, extra reasoning tokens may produce little acceptance improvement. ### Hypothesis 6: Verifier quality determines whether small-model routes scale. Small models become economically attractive when the system can cheaply know when they are wrong. If verification is weak, fallback logic becomes either too aggressive or too trusting. ## 10. Interpretation The useful-token frame changes how model comparisons should be read. A model with higher raw capability is not automatically more economical. It must be judged on accepted work under task-specific constraints. Conversely, a cheap model is not automatically efficient. It may spend less per attempt and more per accepted result. The framework also changes how AI infrastructure should be evaluated. The valuable stack is not simply the stack that serves tokens cheaply. It is the stack that routes, verifies, retries, and escalates in ways that minimize all-in cost per accepted workflow. This is especially important for AI agents. Agents can make many hidden economic decisions: how much context to retrieve, how long to reason, whether to call tools, when to stop, when to ask for review, and when to escalate. Those decisions are invisible in price-per-token comparisons but visible in attempt-level accounting. ## 11. What Would Change the Claim The useful-token frame would be less important if three conditions held at once: - per-token price strongly predicted cost per accepted task across domains; - verification time was negligible relative to generation time; - retry and fallback behavior did not materially change total spend. That is unlikely for high-value knowledge work, but it should be tested rather than asserted. The first benchmark should publish enough raw attempt data for readers to recompute every cost metric under different price and verification assumptions. ## 12. Limitations This paper is a measurement proposal, not an empirical benchmark result. Important limitations: - Published provider prices may not reflect enterprise discounts. - Hidden reasoning-token accounting differs across providers. - Tokenizer differences make some token comparisons unstable. - Acceptance criteria can encode subjective judgments. - Human verification time varies by reviewer expertise. - Short benchmark tasks may not predict production workflows. - Cost per accepted task can be gamed by weakening acceptance criteria. - A narrow benchmark may overfit to tasks that are easy to measure. - Remediation cost is difficult to observe unless failures are tracked after deployment. The remedy is not to abandon task-level economics. It is to make acceptance criteria, verifier type, task distribution, pricing assumptions, and review cost explicit. ## 13. Minimum Publishable Benchmark The first VKR benchmark should publish: - 30 to 60 total tasks; - 3 task families; - 3 to 5 model routes; - public task prompts where licensing permits; - model/provider versions; - prompt versions; - pricing snapshot date; - acceptance criteria; - manual-review notes; - CSV of attempts and outcomes; - aggregate tables; - clear limitations. The release should be framed as a method demonstration. The claim is not "this model is best." The claim is "this is how to account for inference as useful work." ## Reporting Standard The benchmark should be easy to disagree with productively. A reader should be able to change the reviewer hourly rate, swap in a different pricing snapshot, remove a disputed task, or recompute cost per accepted task under a stricter acceptance rule. Minimum release package: - paper or technical memo; - task cards; - prompt files; - model route definitions; - verifier instructions; - raw attempts CSV; - aggregate tables; - pricing snapshot; - reviewer notes or anonymized review labels; - exclusions and failed runs; - license terms for task artifacts; - reproducibility notes. The strongest version of this work is not a static essay. It is a small public measurement kit that other researchers can rerun, critique, and extend. ## 14. Research Agenda The full useful-token program has four follow-on questions: 1. Which task families show the largest gap between price-per-token and cost per accepted task? 2. When does verifier-gated fallback beat single-model inference? 3. How sensitive are deployment economics to reviewer hourly cost? 4. How much reasoning-token spend is economically justified before marginal acceptance gains flatten? These questions connect AI economics, evaluation, deployment risk, and workflow design. That is the right surface for VKR AI Research: not model cheerleading, but measurement of the infrastructure and economics of useful AI work. ## Appendix A: Example Task Card ```text task_id: research_citation_014 family: short_research_synthesis input: two source documents + one question output: 5 to 8 bullet synthesis with citations accepted_output_unit: cited claim pass_condition: - each cited claim is supported by the cited source - no source is used to support a stronger claim than it contains - no material contradiction between bullets fail_condition: - unsupported factual claim - wrong citation - missing limitation where source uncertainty is material verification: - reviewer audits every cited claim - reviewer records accepted_claim_count and rejected_claim_count cost_fields: - model cost - reviewer minutes - verifier model calls, if any ``` ## Appendix B: Example Failure Taxonomy | Failure Mode | Description | | --- | --- | | Format failure | Output does not satisfy required structure. | | Extraction failure | Output misses or miscopies source data. | | Reasoning failure | Output uses facts but reaches wrong conclusion. | | Evidence failure | Output makes claims not supported by cited source. | | Tool failure | External tool or sandbox fails. | | Context failure | Model ignores or loses a relevant instruction. | | Over-escalation | Route escalates a task that should have been handled cheaply. | | Under-escalation | Route accepts a task that required review or fallback. | ## References [^epoch-trends]: Epoch AI, ["Trends in Artificial Intelligence"](https://epoch.ai/trends), updated February 5, 2026. [^epoch-inference]: Ben Cottier, Ben Snodin, David Owen, and Tom Adamczewski, ["LLM inference prices have fallen rapidly but unequally across tasks"](https://epoch.ai/data-insights/llm-inference-price-trends), Epoch AI, March 12, 2025. [^aa-methodology]: Artificial Analysis, ["Language Model Benchmarking Methodology"](https://artificialanalysis.ai/methodology), accessed July 5, 2026. [^aa-intelligence]: Artificial Analysis, ["Artificial Analysis Intelligence Benchmarking Methodology"](https://artificialanalysis.ai/methodology/intelligence-benchmarking), accessed July 5, 2026. [^swebench]: SWE-bench, ["Official Leaderboards"](https://www.swebench.com/), accessed July 5, 2026. [^metr-time]: METR, ["Task-Completion Time Horizons of Frontier AI Models"](https://metr.org/time-horizons/), last updated May 8, 2026. [^stanford-index]: Stanford HAI, ["The 2026 AI Index Report"](https://hai.stanford.edu/ai-index/2026-ai-index-report), accessed July 5, 2026.