Stop Comparing AI by Token Price—Track Cost per Accepted Outcome

The metric

Cost per accepted outcome equals all workflow cost divided by the number of outputs that pass the real acceptance standard. Include model and platform charges, tool calls, retries, human review and correction, failures, infrastructure, and allocated implementation or maintenance.

Cost per accepted outcome = (AI + tools + labor + failures + operations) / accepted outcomes

Define the outcome first

Use a unit a customer or operator actually values: an approved support resolution, merged code change, qualified record, compliant document, published asset, or reconciled invoice. “Response generated” is usually an activity, not an outcome.

Build an acceptance rubric

Write pass/fail checks before testing. Include factual correctness, completeness, policy compliance, brand or technical standards, required citations, accessibility, and maximum correction time. Use independent review for consequential work.

Run a representative sample

Test each candidate on the same 30 to 100 real, safely handled tasks. Preserve difficult and ordinary cases. Pin model and settings. Record input/output tokens, cache, tools, latency, retries, failures, review minutes, correction minutes, and final disposition.

Calculate the hidden denominator

If a cheap model produces 100 drafts but only 55 pass, divide total cost by 55—not 100. If a premium model produces 80 accepted outcomes with less review, it may be cheaper despite a higher token price. Report uncertainty when labor estimates or volumes vary.

Decide and keep measuring

Choose the workflow that meets quality and risk thresholds at the best sustainable cost. Monitor acceptance rate, severe errors, drift, latency, and cost after launch. Re-run the comparison when prompts, models, tools, prices, or source data change.