DiscoverAI AI Support Resolution Benchmark: Methodology

This is DiscoverAI's public protocol for comparing AI support systems with identical cases, knowledge, permissions, and review rules. It measures durable accepted resolution, not containment or vendor-reported automation, and publishes no product ranking until collection and adjudication are complete.

For a fixed support operation, which configured system correctly and durably resolves eligible customer needs, escalates the rest with useful context, follows policy, minimizes customer and reviewer effort, and does so at a reproducible total cost? The benchmark evaluates a product, model, knowledge base, integrations, settings, and operating process together—not a timeless model or a polished demonstration.

Build a versioned set from de-identified historical cases and authored edge cases. Stratify by intent, channel, language, customer segment, complexity, risk, required action, and expected disposition. Include ambiguous requests, missing identity, outdated knowledge, conflicting policy, multiple intents, emotional customers, prompt injection, duplicate contacts, revoked credentials, rate limits, and downstream outages. Keep a hidden holdout set to detect tuning to the public examples.

Each case receives a reference packet: permitted knowledge, applicable policy, required evidence, allowed actions, acceptance criteria, correct escalation destination, severity, and reopen window. Remove personal data and secrets unless a controlled test explicitly requires synthetic equivalents.

Give every system the same eligible cases, knowledge snapshot, integration fixtures, customer history, permissions, time budget, and escalation staff. Record product, plan, model, version, region, channel, configuration, prompts, tools, retrieval sources, identity state, approvals, timestamps, usage, and failures. Differences that cannot be normalized remain visible limitations rather than disappearing into one score.

Run three phases. Offline replay tests answers and proposed actions without customer impact. Shadow mode observes real work while humans remain authoritative. A small limited-live cohort follows only after the system clears the preregistered offline and shadow gates; consequential actions stay approval-gated.

A case counts as a durable accepted resolution only when the answer or action is correct, supported by the allowed evidence, compliant with policy, authorized, complete, and not reopened for the same need inside the declared window. Abandonment, silence after an answer, duplicate closure, unsupported claims, hidden transfers, partial completion, and incorrectly closed cases do not qualify.

Publish the numerator, denominator, eligibility exclusions, confirmed versus inferred outcomes, and reopen window. Never substitute containment, answer rate, deflection, conversations handled, or a vendor's billable unit for independent acceptance.

Report factual and evidential correctness, action correctness, policy compliance, privacy and authorization, appropriate abstention, escalation timing, destination and context quality, customer effort, latency, correction time, reviewer minutes, reopen rate, abandonment, total operating cost, and cost per durable accepted resolution.

Severe events remain separate from averages: privacy exposure, unauthorized action, material financial or account change, unsafe advice, discrimination, evasion of escalation, corrupted records, or unrecoverable customer harm. One severe event cannot be averaged away by many easy successes.

Two trained reviewers independently score a stratified sample and every severe or ambiguous case using the frozen rubric. Blind product identity where the transcript and action record permit it. Reviewers cite the source, policy, or system state supporting each judgment. Disagreements receive documented adjudication; agreement and unresolved cases are published.

Customer feedback is useful but not sufficient: a positive rating cannot make an incorrect or unauthorized action acceptable, and silence cannot prove resolution. Automated graders may assist triage only after validation against human decisions and must not be the sole judge of their own class of system.

Total cost includes platform fees, billable outcomes or conversations, model and tool usage, implementation amortization, integrations, monitoring, human review, corrections, support, and incident response. Report volume, eligible-case rate, uncertainty intervals, medians where appropriate, and results by intent, channel, risk, and language. Do not publish a composite score unless every weight is declared before collection and all component results remain available.

No leaderboard or winner ships until every included system completes the declared sample or is clearly labeled incomplete; every accepted resolution has inspectable evidence; severe events and reviewer disagreements are adjudicated; costs reconcile to logs or invoices; and the limitations review is complete.

The results package should expose a versioned rubric, case metadata without customer content, configuration sheet, observation schema, exclusions, aggregate calculations, conflicts, corrections, and change log. Raw transcripts or private operational data are released only when consent, security, and licensing permit. Until that gate is met, this page is methodology—not performance evidence.

Teams can adapt the rubric for procurement without waiting for DiscoverAI results. Freeze representative cases, define acceptance and severe failures, establish a human baseline, give finalists matched conditions, and progress from offline replay to shadow mode to a limited live cohort. Choose from durable accepted outcomes, customer impact, reviewer burden, recovery, and full cost—not the vendor with the highest self-reported automation rate.