Atlas Intelligence

Mobile application

The challenge

Atlas had a demo that impressed prospects and a product that couldn't ship. The assistant answered confidently regardless of whether the knowledge base contained the answer, and there was no objective way to determine whether a prompt change had improved or degraded quality — releases were gated on someone's subjective read of twenty sample conversations. Two pilot customers had paused deployments after the assistant gave confidently wrong billing guidance.

Research

• Manual review of 1,200 assistant conversations classified by failure mode
• Coverage analysis mapping ticket categories against actual knowledge-base content
• Interviews with support leads at both paused pilot accounts
• Baseline measurement of human agent resolution rate, time, and CSAT by category

The solution

We built the evaluation harness before touching the model. Every prompt and model change now runs against a versioned eval set including an adversarial subset built from real failure cases, and a regression gates the CI pipeline. On the retrieval side, hybrid search combines dense and lexical retrieval, and a grounding verification step checks that each generated claim traces to retrieved content — ungrounded claims trigger escalation rather than delivery. The system's most important capability turned out to be recognizing what it couldn't answer.

UX decisions

Features

• Hybrid dense and lexical retrieval over versioned corpora
• Grounding verification with claim-level source tracing
• Evaluation harness gating every prompt and model change in CI
• Adversarial eval set built from production failure cases
• Configurable topic guardrails with escalation thresholds
• Agent console with full assistant context on handoff
• Knowledge freshness and coverage-gap reporting

Technology

Architecture

Documents are chunked, embedded, and indexed with both vector and BM25 representations; corpus versions are immutable so any answer can be reproduced against the exact knowledge state that produced it. A FastAPI orchestration layer runs retrieval, generation, and grounding verification as separate stages with independent timeouts — a slow verification step degrades to escalation rather than to an unverified answer. Redis caches embeddings and frequent retrievals. The eval harness runs nightly and on every pull request, publishing scores to a dashboard that product and support leadership both watch.

Results

The assistant resolves 58% of tickets end-to-end. Hallucination rate against the adversarial eval set sits at 0.4%, down from an unmeasured but review-estimated 12% at engagement start. Cost per resolution fell 71% versus the human baseline. Notably, CSAT on assistant-resolved tickets runs 0.3 points above human-handled ones — largely a speed effect on simple issues. Both paused pilots resumed and converted.

Lessons learned

1. 01Building the eval harness first felt like a delay and was the entire project. Without it, every subsequent change would have been a guess.
2. 02The highest-value capability was refusal. Teaching the system to escalate cleanly did more for adoption than any accuracy improvement.
3. 03Immutable corpus versioning was added reluctantly for reproducibility and became essential for debugging — 'the KB changed underneath it' explained a large share of otherwise-mysterious regressions.