Grounded AI Support That Knows When to Ask for Help

Customer context

NovaDesk builds customer support software for mid-market SaaS teams. As their own customer base grew, their internal support volume grew with it — an ironic position for a support software company to be in, and one their team was acutely aware of.

The challenge

NovaDesk's support ticket volume had roughly doubled over eighteen months while headcount grew much more slowly. A significant share of tickets were repeat questions already answered in their documentation, but customers weren't finding those answers, and their initial attempt at a chatbot — built on a general-purpose model with a long system prompt — produced confident-sounding answers that were sometimes simply wrong. That was worse than the status quo: agents now had to catch AI mistakes on top of handling tickets.

Requirements

• Grounded, sourced answers

  Responses needed to come from actual documentation, with visible sources, not general model knowledge.

• Reliable escalation

  The system needed to recognize its own uncertainty and hand off to a human rather than guess.

• Agent oversight option

  Support leadership wanted the ability to review AI drafts before they reached customers, at least initially.

• Integration with existing inbox

  The solution had to work within NovaDesk's existing support tooling, not replace it.

Discovery

We spent the first two weeks with NovaDesk's support team before writing any code — reviewing a sample of resolved tickets, understanding which categories of questions were genuinely repetitive versus which only looked similar on the surface, and identifying where their prior chatbot attempt had failed. A recurring theme: the previous system had no concept of “I don't know” — it always produced an answer, confident or not. That became the central design constraint for what we built.

Solution

We built a retrieval-augmented support assistant grounded in NovaDesk's help center, product documentation, and a curated set of previously resolved tickets vetted by their support lead. Every response is generated from retrieved source material, with citations shown to both the customer and the agent. A confidence-scoring step — combining retrieval relevance, question complexity signals, and customer context — determines whether a response is sent automatically, held for agent review, or routed directly to a human.

Technical architecture

Incoming tickets are embedded and matched against a vector index built from NovaDesk's help center and documentation, refreshed nightly through an ingestion pipeline. Retrieved passages are passed to the language model alongside the ticket content to produce a grounded draft. The confidence-scoring step runs before any response is surfaced, and the routing decision (auto-send, queue for review, or escalate) is based on that score combined with simple business rules NovaDesk's team helped define — for example, always escalating anything mentioning billing disputes or account cancellation, regardless of confidence score.

Implementation approach

We shipped in review-only mode first: for the first several weeks, every AI-generated response went to an agent for review before sending, regardless of confidence score. This let NovaDesk's support lead build trust in the system's accuracy and tune the confidence threshold based on real data, rather than our assumptions. Auto-send was enabled gradually, category by category, starting with the highest-confidence, lowest-risk question types.

Key features

• Grounded response generation

  Every answer is generated from retrieved documentation, not open-ended model knowledge.

• Source citations

  Customers and agents can see exactly which help article an answer came from.

• Confidence-based routing

  Low-confidence tickets go to a human instead of getting an automated guess.

• Category-based rollout controls

  Auto-send can be enabled or disabled per question category, not all-or-nothing.

Integrations

• Support inbox

  Direct integration with NovaDesk's existing ticketing system.

• Help center CMS

  Nightly ingestion keeps the knowledge base in sync with published articles.

Challenges & decisions

The hardest design decision was where to draw the confidence threshold. Too conservative, and the system rarely helps; too aggressive, and it repeats the previous chatbot's failure mode. We resolved this by not trying to set the right number upfront — instead building the review-only rollout specifically so the threshold could be tuned against real outcomes rather than a guess, and by making the threshold configurable per question category rather than a single global setting.

Representative outcomes

Technology stack

In their words

Related services