How to Add an AI Chatbot to an Existing Product Without Breaking It

Adding an assistant to a product that already has users is a different job from building a chatbot demo. The demo has to impress once; the assistant has to be right, safe and affordable thousands of times a day, inside a product people already trust. This guide is the plan we follow when a client asks us to add an LLM feature to a live product. It is written for product and engineering leads, and it assumes you are integrating a hosted model rather than training one.

Step 1: decide the one job it does

The most common failure is scope. “Add AI chat” is not a feature; it is an invitation for the model to attempt everything and be mediocre at all of it. Pick one job, in one sentence, that your users already try to do and that your data can support:

• Answer product and account questions from your documentation and the user’s own records.
• Draft a message, report or listing from structured data the product already holds.
• Guide a user through a multi-step workflow, filling the form as they describe what they need.
• Summarise or classify incoming items (tickets, applications, reviews) for a human to act on.

The sentence decides everything downstream: what data to retrieve, which actions to allow, how to evaluate, and what the assistant should refuse. Everything else goes on the “later” list, exactly as it would for any other first release.

Step 2: ground it in your own data (retrieval)

A hosted model knows nothing about your product. The standard way to fix that without training is retrieval-augmented generation: at request time, find the handful of passages from your own content that are relevant to the question and put them in the prompt. The original RAG paper describes the idea; in production it comes down to four engineering decisions.

What goes in the index. Documentation, help-centre articles, policies, product catalogue, and, for logged-in users, their own records. Each source needs an owner and a refresh schedule; stale documentation produces confident, outdated answers.

How it is chunked. Split content into passages that make sense on their own (a section, a FAQ entry, a product record), keep the title and URL with each chunk, and avoid splitting mid-sentence. Chunking quality shows up directly in answer quality.

How it is searched. Vector similarity over embeddings catches paraphrases; keyword search catches exact product names, error codes and SKUs. Most production systems use both and merge the results. PostgreSQL with the pgvector extension is enough for the majority of products and keeps the data where your other data already is.

Permissions. Retrieval must respect the same access control as the rest of the product. A user’s documents are retrieved only for that user; internal notes never reach the index. This is the single most important security property of the feature, and it is enforced in the retrieval query, not in the prompt.

Step 3: let it act, carefully (tool calls)

An assistant that can only talk is limited; one that can look up an order, create a draft or change a setting is useful. Every major provider supports tool calling, where the model asks your code to run a named function with structured arguments. The OpenAI function-calling guide and the Claude tool-use documentation describe the mechanics; the design principles are the same whichever you use.

• Tools are ordinary API calls with ordinary authorisation. The model proposes; your code checks the current user’s permissions and validates every argument against a schema before anything runs.
• Read tools first, write tools later. Lookups are low-risk. Anything that changes state (sending a message, placing an order, changing a plan) needs a confirmation step shown to the user before it executes, at least until you have months of evidence.
• Keep the tool list short and the descriptions precise. Ten well-described tools outperform forty vague ones, and cost fewer tokens on every request.
• Log every call with its arguments and result. This is both your audit trail and your evaluation data.

Step 4: guardrails

Guardrails are the parts of the system that keep the assistant inside its job. They live in code and configuration, not only in the prompt.

Scope. The system prompt states the job, the tone and what to refuse, and gives the model a graceful way out (“I can help with X; for Y, here is how to reach support”). Test the refusals; they fail more often than the happy path.

Prompt injection. Retrieved content and user-supplied documents can contain instructions aimed at the model. Treat all retrieved text as data: never let it change which tools are available or what the assistant is allowed to do, and keep write actions behind user confirmation. The OWASP Top 10 for LLM applications is the reference list of risks; prompt injection is its first entry for a reason.

Personal data. Decide what the assistant may see and what is sent to the provider. Strip or mask identifiers that are not needed for the task, and check the provider’s data-processing terms against the privacy law that applies to your users (DPDP in India, GDPR in Europe, the state laws in the US). The NIST AI Risk Management Framework is a useful structure for writing this down.

Output checks. Validate structured outputs against a schema. For free text, add a lightweight check for things that must never appear (other customers’ data, internal URLs, prices the product does not have).

Rate limits and abuse. A chat endpoint is an expensive API. Rate-limit per user and per IP, cap conversation length and context size, and watch for automated use.

Step 5: evaluation before launch, and after every change

This is the step that separates a feature from a liability. Before launch, build an evaluation set: fifty to a few hundred real questions (from support tickets, search logs, sales calls) with the answer you would accept, and a set of questions the assistant must refuse. Run it on every change to the prompt, the retrieval, the model or the tools, and track:

• Answer quality, judged by a person on a sample and by a model against the reference answer for the rest.
• Groundedness: does the answer cite retrieved sources, and do the sources actually support it?
• Refusal accuracy: does it decline what it should, and not decline what it should not?
• Tool-call correctness: right tool, valid arguments, right outcome.
• Latency and cost per conversation.

Open-source frameworks such as Ragas and promptfoo run these comparisons in CI so that a prompt edit cannot silently regress quality. The evaluation set grows from production: every escalated or thumbs-down conversation becomes a test case.

Step 6: architecture that fits the product you have

The assistant should be a bounded component, not a new way of doing everything.

• A single service owns prompts, retrieval, tool definitions and the provider call, exposed to the product over an internal API. The front end streams tokens from it; the rest of the product does not know which model is behind it.
• Streaming matters for perceived speed: show tokens as they arrive and show the sources as soon as retrieval returns.
• Provider as a dependency. Keep the provider SDK behind an interface so that a model or vendor change is configuration. Route simple questions to a smaller, cheaper model and reserve the large one for hard ones.
• Caching. Identical or near-identical questions, and the static part of the prompt, should not be paid for twice. Most providers now support prompt caching for the fixed prefix.
• Observability. Trace each conversation end to end: retrieval results, prompt, tool calls, tokens, latency, cost, and the user’s feedback. You will need this on the first day a user reports a wrong answer.

Step 7: cost control

Costs are proportional to tokens, and tokens are controllable. Set a budget per conversation and per month with alerts before launch. Keep retrieved context tight (the best five chunks, not the best fifty). Cap conversation history, summarising older turns rather than resending them. Measure cost per resolved conversation against what the same work costs by email or phone; that comparison is how the feature pays for itself, and it is the number leadership will ask for.

Step 8: rollout

Ship it the way you would ship any feature that touches trust.

1. Internal first, to your support and product teams, with a feedback button on every answer.
2. A small cohort of users, with the assistant clearly labelled as such and an obvious path to a human. Compare resolution rate and escalation rate against the same cohort before.
3. Widen as the evaluation set and the production metrics hold. Keep the human path forever; the assistant handles the routine so that people can handle the rest.

Tell users what the assistant can and cannot do, what data it uses, and that answers are generated. Honesty here is a product decision as much as a legal one, and it is the difference between an assistant users rely on and one they route around.

Where to start

If you have an existing product and a backlog of repetitive questions or tasks, you have the ingredients. The first week is the one-sentence job, the data inventory and the evaluation set; the build follows. Our AI feature sprint is scoped around exactly this shape, and our Python and Node.js teams have built the retrieval, tooling and evaluation layers described here. If you would like a second opinion on where an assistant fits in your product, a tech lead will look at it with you.