Previous post: Ray Casting in the Browser With Learned Maps Next post: Quiet Gears: The Search Finds Helical

Grounded Doc Chat

A website chatbot that answers from project pages, cites its sources, and refuses questions it cannot support.

Overview

Most chatbots can answer from general knowledge without making it clear where their claims came from. This chatbot takes a narrower approach: it retrieves relevant sections from the site’s project pages and uses them as the only sources for its answers.

Every factual claim must have a citation. If the retrieved pages do not support an answer, the chatbot refuses instead of falling back on general knowledge. Questions that fail the relevance check are refused before the language model is called.

How It Works

Question
↓
Retrieve the three most relevant sections
↓
Check the relevance score
├── Below threshold → Refuse without calling the model
└── Above threshold → Generate an answer with citations

Finding Relevant Pages

The site’s posts form the chatbot’s entire knowledge base. At build time, scripts/build-rag-index.mjs splits them into sections and writes the index to src/lib/rag-corpus.ts. Each section retains its post title and heading so the chatbot can identify where it came from.

When a question arrives, the server uses BM25, a keyword-ranking method, to find the three most relevant sections. It does not use vector embeddings or an external search service. The index is committed with the site, so retrieval has no separate service cost. Its scoring statistics use a fixed reference snapshot, preventing new posts from changing the baseline scores of existing content.

Answering From Sources

The three retrieved sections are passed to the language model as numbered sources. The model is instructed to cite factual claims using markers such as [1] and to say plainly when the sources do not answer the question.

There is no general-knowledge fallback. If the relevance score is too low, the server returns a refusal immediately, without calling the language model.

Using the Current Page

The chatbot also knows which page the visitor is viewing. Blog pages expose their URL, and the client sends it with the question. The server checks that URL against the site’s post list and gives sections from that post a 1.5× relevance boost.

A request to summarize the current page can use that page’s own sections, provided the question explicitly asks for a summary and refers to the page. Other questions are still searched across the entire site, and being on a project page does not exempt an unrelated question from the refusal check.

Deciding When to Refuse

The relevance threshold is fixed at 4.25. Answerable questions in the test set score above it; unrelated questions generally score below it or retrieve no matching sections. The threshold is calibrated against both kinds of test cases.

Common words such as can, could, should, might, me, and report are ignored during scoring. Without this filtering, ordinary wording or a short button label could make an unrelated question look relevant.

This gate is deliberately conservative, but it is not a universal test of whether a question is answerable. Its reliability depends on the site’s content, the wording of the question, and the test cases used to set the threshold.

Server and Request Handling

The retrieval endpoint, functions/rag.ts, shares several safeguards with the site’s agent endpoint:

  • A rate limit of 300 calls per hour per address.
  • Input cleanup and protection against user messages that try to introduce forged system instructions.
  • A single language-model gateway request, with fallback between openrouter/free and kilo-auto/free.

The retrieval endpoint keeps only the last user message after filtering the input. The agent endpoint uses a separate, explicit role filter.

The retrieval endpoint limits questions to 1,000 characters and replies to 1,024 tokens. The agent endpoint allows 2,000-character inputs and 2,048 reply tokens, plus a thinking budget. Replies shorter than 24 characters are rejected as errors rather than displayed.

The Chat Interface

A small button sits in the corner of each page. The chat code loads only when the visitor first opens the panel, so visitors who never use it do not pay the cost of loading the interface. If an open session was saved, the panel can reopen on the next visit.

The open state, conversation history, and current draft persist across page loads. On phones, the panel fills the screen. Pressing Escape closes it and returns keyboard focus to the chat button.

A status line displays retrieval time, total response time, the top relevance score, the current post, and whether the request was refused. Below each answer, the interface shows only the sources actually cited. Other retrieved sections remain hidden, and only the highest relevance score is displayed.

Tests and Evaluation

The offline retrieval tests live in src/__tests__/rag-corpus.test.ts. They run in continuous integration without network access and fail the build if retrieval performance regresses.

Test group Cases Result
Answerable questions 18 18/18 retrieve the expected post in the top three
Unanswerable questions 7 7/7 remain below the answer threshold
Page-aware questions 4 4/4 retrieve from the expected post
Refusals on a post page 2 2/2 are still refused

A second test harness, src/__tests__/rag-live.test.ts, exercises the real request handler with live model calls. Run it with npm run eval:live. It checks refusal flags, citation markers, and whether citations refer to retrieved material.

The volume test, npm run eval:volume, simulates multiple users and writes its results to eval-report.md. Both live harnesses are run manually rather than as part of npm run check. Since generated wording varies between calls, they test response structure and citation behavior rather than exact answer text.

These tests check retrieval and refusal behavior: whether the expected sections reach the model and whether the tested out-of-scope questions are stopped beforehand. The chat also displays cited excerpts so readers can check the answer against its sources themselves.

Limitations

Retrieval relies on literal word matches. A question that uses different wording from the relevant page may fail even when the answer is present. The current threshold is calibrated to this site’s content and must be rechecked as the corpus grows.

The test suite checks both sides of the threshold. If no single threshold can separate the answerable examples from the refusal examples, keyword retrieval is no longer sufficient and needs to improve.

The tests do not independently judge whether every generated statement is faithful to the retrieved text. A citation can point to a relevant passage without proving that the accompanying claim is correct. The system’s safeguards and tests therefore reduce the risk of unsupported answers, but do not guarantee that every response is accurate.

Displayed timing figures come from live requests and vary with retrieval and model response times.

References