A chatbot that answers questions from your organisation's documents is one of the most common generative AI projects. A clear plan avoids the usual pitfalls.
1. Define the Scope
Which documents, which users, which questions? A narrow, well-defined scope — say, HR policies for staff — succeeds far more often than "answer anything about the company".
2. Prepare the Documents
Collect authoritative, current versions. Remove duplicates and outdated drafts. Convert PDFs and scans to clean text, and record source, date and access permissions.
3. Build Retrieval
Chunk documents along their structure, embed the chunks, and combine semantic and keyword search. Respect permissions so users only retrieve what they're allowed to see.
4. Write the Prompt
Instruct the model to answer only from the retrieved passages, cite its sources, and say when the answer isn't there. Keep the tone and length suited to users.
5. Evaluate Before Launch
Create 50–100 realistic questions with expected answers and source documents. Measure retrieval recall and answer correctness separately, and include questions the bot should decline.
6. Launch Carefully
Pilot with a small group, show sources with every answer, and provide an easy way to report wrong answers and reach a person.
7. Maintain It
Re-index when documents change, review flagged answers regularly, add failures to the test set, and assign an owner. Most document chatbots fail from stale content and neglect, not from the model.