How to Build an Internal Knowledge Base Your Team Can Ask Questions

Webisoft AI team Updated 5 min read

On this page

TL;DR: “Ask the wiki a question, get an answer with sources” is now an off-the-shelf capability, not a development project. The architecture underneath is retrieval-augmented generation (RAG): the system finds the relevant passages in your documents and the model answers from those passages, citing them. What separates the assistants people trust from the ones they abandon is everything around the model, a curated document set, grounding rules, permission-aware access, and someone owning the feedback loop. This page is the build sequence.

Why RAG, in one paragraph

A large language model alone knows nothing about your PTO policy, your deployment process, or your refund rules, asked anyway, it invents plausible answers, which is hallucination pointed directly at your operations. RAG changes the question from “what does the model know?” to “what do our documents say?”: retrieve the relevant passages, generate an answer constrained to them, cite the sources. Answer quality becomes a function of document quality, which is both the good news and the workload.

Setup: six steps, mostly editorial

  1. Scope the first corpus narrowly. Don’t index “everything.” Pick one high-question domain, IT how-tos, HR policies, or your support docs, where questions are frequent and answers are documented. A narrow assistant that’s right builds the trust a broad one will need. HR policy Q&A is a common first pick; the HR hub covers its specific sensitivities.
  2. Curate before you connect. The retrieval step happily serves the 2019 version of a policy next to the 2026 one, and the model can’t tell which is current. Before indexing: delete or archive superseded documents, resolve contradictions, add a “last reviewed” date to each doc, and split monster documents into single-topic pages (retrieval works on passages, one topic per page retrieves clean). If your processes aren’t written down yet, AI-assisted SOP writing is the fastest way to generate the corpus.
  3. Pick the platform where the documents already live.
    • Microsoft shop: Copilot answering over SharePoint/Teams content.
    • Google shop: Gemini over Drive.
    • Notion/Confluence/Slack: their native AI Q&A over workspace content.
    • Assistant-centric: ChatGPT or Claude projects/workspaces loaded with the document set, quick to pilot, more manual to maintain.
    • Custom RAG build: justified at large scale, many source systems, or strict architecture requirements. Not step one. The deciding factor is usually permissions: platform-native tools inherit existing file access, so the assistant can’t leak a document to someone who couldn’t open it anyway.
  4. Set the grounding rules as the assistant’s standing instructions (see below). Answers only from retrieved documents, citations always, honest refusals when coverage is missing.
  5. Pilot with one team for two weeks. Collect every wrong or missing answer. You’ll find most failures are documentation failures wearing an AI costume, the doc was stale, ambiguous, or absent.
  6. Launch with a feedback mechanism and an owner. A thumbs-down button (or a #kb-feedback channel) and a named person who triages misses weekly: fix the doc, adjust the rules, or add the missing page. An unowned knowledge assistant degrades into a confident liar within a quarter.

Example: the grounding instructions

“You answer employee questions using only the retrieved company documents provided with each question. Rules:

  • Answer from the documents. If they don’t contain the answer, say: ‘I don’t find this in the knowledge base’ and suggest who or which team to ask. Never answer from general knowledge about how companies usually do things.
  • Cite the source document (title + link) for every answer. If passages conflict, say so, cite both, and prefer the one with the newer ‘last reviewed’ date, flagging the conflict for the KB owner.
  • Quote policy specifics (numbers, deadlines, amounts) exactly as written.
  • For questions about an individual’s personal situation (their salary, their performance, their visa), do not answer, direct them to HR.
  • Note the ‘last reviewed’ date of the source when it’s older than 12 months.”

The refusal rule is the trust engine. An assistant that says “I don’t find this” when coverage is missing gets believed when it does answer. One that always answers gets one policy question wrong in week two, the screenshot circulates, and adoption dies. The conflict rule turns the assistant into a documentation-bug detector as a side effect.

Where this pays off next

The same grounded-answers pattern extends outward once the internal version works: your support team drafting customer replies from the same docs, sales answering RFP and security questionnaires from an approved answer library (see the sales hub), and eventually an AI agent that acts on answers, filing the ticket, not just explaining how. Walk before that run: the agent inherits every flaw in the knowledge base underneath it.

Pitfalls

  • Indexing the swamp. Connecting the whole drive, contradictions and all, produces authoritative-sounding answers from obsolete documents. Curation isn’t optional prep, it is the project.
  • No citations. Uncited answers can’t be verified or debugged. Citation on every answer, no exceptions.
  • Permissions as an afterthought. Test with a low-privilege account before launch: ask about salaries, ask about the M&A folder. If anything surfaces, stop.
  • No refusal path. Forced answers on missing coverage are manufactured hallucinations. The “I don’t find this” behavior is a feature, not a failure.
  • Nobody owns it. Docs drift, answers rot, trust evaporates. The weekly triage habit is what “maintenance” actually means here.
  • Measuring usage instead of accuracy. High question volume with silent wrong answers is worse than low volume. Sample real Q&A pairs monthly against the source docs.

The launch checklist

  • First corpus scoped to one domain with frequent, documented questions
  • Superseded docs archived; contradictions resolved; “last reviewed” dates added
  • Long docs split into single-topic pages
  • Platform inherits (or correctly enforces) source permissions, verified with a test account
  • Grounding rules set: documents-only, citations, refusals, exact policy quotes
  • Two-week pilot run; misses triaged into doc fixes
  • Feedback button + named owner + weekly triage slot
  • Monthly accuracy sample scheduled

FAQ

Do we need to build custom software for this? Rarely. Copilot, Gemini, Notion, Slack, and assistant projects/workspaces all ship grounded Q&A. Custom RAG is for scale and unusual source systems.

How is this different from just searching our wiki? It synthesizes an answer across documents and cites sources, instead of returning links. Better when answers span pages; riskier when docs are stale, hence curation and citations.

What stops it from making up answers? Documents-only answering, mandatory citations, an explicit refusal path, and a feedback loop that catches what slips through.

What about confidential documents, can everyone ask about everything? No, the assistant must inherit source permissions. Platform-native tools do this; verify with a low-privilege test account before launch.

Get one practical AI implementation brief per week, join the free newsletter.

Frequently asked questions

Do we need to build custom software for this?

Usually not anymore. Copilot answers over SharePoint/Teams content, Gemini over Google Drive, ChatGPT and Claude support project/workspace document collections, and Notion, Slack, and most helpdesks have native Q&A over their own content. Custom RAG builds make sense at large scale or with unusual source systems, not as a starting point.

How is this different from just searching our wiki?

Search returns documents; the assistant returns an answer synthesized from passages across documents, with citations. That's dramatically better when the answer spans three pages nobody reads, and worse if your docs are wrong, because a confident answer hides its stale source better than a search result does.

What stops it from making up answers?

Three things: restricting answers to retrieved passages, requiring source citations on every answer, and an explicit 'if the documents don't cover it, say so' rule. Then a feedback loop, because some hallucinated answers will still get through and you want them reported, not screenshot into Slack as policy.

What about confidential documents, can everyone ask about everything?

The assistant must respect source permissions: a question about salaries shouldn't surface an HR spreadsheet to whoever asks. Platform-native tools (Copilot, Gemini) inherit existing file permissions, which is a major reason to prefer them. Verify this with a test account before launch, not after.