Skip to content
ASSESSMENT AND IMPLEMENTATION

Find answers in your company’s approved knowledge.

An assistant that searches approved documents, cites sources and respects your team’s access permissions.

Let’s discuss your process

Choose questions your documentation can actually answer.

A useful first scope is one question family: opening a service request, choosing an equipment checklist or requesting business travel. For each question, verify that an approved, current and understandable source exists. If the procedure lives only in a colleague’s memory, document it before introducing an assistant.

Read the details

Retrieval-augmented generation, or RAG, retrieves relevant passages before generating an answer. It does not make every generated statement correct. Retrieval may find an obsolete version, the source may be ambiguous, and a citation may fail to support the sentence beside it. Those are different defects and need separate evaluation.

Worked scenario: which returns procedure applies?

Illustrative scenario: a team member asks how to handle a return for a discontinued product. The assistant finds the current procedure, shows its revision date and identifies the step requiring the commercial owner. If only an archived version exists, it explains that limitation instead of treating the old instruction as current.

Read the details
Expected behaviour before release
Question or conditionExpected outcome
Answer exists in the current procedureA summary linked to the supporting passage
Two instructions conflictAn explicit conflict and an owner to consult
Information outside the approved collectionAn explanation that the required source is missing
Document not authorised for this userNo retrieval, excerpt or citation from that document

Check permissions before content reaches the answer.

Document visibility must be enforced during retrieval. Asking the model to keep already-retrieved confidential material secret does not replace access control. The scope should also cover snippets, links, caches and previous conversations when a person’s authorisation changes.

Read the details

Azure guidance describes filtering search results by user or group identifiers and makes clear that the filter itself does not authenticate a user. Retrieval therefore needs a verified application identity. The appropriate implementation depends on the repository, available APIs and the organisation’s existing access model.

Build a testable path from sources to answers.

Read the details
  1. Curate the first collection

    Assign an owner, status, review date and audience to each collection. Define exclusions and what happens when a document is replaced, archived or deleted.

  2. Prepare questions and expected answers

    Include common questions, missing information, ambiguous requests and restricted topics. Evaluate retrieval, answer content, citation support and appropriate abstention separately.

  3. Run with a limited user group

    Collect unresolved questions and corrections without retaining unnecessary conversation data. Categorise each failure as a source, retrieval or answer problem so remediation addresses the underlying cause.

Design the answers the assistant must decline.

Retrieved documents can contain instructions that try to redirect a model. OWASP explains that RAG does not eliminate this risk. The initial scope separates information retrieval from actions in business tools, and includes adversarial document examples in the evaluation set.

Read the details

Traditional search may be enough for an exact code lookup. An assistant cannot maintain procedures that nobody owns. Ask for source maintenance, deletion handling and a clear path for unanswered questions in the proposal. For multilingual teams, evaluate the supported question and document languages explicitly.

FROM IDEAS TO A BRIEF

A template to work from.

Knowledge source register

Sources with valid versions, access rules, updates and retrieval tests.

Download the Markdown template

AI permission matrix

A permission matrix with approver, tests and a revocation procedure.

Download the Markdown template

AI evaluation dataset

Cases with expected results, criteria and a separate final evaluation set.

Download the Markdown template

Practical questions

Can we connect an entire Drive or SharePoint repository?

First check its structure, permissions, versions and available access. Start with bounded collections. A connector’s existence does not demonstrate that it correctly propagates every authorisation change or document deletion.

How do we identify invented answers?

Use a set of known-answer and unanswerable questions. Reviewers check whether each material claim is supported by its cited source. A visible link is helpful, but it is not sufficient evidence that an answer is correct.

References and method

Microsoft Learn — Security filters in Azure AI Search

Technical context for identity-based result filtering; authentication and permission updates remain application requirements.

OWASP — LLM01:2025 Prompt Injection

Risk of malicious instructions in documents consulted by a language model.

Which process should improve first?

Let’s discuss your process