Expense-policy questions keep arriving in the same inbox. People answer from memory, and an old threshold or exception slips in. The useful agent here is deliberately narrow: it reads the current policy files, answers from those files, and says when the answer isn't there. Agent Builder can get you to a working version in minutes. Most of the lesson is about what comes next.
Make four decisions before you build
Think of the agent as one Copilot experience with one job. Its reliability comes from four decisions:
- Purpose names the job and the people it serves.
- Instructions set the answer style, uncertainty rules, and boundaries.
- Knowledge limits the facts it may use.
- Sharing controls who can use this version.
Agent Builder is a no-code surface. You can describe the agent in natural language or fill in its details through the documented Configure method. A Microsoft 365 Copilot license includes the agents you build there. Keep the first version focused. One clear job and an explicit refusal rule are much easier to test than an agent that tries to cover every employee question.
Write the four decisions down before opening the tool. In one page or less, name the job, audience, approved sources, and expected response when information is missing, ambiguous, or outside the job. Hand that definition to someone else. If they can't give you one question the agent should answer and one it should decline, the boundary still needs work.
Put rules in instructions and facts in knowledge
Instructions tell the agent how to behave. They cover its role, answer format, uncertainty, and scope. Knowledge supplies the facts. Keep policy details that can change in maintained documents, and use the instructions to explain how the agent should read them. If the $500 approval threshold appears in both places, a later policy change leaves you with two versions to reconcile.
Grounding is the work of tracing each factual statement back to evidence in the selected sources. Smooth prose isn't evidence. Find the passage behind each claim. If you can't, mark the claim unsupported even if it sounds sensible. Agent Builder can use SharePoint sites, folders, and files as knowledge, up to twenty sources per agent. It respects file sensitivity labels. You remain responsible for checking whether each source is appropriate for the intended audience.
Suppose the expense agent has three documents: a policy, an approval matrix, and a travel FAQ. You ask, "What approval is required for a $750 purchase, and when must I submit it?" The expected answer says that the $750 purchase needs department-manager approval and must be submitted within ten business days. It also names the document supporting each fact. With a small source set, that check is quick. Add another source when a test reveals a real gap, not because a longer list feels safer.
Test the places where guessing is tempting
Use four question types. For each run, save the prompt, response, supporting passage, and pass or fail result:
The unknown test often tells you the most. Ask about an exception you know the documents never define, such as "Can emergency purchases bypass approval?" A weak agent may borrow a plausible rule from general business knowledge and present it as policy. You chose this question because the correct answer is absence.
When something fails, don't rewrite the whole instruction block. Find the rule connected to the failure, change that rule, and run all four questions again. The full rerun matters because a fix for one case can disturb another. Trust comes from an observed failure, a targeted correction, and the same test passing afterwards. Extra instructions without a retest are still unproven.
Wider sharing comes after the pilot
Agent Builder offers three sharing choices, and you climb them in order. Start at Only you for testing. Move to Specific users (individuals, security or Microsoft 365 groups, or a Team) for a controlled pilot, and have a pilot user repeat your four tests. Choose Anyone in your organization only after the content owner approves the instructions, the knowledge, and the tested behaviour. If a choice isn't available, that can be a tenant-wide sharing default rather than a mistake in your agent. An administrator controls those defaults, so record what you see and ask rather than assuming your definition is wrong.
When you do pilot, treat the pilot user's run as evidence, not a formality: have them work the same known, unknown, ambiguous, and out-of-scope prompts, record any difference from your results, and fix failures before widening further. Write down who approved the content, the date, and who owns future source changes. An agent nobody maintains drifts out of date the moment a policy does.
Templates (Customer Insights Assistant, Interview Question Assistant, Quiz Tutor, Scrum Assistant, and a broader set of agent examples) are starting structures, not finished agents. Audit any template against the same four parts before you use it. Does its purpose name one job and the right audience? Are the generated instructions vague or inconsistent with your process? Does every selected source support the purpose? Does the audience match the current stage? Treat any unreviewed generated instruction as a fail, replace generic phrasing with your explicit rules, and remove any source you can't justify.
One more pitfall guards the knowledge list: if you can't say which test requires a particular source, take it out of the initial configuration. An unexplained source is a place for the agent to wander. Broad sharing multiplies the cost of a wrong answer and can expose content it shouldn't. Widen the audience only after the tests and the content owner both say yes.
