AI Agents And Their Limits
AI agents are systems that plan a task, call tools (like search, calculators, calendars, or internal databases), and produce an output that may trigger further actions. In 2026, many agents run as a loop: the model proposes a step, a tool returns results, the model updates its plan, and the loop ends when a stopping rule triggers. A practical example is a “medication refill assistant” that drafts questions for a clinician, checks a formulary list from a connected database, and then prepares a message for a patient portal. The agent’s behavior depends on three design choices: tool access, memory scope, and permission boundaries. When any one of those is sloppy, the agent can become confident while being wrong, or it can act on data it should not touch.
Common Pain Points And Misreads
People often treat an agent like a chat box with extra steps, then assume it “remembers” everything the user said. Many systems only retain conversation context for a limited window, and some store summaries rather than raw text. That distinction matters for health-adjacent tasks because a summary can omit dosage details, timing, or contraindication flags. Another frequent misread is confusing tool access with data access. An agent might be allowed to call a “drug lookup” tool but not allowed to read a patient’s full record, which changes what it can safely answer.
Supporting technologies shape the outcome. Tool calling typically relies on a structured interface (often JSON-like arguments) that the agent fills in; if the schema is wrong, the tool returns errors or partial results. Retrieval systems add another dependency: if a knowledge base is outdated, the agent can cite plausible but stale information. Memory systems add a third dependency: “long-term memory” features often store user preferences or extracted facts, and the storage policy varies by product. I saw a demo in March 2026 where the agent stored a user’s “preferred pharmacy” but not their allergy list, which made later advice feel inconsistent—annoying, and potentially risky.
Permissions are where the biggest surprises happen. Some agents run with broad permissions in development mode, then get restricted in production; the behavior difference can be subtle until a tool call fails. Others use user consent prompts that are easy to click through, which turns “permission” into a UI checkbox rather than a real control. If the agent can write to an external system, the permission model must cover what it can read, what it can write, and what it can delete. Without those boundaries, the agent can leak data through logs, send messages to the wrong recipient, or trigger actions that require human review.
Tools, Memory, And Permissions
Tool access in 2026 usually comes from a curated set of functions with explicit input and output types. A safe design keeps tools narrow: a “symptom checker” tool might only return structured triage categories, while a “send message” tool might require a verified destination and a confirmation step. Memory typically splits into short-term context (the current conversation window) and longer-term storage (preferences, extracted entities, or user-defined notes). The permission model should map to those memory layers: the agent should not use long-term memory to take actions unless the user has granted that specific permission.
For health-adjacent use, the most relevant question is not whether the agent can answer, but whether it can act. An agent that drafts questions for a clinician can be helpful when it cites the user’s stated symptoms and uncertainty. An agent that schedules appointments, changes prescriptions, or submits insurance claims crosses into regulated territory and needs stronger controls. Even when the agent is “only drafting,” it can still cause harm if it invents details or omits red flags. The safest implementations treat the agent as a drafting and retrieval assistant with explicit human review for anything that affects care decisions.
How To Evaluate An Agent Safely
Audit Tool Access First
Start by listing every tool the agent can call and what data each tool can access. Look for separation between “read-only” tools (like searching public references or checking a formulary) and “write” tools (like sending messages or updating records). A realistic outcome target is simple: you want the agent to fail safely when a tool is missing or permission is denied, instead of guessing. In practice, test with a benign prompt that requires a tool call, then confirm the tool call arguments match the expected schema. If the agent reports “tool error” but still produces a confident answer, treat that as a reliability gap.
Constrain Memory Scope
Ask what the system stores: raw chat logs, extracted facts, or summaries. If the product offers memory toggles, test them by changing a detail and checking whether the agent keeps the old value. A mild frustration is common here—some interfaces hide memory settings behind multiple menus, and users miss them. For health-adjacent tasks, prefer memory that stores non-clinical preferences (like preferred language or communication style) and avoid storing sensitive clinical history unless the user explicitly opts in. If the agent supports “forget” or data deletion, verify the behavior by running a follow-up prompt after deletion.
Use Permissioned Action Flows
For any action that affects people, require a human confirmation step that includes a preview of what will be sent or changed. The permission should be granular: “send a message to the patient portal” differs from “send an email to an address on file.” A realistic number to watch is the number of confirmations: if the agent asks for one broad consent that covers everything, the user’s ability to catch mistakes drops. In a controlled test, try a scenario where the destination is ambiguous; the agent should ask a clarifying question rather than guessing. If it proceeds anyway, the permission flow is not doing its job.
Measure Reliability With Small Tests
Run short, repeatable tests that check for hallucinations and missing context. Use prompts that require the agent to quote or reference the retrieved source, then verify the quote matches the source text. Track outcomes across 10–20 runs with the same input; variability reveals whether the agent is stable or drifting. I once compared two agent settings in a sandbox: one used retrieval with citations, the other used “answer from memory,” and the citation version stayed consistent while the memory version changed its wording and omitted key qualifiers. That kind of drift is exactly what you want to detect before trusting the agent with anything health-related.
Anonymized Case Examples
Example 1: Drafting a clinician message. A patient uses an agent to summarize symptoms and ask about next steps. The agent is allowed to read the user’s conversation context but not to access medical records. It drafts a message with timestamps, asks the user to confirm severity and duration, and then stops. The patient reviews the draft and sends it through the portal manually. The key safety feature is the agent’s refusal to “decide care” and its requirement for user confirmation before any message is sent.
Example 2: Medication list cross-check. An agent checks a medication list against a public formulary tool. The tool is read-only and returns structured results with dates. The agent flags mismatches and asks whether the user wants to contact a pharmacist. It does not store the medication list in long-term memory by default. The learning point is that tool outputs can be correct while the overall advice still fails if the agent treats the tool result as complete medical guidance.
Agent Checklist For 2026
| Decision Point | What To Look For | Risk If Missing | Test You Can Run |
|---|---|---|---|
| Tool boundaries | Clear list of tools and read vs write permissions | Agent acts beyond intended scope | Prompt that requires a blocked tool; confirm it refuses |
| Memory scope | Short-term vs long-term storage described | Stale or sensitive data reused | Change a detail; check whether it persists after update |
| Action confirmations | Preview + explicit user confirmation for writes | Wrong recipient or wrong content sent | Use an ambiguous destination; confirm it asks instead of guessing |
| Source grounding | Citations or retrieved evidence for factual claims | Hallucinated details look credible | Ask for a quote; verify it matches the referenced text |
Step-by-step checklist you can follow in 15 minutes: (1) Identify the agent’s tool list and confirm which tools are read-only. (2) Turn off long-term memory if the interface supports it, then test whether the agent still answers using only the current conversation. (3) Trigger one action that would require a write, then confirm the agent asks for confirmation with a preview. (4) Ask for one factual claim that should be sourced, then verify the claim is grounded in retrieved material rather than phrased as certainty.
Common Mistakes That Break Trust
One mistake is trusting the agent’s tone. A calm explanation can still hide missing context, especially when the agent compresses a long conversation into a short memory summary. Another mistake is entering sensitive health information without checking data handling terms. Many products separate “model input” from “tool input,” and logs can capture both; if you cannot find a clear retention policy, treat the system as a place where sensitive details may persist longer than you expect.
People also overestimate what memory features do. A “remembered preference” might be a single extracted attribute, not a full record of allergies, diagnoses, or medication changes. When the agent later references that preference, it can appear to know more than it does. A final mistake is skipping verification for tool outputs. If a formulary tool returns a match, the agent still needs to map that match to the user’s exact situation, which includes dose, timing, and contraindications that the tool may not cover.
FAQ
What Does “Memory” Mean In Agents?
Memory usually refers to stored context beyond the current chat window, such as extracted user preferences or summaries. Some systems store raw text, others store structured facts, and many store nothing by default. The safest approach is to check the product’s memory settings and test whether a changed detail persists.
How Do Agent Permissions Work?
Permissions define what the agent can read and write through connected tools, plus whether it can act without confirmation. A good permission model separates read-only retrieval from write actions like sending messages or updating records. You can test this by prompting the agent to perform a blocked action and checking that it refuses.
Can AI Agents Give Medical Advice Reliably?
Agents can draft questions, summarize user-provided information, and point to general references when grounded in retrieved sources. Reliability drops when the agent guesses missing clinical details or treats tool outputs as complete medical guidance. For care decisions, a clinician should review anything that affects diagnosis or treatment.
What Are Tool-Calling Failures?
Tool-calling failures include wrong tool arguments, schema mismatches, missing permissions, or outdated retrieval results. The agent may still produce an answer even when the tool call fails, which can look confident while being unsupported. Testing for grounded outputs helps catch this.
How Should I Test An Agent Before Trusting It?
Run small repeatable tests: ask for a sourced factual claim, verify citations match the referenced text, and confirm the agent refuses blocked actions. Then test memory by changing a detail and checking whether the agent updates its behavior. Keep sensitive data out until you confirm the retention and permission behavior.
Author's Insight
Agent behavior in 2026 depends less on the language model alone and more on the surrounding system: tool schemas, retrieval freshness, memory storage rules, and permission gates. Many failures come from mismatched assumptions, such as treating a summary as complete medical history or assuming a tool result covers contraindications it never checks. Evidence-based evaluation focuses on repeatable tests, grounded outputs, and explicit confirmation for any write action. If you cannot find clear documentation for memory retention and permission scope, treat the agent as a drafting assistant rather than a decision-maker.
Key Takeaways
- AI agents combine planning with tool calls; tool access and schema correctness shape outcomes more than the wording of the response.
- Memory often stores summaries or extracted facts with limited scope; test whether changes persist and whether sensitive details are retained.
- Permissions should separate read-only retrieval from write actions, with preview and explicit confirmation for anything that affects people.
- Reliability improves when the agent grounds factual claims in retrieved sources and refuses blocked tool actions instead of guessing.
