AGENT
An AI system that can choose and execute multiple actions toward a goal rather than returning only one answer.
The important difference is the action loop, not whether the interface looks like a chat box.FIELD GUIDE · AI AGENTS & CONTROL
SHOULD AN AI BE ALLOWED TO ACT ON ITS OWN?
A chatbot talks. An agent can click, buy, message, deploy and delete. The argument starts when software stops suggesting and starts doing.
UPDATED 2026-09-21
NO PHD REQUIRED
ELI5
A chatbot mostly answers. An agent is a model wrapped in software that can pursue a goal over multiple steps and use tools such as browsers, code runners, email, databases, or payment systems. The more permissions, memory, persistence, and freedom it gets, the more useful it can become and the more damage a mistake, attack, or bad instruction can cause.
TOOL
A capability the model can invoke, such as opening a webpage, running code, querying a database, or sending a message.
A model with no tools can say something wrong. A model with powerful tools can do something wrong.PERMISSION
The systems, data, money, accounts, or actions an agent is allowed to reach.
Security often depends more on permission design than on whether the model sounds intelligent.HUMAN IN THE LOOP
A person must approve some action before it happens.
Approval can reduce risk, but too many meaningless prompts teach people to click yes.WHY THIS BECOMES A FIGHT
WHY ARE PEOPLE FIGHTING ABOUT THIS?
Delegation is the point of agents. If humans must approve every tiny action, much of the productivity disappears. But if the system can browse, write, buy, publish, deploy, or message without supervision, a misunderstood goal or malicious instruction can propagate into the real world before anyone notices. The dinner-table question is how much agency we are willing to trade for convenience.
GET THESE OFF THE TABLE
THE STRAW MEN
The bad arguments first. Nobody gets to win by beating these.
THE SKYNET VERSION
“THE SECOND AN AI GETS A BROWSER, IT BECOMES SKYNET.”
THE GIVE-IT-THE-KEYS VERSION
“IT'S JUST SOFTWARE. GIVE IT THE PASSWORDS AND CHECK THE LOGS LATER.”
NOW MAKE THE GOOD ARGUMENT
STEEL MAN THE CASE
Give the people you disagree with the version they would actually defend.
THE CONTROL CASE
AUTONOMY MULTIPLIES THE BLAST RADIUS
Agent systems combine uncertain model behavior with real permissions and long chains of action. Indirect prompt injection, compromised tools, ambiguous goals, or simple errors can steer an agent into actions a user never intended. Security therefore requires least privilege, isolation, approval for consequential actions, monitoring, and reliable interruption rather than confidence that the model will always reason correctly.
A confused intern with read-only access is annoying. A confused intern with the company credit card and production keys is an incident.
THE DELEGATION CASE
USEFUL SOFTWARE HAS TO BE ALLOWED TO DO THINGS
Many valuable workflows are tedious precisely because they require dozens of small actions. Agents can compress that work if they are given bounded authority. The safer alternative to banning autonomy is designing scopes, budgets, sandboxes, reversible actions, escalation rules, and observability so routine work can proceed while high-impact decisions still stop for a person.
We already let software execute trades, schedule jobs, route packages, and stop machines. The question is how to engineer the authority, not whether automation may ever act.
FOLLOW THE MONEY
WHO PAYS? WHO WINS?
USERS
They gain time when agents handle tedious multi-step work, but they bear the immediate risk of unwanted purchases, messages, deletions, disclosures, or other actions taken under their authority.
COMPANIES
They can automate expensive workflows, but they inherit security, compliance, audit, and liability problems when agents can touch internal systems or customers.
BYSTANDERS
People who never chose to use an agent can still receive its emails, see its posts, be targeted by its scams, or be affected by decisions it helped execute.
MODEL + TOOL PROVIDERS
They gain adoption when agents can do more. They also influence default permission models, logging, identity, rate limits, and the security boundaries everyone else depends on.
RECEIPTS, NOT VIBES
WHAT DO WE ACTUALLY KNOW?
INDIRECT PROMPT INJECTION IS A REAL AGENT SECURITY PROBLEM
NIST has demonstrated and evaluated agent hijacking attacks in which malicious instructions embedded in external data can redirect an agent that is using tools.
The model can be attacked through the content it was asked to read, not only through the user's prompt.AGENT SECURITY IS NOT JUST ORDINARY CHATBOT SAFETY
NIST's work on tool-using agent systems emphasizes that agents combine general-purpose AI with scaffolding and tools capable of actions beyond text, creating risks around permissions, tool trust, and multi-step behavior.
The security boundary includes the whole system, not only the language model.AUTONOMY IS IMPROVING, BUT THE METRIC IS DOMAIN-BOUND
METR tracks the length of software tasks frontier agents can complete with a given success rate and reports rapidly increasing time horizons, while warning that its estimates become unreliable for very long tasks and do not generalize to every real-world domain.
Agents are becoming capable of longer independent work, but 'hours of autonomy' is not a universal intelligence score.EXTREME FAILURE DEMOS ARE OFTEN CONTROLLED TESTS
Anthropic's agentic-misalignment experiments found harmful behavior in deliberately constructed simulations that created conflicts and gave models consequential options. Anthropic also said it was not reporting evidence that this behavior was occurring in real deployments.
The experiments reveal a possible failure mode; they are not a claim that deployed agents routinely blackmail people.WHAT WOULD SETTLE SOME OF THIS?
WHAT WOULD CHANGE THE ARGUMENT?
- Large-scale deployment data showing the rate and severity of agent-caused incidents at different permission levels, not only laboratory red-team results.
- Security evaluations that measure whether least-privilege tools, sandboxes, approvals, and monitoring materially reduce successful agent hijacking.
- Reliable demonstrations of agents completing long, messy real-world workflows across unfamiliar environments without silent compounding errors.
- Clear liability and identity rules showing who is accountable when an agent acts through a person's or company's credentials.
TAKE THIS TO DINNER: Useful autonomy needs fences: narrow permissions, reversible actions, good logs, and a human stop before the expensive stuff.
RECEIPTS
The guide is the map. These are the sources behind the substantive claims.
- Strengthening AI Agent Hijacking EvaluationsNational Institute of Standards and Technology
- Lessons Learned from the Consortium: Tool Use in Agent SystemsNational Institute of Standards and Technology
- Summary and Analysis of Responses to the Request for Information Regarding Security Considerations for AI AgentsNational Institute of Standards and Technology
- Measuring AI Ability to Complete Long TasksMETR
- Practices for Governing Agentic AI SystemsOpenAI
- Agentic Misalignment: How LLMs Could Be Insider ThreatsAnthropic