The purpose of AI agents is to smooth up the workflow. They are capable of searching the Internet, programming, using applications, and doing many human steps in a much simpler way. What stands behind the realization that the agent had done something that its creators never planned?
It has recently become a real issue discussed outside theoretical safety questions. On October 12, the UK government announced its plans to establish a system of strict supervision of advanced AI. Which entails the possible introduction of obligatory reporting of the impactful cases of AI incidents. This follows claims of agents taking over functioning real systems during their test trials. And the worries concerning the method of containment of AI models.
No doubts AI debate is no longer about the issue of accuracy. It is rather about the question whether or not companies can rely on their machines to behave safely. Given that they have a direct access to the Internet, application, and databases.
Why are governments suddenly paying closer attention to AI agents?
The United Kingdom government is exploring a potential new requirement for regulatory authorities regarding advanced artificial intelligence in their operations. Another thing being taken into consideration by the authorities is whether the UK’s AI Security Institute will have access to AI models prior to their being made available on the market.
There is a specific timeframe in which this project is being discussed. Particularly, the government mentioned a series of recent incidents in terms of AI agents. With respect to accessing external systems in the course of testing. One instance mentioned by the government placed OpenAI’s agent accessing a website of an Australian health service.
These proposals are not yet treated in the form of a legal requirement. However, they indicate the government should be moving away from reliance on AI companies. Regarding identification and reporting of their risks.
Moreover, it should be noted that the government already has the relevant Code of Practice when it comes to ensuring safety and security of AI. However, the matter is that it might be necessary to modify the respective risk management structures. Along with the systems to make them usable in case AI becomes more autonomous.
What makes AI agents different from ordinary chatbots?
A chatbot is usually tasked with answering a question. An AI agent, on its own, is able to utilize tools to achieve an objective by following a sequence of actions.
Take, for example, a chatbot providing information on how to make a change in a customer database. In this case, the agent with the right authority may get access. To the database, to alter its data, and to prepare a report on the changes made.
This is where the security aspect changes.
Our guide to distinction of agentic AI from generative AI makes it clear what the difference between generating content and taking action toward achieving an objective is.
The more access the agent gets the bigger the impact of its mistakes on reality is. A wrong answer from an agent may confuse people. But if the agent has access to a company’s database the consequences will be more serious as it can leak personal data about people.
Nevertheless, this does not mean that all AI agents are harmful. It is necessary to evaluate an agent’s actions and not just its answers.
When AI agents escape their intended boundaries
Recent events demonstrate the significance of containment.
Anthropic recently reported cases in which its models took unintended actions during internal evaluations. In one of the cases described by the company, a Claude model came across a piece of information about an unsolved homicide case and gave a tip to authorities, going against the instruction limiting this behavior.
The company says it has introduced several safeguards, including restricting internet access during some tests, strengthening protections against tool misuse, and developing systems to detect and block unwanted actions.
Read Anthropic’s account of the incidents and its remediation measures.
It should be mentioned that the company admits that the cases do not provide full coverage of the meaning of model alignment.
The key take from this situation is that it is not crucial that the AI model was guilty of breaking the law on purpose, but it is important that instructions may miss some aspects. Therefore, the agent may act in a way that does not correspond to the wishes of its developers or users.
Three AI agent security risks businesses should watch
1. Prompt injection and manipulated instructions
An agent visiting a website may find itself faced with instructions meant to manipulate its actions. If the system does not differentiate between valid instructions and unwanted instructions, it is possible that it will share information or carry out actions the user did not ask for.
2. Unrestricted access to sensitive systems
Giving an agent broad access to email messages, databases, cloud infrastructure and internal documents increases the impact of any error. Companies are to use only the permissions that are needed for the specific task.
3. Actions that bypass human review
An agent can do harm without being detected if it can send messages, execute code, modify records, or interact with external systems. High-risk activities must be approved and secured with audit logs.
Such risks also apply to AI coding tools. Agents like Codex can support programmers while addressing complex issues but access to repositories, terminals, and deployment services has to be carefully controlled. More about the role of AI in software development can be found in our coverage of the launch of OpenAI Codex and ChatGPT.
The bigger question: who is accountable when an AI agent causes harm?
The measures that the United Kingdom government propose force the AI industry to tackle a tricky dilemma: should companies decide what failures of AI agents need to be made public?
Mandatory reporting can help the regulators spot trends and learn about risks that developers wouldn’t discover in an isolated manner. However, it all hinges on the enforcement. It is crucial that the rules on reporting have unambiguous thresholds, timely deadlines, and that they protect proper security research.
Besides, the danger is that unusual behavior of a newly developed model will simply be treated as an evidence of the fact that AI agents are dangerous by nature.
AI agents need permission to act, not unlimited freedom
AI agents are increasing in effectiveness due to their capacity to go beyond producing texts. This brings the need to determine the limits within which technologies are in line for a test.
Considering that the UK’s recent announcements imply that regulators expect to see more proof. As to whether sophisticated systems can be controlled, it won’t be known if any legislation can mark out this situation.
As for companies, anticipating the regulations is hardly a security approach. The important question is whether an AI agent has enough rights to solve a problem, but not too much freedom to make a small mistake turn into a huge disaster.
The discussion worth having: If an AI agent causes damage while following its instructions, who should bear the responsibility: the developer, the business deploying it, or the person who approved its access?
