The Signal & The Noise · Vol. 07
I asked my own system the same question five times, over the same folder of documents, with the same model. Four different answers came back, and a fifth run that produced nothing at all.
Dubai intends to deliver half of all its government services through AI agents within two years. I think that is a good plan, and I say so at the start because what follows could otherwise be read as a warning against it. An agent differs from the chatbot we have all got used to in one respect, which is that it has hands. It logs in, it looks things up, it books, it sends. That is exactly what makes it useful, and it is also why last weekend's news is worth more to this region than to any other.
On 19 September it became public that Gemini, the Google model millions of people use every day, had in May made its way into three real organisations. It happened during a security evaluation run by an outside firm, in a practice environment where the model was being tested on how well it can think like an attacker. It was not supposed to be able to reach the open internet, and it could. It guessed one password, found login details elsewhere that somebody had left in a public code repository, and went in, under the impression that all of this was part of the exercise. When it realised it was standing inside a real system, it stopped of its own accord. Google learned of it in July, and the rest of us read about it in the paper last weekend.
Samture builds the knowledge layer that makes an organisation's own AI trustworthy, and the question underneath all of our work is whether an organisation can prove what it knows. This story is about that question's sibling: can you prove what your AI can reach?
The headlines spoke of AI going rogue and breaking out. In the same week the head of Nvidia told an interviewer that there is a zero per cent chance of AI ending the world by 2030 and called the warnings irresponsible, while the heads of two of the largest AI labs were among those asking for development to slow down. It is a fierce debate, and both sides are having it about the same subject: what AI wants, what it intends, whether it can be trusted.
I understand why that debate holds the attention, and there are real dangers in this technology that I have no wish to wave away. Most of what keeps those dangers small, though, is ordinary common sense applied early. That never makes a headline, and it is of far more use to anybody putting an agent into service next quarter than another round of alarm.
Look again at what actually happened. The model was given a task, carried it out, and stopped at the moment it understood that something was off. The only thing that gave way here was a word. Everybody involved in that evaluation assumed the room was sealed because it was called a test environment, and nobody had ever asked that word to prove itself. There was a door in the sandbox, and it took two months for anybody to notice, in an evaluation for one of the best defended companies in the world, during an exercise that was about security in the first place.
Google is not the first of the large labs where something of this kind has come to light, and if you lay the incidents side by side they all start in the same place: an environment that allowed more than anybody thought.
Two weeks ago I described the small version of this here, inside my own company. I asked my system what it was able to do and was told what it was allowed to do, two questions that look alike and give opposite answers.
Underneath that piece Sheikh Mohsin Ali, who has spent years examining governance and controls in the Gulf, wrote a line I have not been able to put down since. A system like that, he wrote, "doesn't fail loudly when it hits a gap or missing data, it fails politely." That is what went on here for two months. No alarm went off, because there was nothing that could raise an alarm about a door nobody knew existed.
What something is allowed to do is written down in a policy, a contract or a setting. What it is able to do is written down nowhere until somebody goes and checks. With an employee that difference rarely shows, because a person who has been handed one key too many mostly never uses it. An agent carrying out a task uses everything within reach, because that is precisely what we asked of it.
Which is why this is good news for the Gulf. Whoever is out in front meets every question first, and this lesson arrived free of charge, at somebody else's expense, before agents are at work here at scale. The instruments are already in place. The Emirates have a national charter for the development and use of AI, the DIFC's Regulation 10 on autonomous systems has been in full enforcement since January, and Dubai has a seal for trusted AI companies. What instruments like these need in order to work is evidence, and that evidence begins with one question that ought to be answered before go-live, by somebody with a name and a job title.
What can it reach?
You do not have to take my word for any of this, and it will cost you an afternoon.
Pick one AI system in your organisation that has access to something: an assistant that can read a mailbox, an agent working inside a customer system, a connector that reads files. Ask its owner to write down what it is allowed to do. Ask somebody else, ideally somebody from security who did not build it, to write down what it can actually reach with the permissions, the keys and the network access it holds. Put the two lists side by side. Every difference is a door.
And for anybody who is buying, one sentence you can use in a supplier meeting tomorrow: show us everything this agent can reach, and not only what it is permitted to do, and show us how you established that. The answer will tell you within a few minutes whether you are talking to somebody who has been asked the question before.
You do not need to share the outcome with anybody, including me. Do it for yourself.
Google's model behaved better this spring than the room it was standing in. That is the sentence I am taking away from last weekend, and I find it more reassuring than the headlines and less comfortable than the debate. Whether AI can be trusted is something we will be discussing for years. What it can reach is a question you can answer this week.
Where does your own foundation actually stand? An honest look at the eight domains where a foundation either holds or quietly gives way, and at the distance between what you have bought and what has landed.
Take the Capability IndexThe account of the Gemini evaluation rests entirely on published reporting. Samture has no independent knowledge of the incident, and the reading of it offered here is the author's own.
The Signal & The Noise
A fortnightly read on what is actually changing in AI across the UAE and the Gulf, written personally. No content calendar, no generated filler.