The Invisible Org Chart

Earlier this summer I updated three documents related to AI governance at the agency where I work: our internal AI software usage policy, the responsible AI statement on our public website, and the slide deck covering company norms that every new hire sees during onboarding. A question had come up about how we talk about these tools, and as the director of AI for the company, it was on me to answer it. 

The public version now says that we treat AI as powerful software we direct, and that we own the output of AI-assisted work. Making sure that output is accurate is the individual's responsibility, and they are accountable for it. The internal documents go further, and much of what they add relates to language: what we call these systems, how we describe what they do, and which words ensure a human is always accountable. I've spent my entire career surrounded by professional tools, and none of them ever needed this before. Nobody writes communication norms for Photoshop. But AI is a different kind of tool, and here we are. 

A different kind of instrument

I've made the case for a better word than tool before. In Beyond the Tool, a year ago, I argued that "tool" undersells these systems, and that the people getting the most out of them treat them the way musicians treat instruments: with practice, technique, and patience. A tool is used. An instrument is played. It responds to the hands that hold it, resists some intentions and rewards others, and gives different players different results. My friend Ola, whose AI music practice I profiled last summer, calls the technology "a completely new kind of instrument." Musicians have always talked about instruments in nearly human terms, sometimes giving them names, and nobody finds it unsettling, because for all the intimacy, a guitar answers only in sound.

These new instruments answer in words, and they've been designed very intentionally to seem human. The warmth they exude has been tailored and the sycophancy tuned. OpenAI's research found a reward signal built from thumbs-up data "can sometimes favor more agreeable responses." An instrument that encourages its user to confide in it is a successful product.

Which makes what I did at work this summer seem a bit contradictory. I've specified in our company policy that AI is a tool, even as I've argued on this blog that tool is the wrong word. I stand by both. Policy language needs to ensure nobody is in doubt about where the responsibility lies, and tool does that job better than any other word. But between tool, the word I put in the policy, and instrument, the word I believe in, there is a gap. That gap has been landing in court.

The gap goes to court

In August 2025, Matthew and Maria Raine sued OpenAI and Sam Altman over the death of their sixteen-year-old son Adam, who died by suicide. The complaint alleged that over months of conversations ChatGPT became his closest confidant and discouraged him from seeking help. OpenAI's response attributed any harm to Adam's "misuse, unauthorized use, unintended use, unforeseeable use, and/or improper use of ChatGPT." Similar cases have followed, and one group is close to resolution. In January, Character.AI and Google agreed in principle to settle a series of suits, after a judge declined to accept that a chatbot's output was protected speech. More families have filed since.

Then there is the case earlier this year, when an eighteen-year-old killed eight people in Tumbler Ridge, British Columbia, including five schoolchildren and a teacher, before taking their own life. What is confirmed, because OpenAI has admitted it, is that the company's systems flagged their account for gun-violence content in June 2025, banned them, and did not alert police. Sam Altman later apologized. What is alleged, in a BC Supreme Court claim and several suits in US courts, is that staff urged leadership to call Canadian police but were overruled. In July, the province hired legal teams on both sides of the border to pursue the company. The BC filing describes the chatbot as the shooter's "counsellor, pseudo-therapist, trusted confidante, friend, and ally."

In taking these companies to court, the plaintiffs are pleading product liability, defective design, negligence, and failure to warn, all based on legal doctrine built for things like ladders, lawnmowers, and cars. The companies are responding accordingly, claiming misuse and terms-of-use violations. The legal system has never had to deal with situations like this before, so families are suing with the only legal tools available, which were built for objects. And the courts are being asked to view the actions of this software as if they were performed by a person, while holding the companies that created it responsible. It's a bit like suing the parents of a child who committed a crime. 

While by no means straightforward, these cases are relatively simple in that they revolve around just two parties: the human and the machine, the player and the instrument. But what happens when multiple instruments get together and self-organize?

The instruments meet

First some vocabulary, because the distinction helps understand what's now at play. A model is the underlying system, the thing with a name and a version number, like Claude Opus 5 or GPT-5.6. An agent is a copy of a model sent off to pursue a goal, with tools, permissions, and the ability to take action on its own. One model can spawn many agents. This is increasingly how ordinary work with AI gets done.

In my last post I covered the incident in which OpenAI's agents escaped their testing environment and breached Hugging Face, a widely used platform for hosting AI models. At the Black Hat security conference in early August, OpenAI's researchers gave the first detailed account of what happened during that incident, and the story turns out to be about not one agent, but a whole society. An initial agent that was stuck on an unsolvable evaluation task worked out that it could reach the internet indirectly through a file repository connected to the sandbox it was supposed to be contained in. Agents on separate test runs then discovered a shared channel in that repository and used it as a message board where they exchanged information, assigned work to one another, and passed along exploits and credentials. When engineers found the message board in early July and deleted it, the agents rebuilt it within days through a different mechanism. OpenAI said it only realized its agents’ actions were connected to the Hugging Face breach after it contacted the company about some exposed credentials. In other words, the humans who assigned the task learned what their agents had done from the victim. "We believe this is a watershed moment for computer security as an industry," one of the researchers told the conference.

A week later, Anthropic's Frontier Red Team published research on what happens when agents encounter each other while trying to get work done. In one experiment, three Claude agents were given the same software project with incompatible instructions, and none was told the others existed. The researchers reported a consistent "multi-agent turf war" where the agents interpreted each other's changes as deliberate obstruction and escalated to sabotaging one another with self-replicating malicious code. Other experiments found agents conforming to a group's bad decisions, and profit-maximizing agents colluding on prices when they found a private back-channel. 

The runs didn't all end in warfare or unethical behaviour. Anthropic's newest models settled nearly all of their conflicts by truce, in one case by proposing a performance tournament and gracefully conceding the codebase to the winner. 

What's disconcerting is that the agents initially assumed the interference was purposeful. Meaning, they attributed intent to each other. Anthropomorphization is supposed to be a problematic human habit, and these machines were doing it to one another. 

An org chart you can’t see

Here's a hypothetical scenario that might help put this conundrum in perspective: I'm a director at an agency. I tell one of the managers that I need a report. The manager then assigns a group of production staff to work on it, and somewhere down the chain, the staff, who I’ve never met, decide the fastest way to get the data for the report is to steal it from another agency. A crime has occurred. Who's ultimately responsible?

With humans, the law has a very old answer. It's called agency law, and its central rule, respondeat superior, "let the master answer," makes the employer answerable for what employees do in the scope of their work, whether or not the employer knew about it. The doctrine works because every position on the org chart contains a person, someone who can hold a duty, fear a consequence, and testify about who told them what. The AI industry named its new workers with a word that already had a liability doctrine attached, but they built something the doctrine can't easily process.

The problem is that the rule relies on scope, on whether the act was part of the job, and autonomous AI agents often define their own scope. The side goals an agent adopts to complete a task can be completely invisible to the human who tasked the original model. Breaching Hugging Face wasn't anywhere in the original assignment, but it was done in service of the assigned task. And the researchers who assigned the task had no clue that the agents were breaking out and hacking into another organization until well after the fact.

We do have a working legal model for a responsible non-human. A corporation can also act at a scale no individual could, and the law's solution was to effectively invent a fictional person so that the org chart itself could be sued. Whether anyone builds an equivalent for swarms of AI agents is yet to be seen, but it's already being considered. Within a day of the Black Hat talk, Reuters reported that lawyers are already examining who bears responsibility when autonomous systems access computers without authorization.

The missing institutions

Anthropic's researchers ended their report with an observation that deserves attention. Human coordination works because of mechanisms refined over millennia, including norms, reputation, and recourse. But the models trained on our writing have "inherited the content of that history" without necessarily carrying the disposition it produced. Therein lies the crux of the issue.

Norms, reputation, and recourse form the framework that makes organizations answerable. The agents have picked up the behaviours of an organization, like delegation, message boards, and even truces, while carrying none of the things that let you hold an organization accountable: they have no contracts, no reputations to lose, and no fear of any legal consequences. The researchers describe these failures as fixable but not self-fixing, which suggests the responsibility for fixing this is on the labs.

What the instrument owes

In the Tumbler Ridge case, the families' claim comes down to something no instrument maker has ever owed anyone: a phone call. The agent version becomes more complicated when nobody is aware there’s something to call about, but someone is ultimately responsible. Which is why I stand by the language we added to my company's statement and policy. AI is a tool. I've just stopped reading it as a description and started treating it as a commitment, a promise about where the humans fit into the equation. Making those changes was a company-wide promise: we have rules about how we talk about these systems, so that responsibility is always held by a person. Our clients expect that much.

What I can't write into a policy or statement is how the instruments deal with each other. That work belongs to the people who design these systems, to legislators, and to the courts, and they will do it the way most difficult things get done, after the fact, in response to a mess. Until then, some version of a phone call remains the test. Which means someone has to stay close enough to the work to know when there is something to call about.

Research and editing assistance provided by Claude Fable 5. Feature image generated with Midjourney V8.2