# I Was Told My AI Risk Keynote Was Too Dark. Then Anthropic Published the Footnotes.
By 

## Article Content
A few weeks ago, I closed out the 2026 Governance, Risk and Control Conference in San Diego with a keynote built around an idea I have been developing for some time: the acceleration gap. The premise was simple. Risk is not just increasing. The speed at which risk emerges, evolves, spreads and compounds is accelerating faster than the governance systems most organizations have built to understand and manage it. The audience response was overwhelmingly positive. My speaker rating was 4.69 out of 5, more than 92 percent of respondents rated me positively, and more than 80 percent gave me the highest possible rating. The written comments included phrases such as “best session of conference,” “the highlight of the conference,” “monumental,” and “I could listen to you all day.” There were also a few outliers, as there always are. One comment suggested that I had painted too dark a picture of the future. Another suggested that I had ended the conference by telling the audience how wrong they were. I spent some time thinking about those observations because, as I have written before, I have a long history of internalizing the negative comments more than the positive ones. Was the keynote too dark? Had I leaned too heavily into the risk side of the story? Then, less than a month later, Anthropic published its September 2026 Threat Intelligence Report. I read it carefully. My reaction was not that I had overstated the risk. It was that I might have understated the speed. What I actually said on stage The keynote was never an argument that technology is bad, that AI is inherently dangerous, or that the future is bleak. That is not my message and never has been. My argument was that the nature of risk itself has changed. I described several forms of risk that are becoming increasingly important. Velocity risk, where threats emerge faster than organizations can understand or react to them. Autonomous risk, where systems begin to act, decide and transact without waiting for direct human involvement. Ecosystem risk, where one vendor, one API, one credential, one token or one software update can suddenly become everyone’s problem. Reality risk, involving deepfakes, synthetic identities, fabricated evidence and the growing challenge of determining what is authentic. Negligent acceleration, where organizations deploy powerful technologies faster than their governance structures can absorb them. Those themes were central to the keynote. The point was not that the future is terrifying. The point was that governance has a speed problem. Anthropic’s September report makes that speed problem difficult to dismiss. The report covers activity the company says it identified and disrupted between December 2025 and August 2026. It examines seven areas of misuse: cyber operations, influence operations, surveillance, scams and fraud, biological misuse, conventional weapons development, and illicit model distillation. The threat actors include suspected state-sponsored groups, financially motivated criminals, commercial surveillance vendors, propaganda organizations, and politically motivated individuals. Anthropic is also careful to point out that these are not examples of typical AI use. They are some of the most notable and novel cases its threat intelligence team has encountered. That distinction matters. It would be irresponsible to take extreme cases and suggest that every AI system or every AI user represents a comparable threat. But that is not what makes the report important. What matters is what these cases tell us about direction, capability, scale and speed. And on those dimensions, Anthropic’s findings line up rather uncomfortably well with the risks I was describing on stage. From assistant to orchestrator For the last several years, many organizations have thought about AI risk through a relatively simple mental model. A human asks a machine a question, the machine produces an answer, and governance focuses on the quality or safety of that answer. Is it accurate? Is it biased? Did it hallucinate? Did confidential information leak into the system? Was the employee using an approved model? Was the output reviewed? Those are still important questions. But the risk model is moving on. Anthropic describes this shift with a particularly useful phrase: “from assistant to orchestrator.” In the cyber cases it investigated, AI was not limited to answering questions for an attacker. Anthropic says that a majority of the operations described in the report involved AI being used for direct execution or orchestration. Multi-agent frameworks were used in reconnaissance, exploitation and data exfiltration, while human operators remained involved in selecting targets and reviewing results. That changes the governance question in a very significant way. The issue is no longer simply, “What did the AI say?” Increasingly, the question becomes, “What did the AI do?” A policy designed around chatbot output is not necessarily a policy capable of governing an autonomous agent. A control designed to review an answer is not necessarily a control capable of overseeing an action. An approval system built around a human-speed workflow will struggle when the software underneath it begins operating at machine speed. That is autonomous risk. It was one of the categories I put on the screen in August. Anthropic has now provided real-world examples of what it can look like. Sophisticated attacks no longer require sophisticated attackers One of the most striking section headings in Anthropic’s report is this: “Sophisticated attacks no longer require sophisticated attackers.” That is Anthropic’s language, not mine. The company argues that AI is collapsing the labor and tooling gap that once separated large, well-resourced cyber operations from smaller groups or even individual operators. In the cases it studied, campaigns that previously might have required multiple specialists could be sustained with fewer people and less specialized knowledge. Anthropic goes so far as to say that the apparent sophistication of an attack is becoming a less reliable indicator of who is behind it. Think about what that means for the economics of risk. For decades, expertise itself acted as a constraint. Sophisticated operations required sophisticated people. Advanced knowledge was scarce, specialized tooling was expensive, and coordination required time and skill. AI changes the economics of that model. It does not turn every amateur into an elite operator overnight, but it can give less capable actors access to more knowledge, more automation, more coordination and more persistence than they previously possessed. That expands the number of people capable of creating meaningful risk, reduces the cost of experimentation, accelerates capability development, and shortens the distance between intent and execution. That is exactly what I mean by velocity risk. The AI supply chain becomes part of the attack surface Another part of the Anthropic report deserves far more attention than it is likely to receive: illicit model distillation. Distillation itself is a legitimate AI training method. A larger and more capable model can be used to help train a smaller one. The problem Anthropic describes is something very different. It defines illicit distillation as covert, industrial-scale efforts to extract the capabilities of frontier models and reproduce them elsewhere without authorization. The report describes fraudulent accounts, proxy services, false identities, stolen payment credentials and stolen API keys being used to gain access to models. It also describes user exchanges being captured and reused as training material. This is ecosystem risk. The governance conversation can no longer stop at the model itself. It has to include API credentials, routing services, third-party wrappers, proxy access, identity controls, training-data lineage, model provenance, conversation retention and vendor relationships. A model can be secure while the ecosystem around it remains vulnerable. The risk might not be inside the AI at all. It might be in how people connect to it, how credentials are handled, how data moves through it, how responses are stored, or how third parties wrap and resell access. That is a much broader governance problem. Autonomous risk is no longer theoretical One of the slides in my keynote was deliberately blunt: AUTONOMOUS RISK. The point underneath it was that systems increasingly create consequences without waiting for humans. Agents act. Workflows decide. Machines transact. Governance must learn how to control systems that act rather than systems that merely respond. At the time, that might have sounded speculative to some people. The Anthropic report makes it much less theoretical. The company describes cases where AI systems were used to help conduct multiple stages of cyber operations. In one example, AI-assisted workflows monitored whether malicious tools were being detected and then participated in modifying and rebuilding those tools in an attempt to evade detection. Anthropic’s broader conclusion is that AI is enabling adversaries to move faster across a wider surface area with fewer resources. The human has not disappeared from these systems. That is important. What has changed is the level at which the human operates. The person increasingly specifies intent, objectives or targets while software performs more of the work underneath. The human becomes the strategist while the machine becomes the operator. That is the same structural shift we are beginning to see in legitimate enterprise AI. The same architecture that allows a finance team to automate analysis can automate parts of malicious activity. The same agent frameworks that coordinate business workflows can coordinate hostile ones. The technology itself is not inherently good or bad. Capability is capability. Governance has to understand what happens when that capability becomes cheaper, faster and easier to deploy. Influence, surveillance and fraud at machine scale The report becomes even more interesting when you move beyond cybersecurity. Anthropic documents influence operations involving actors from multiple regions and targeting audiences across six continents. In these cases, AI was used not only to generate content, but to help create the machinery behind influence campaigns, including personas, databases, operating procedures and persistent content-production systems. Its surveillance findings are equally striking. The report describes state-aligned actors and commercial vendors using AI to build surveillance tools, process large volumes of information and help identify or profile targets. And then there is fraud. Anthropic reports that a China-based app studio used Claude to help build a network of more than 20 dating apps and to power AI personas that users were led to believe were human. During a two-week period in April 2026, the company says it identified more than 4,700 distinct AI personas interacting with at least 25,000 people. That is reality risk. The key issue is not that reality disappears. It is that the economics of manufacturing synthetic reality are collapsing. A convincing fake identity once required effort. A fake persona required someone to operate it. A coordinated campaign required staff. A surveillance operation required analysts. A fraud operation required people to maintain thousands of interactions. When the marginal cost of those activities collapses, scale becomes the real story. The edge cases are moving toward the center The report also moves into areas that, only a few years ago, would have sounded extraordinarily speculative. Anthropic documents misuse associated with conventional weapons development, biological research and industrial-scale theft of model capabilities. Anthropic says it encountered cases involving the use of its models in activity related to conventional weapons, including software development, intelligence gathering and procurement support. It also presents five biological misuse case studies involving activity it believed could potentially support biological weapons development. Then there is model distillation. Anthropic says it identified and disrupted additional distillation attacks involving seven labs based in China. The largest campaign described in the report peaked at nearly three million exchanges per day across more than 3,500 fraudulent accounts. Again, the point is not that these are everyday uses of AI. They are not. The point is that issues we might once have placed at the far edge of the risk map are beginning to appear in actual threat-intelligence reporting. Risk categories migrate. Yesterday’s strange edge case becomes tomorrow’s emerging risk, and tomorrow’s emerging risk eventually becomes somebody’s audit plan. Did I paint too dark a picture? So I come back to the original criticism. Was the keynote too negative? I do not think so. It wasn't bold enough! But I also do not think the person who wrote that comment was wrong to feel unsettled by it. The material is unsettling. A world in which risk moves faster, software acts more autonomously, synthetic identities multiply, influence operations scale more easily, surveillance becomes cheaper and expertise becomes easier to replicate is not something we should trivialize. The mistake would be confusing uncomfortable information with pessimism. My keynote argued that the risk landscape was moving beyond familiar, human-speed risk toward velocity risk, autonomous risk, ecosystem risk and reality risk. Anthropic’s report does not prove every prediction I made, nor does it mean every organization will face every threat described in the document. What it does is reinforce the direction of travel. Risk is moving faster. Automation is increasing. Skill barriers are falling. AI-enabled misuse is spreading into more domains, and the amount of time available for governance to understand a threat before that threat becomes operational is shrinking. That was my argument in August. The September evidence makes the curve look steeper. The answer is not fear. It is faster governance. This is the most important part of the story because I do not want anyone to read either my keynote or the Anthropic report and conclude that the answer is to stop using AI. That would be absurd. The opportunity is too significant, the capability is too important, and the economic potential is too large. History tells us that powerful technologies do not disappear because they introduce risk. We learn how to govern them. The difference this time is that governance itself has to accelerate. That is why I ended the keynote not with fear, but with a challenge. Risk is accelerating. Knowledge is accelerating. Your profession is accelerating. I argued that the sustainable response is to become structurally better at four things: Explore what is not yet visible on today’s dashboard. Learn continuously, because expertise now expires faster. Experiment before a crisis forces the experiment upon you. Anticipate weak signals before they become obvious risks. That is not pessimism. It is preparation. And if Anthropic’s latest threat report tells us anything, it is that preparation has just become more urgent. Maybe the keynote was not too dark after all. Maybe the future simply moved faster than the evaluation form.

---
Source: https://jimcarroll.com/2026/09/i-was-told-my-ai-risk-keynote-was-too-dark-then-anthropic-published-the-footnotes/