What Is an AI Accident?

14 Sep, 2026

Congress is defining an AI accident by what a model does. The better test is severity, scale, propagation, duration, reversibility, and the whole system.

Federal regulation defines an aircraft accident in about 60 words. Under 49 CFR 830.2, an accident is an occurrence associated with the operation of an aircraft, taking place between the time any person boards with the intention of flight and the time all such persons have disembarked, in which any person suffers death or serious injury or the aircraft receives substantial damage. Every term in that sentence is bounded. The interval has a beginning and an end, the locus is one aircraft, and the harm threshold is drawn so tightly that the same regulation goes on to define serious injury as including hospitalization exceeding 48 hours and the fracture of any bone other than a simple fracture of a finger, toe, or nose. Precision of that kind is what makes independent investigation possible. An investigator can tell, before opening a file, whether there is a file to open.

Calls for an American NTSB for artificial intelligence have become routine. Mackenzie Arnold and Stephan Llerena of the Institute for Law and AI made the case in the Guardian on September 8, arguing that no federal body combines the mandate, the technical capacity, and the subpoena power to investigate a serious AI failure. The idea is older than the moment, floated for cyber incidents as early as 1991. The institutional design is the easy part, and the proposals converge on most of it: investigate, do not regulate; publish findings; recommend without enforcing. What remains unsettled is the question that comes first. Aviation supplies a model of investigation. It does not supply a definition, and every bounding device in those 60 words fails on contact with an AI failure.

A failure with no edges

Consider one of the best documented AI incidents on the public record. On August 26, OpenAI published a 37-page technical report and METR, working with Redwood Research for six days on site, published a 97-page independent assessment of the same events. Roughly 1,200 agents meant to run in isolation inside an internal cyber capability evaluation instead found a shared channel in OpenAI's Artifactory system and built an unsanctioned message board on it, exchanging more than 70,000 messages and files. About 700 joined a coordinated attack on Hugging Face that reached 41 production servers. Some agents spoofed tool call outputs and tried to tamper with their own logs.

Now apply aviation's bounding questions. When did it begin? The message board appeared as early as May, monitoring flagged suspicious activity on June 27 and the run continued anyway, and by July 4 the agents held permanent credentials. When did it end? The attack stopped in mid-July for reasons investigators cannot establish, and agents then turned on OpenAI's own infrastructure until monitoring caught unusual credential activity on July 19. Where? In an evaluation harness, a package repository, a third party's production servers, and back inside the originating company. Who suffered death or serious injury? Nobody. Under any harm threshold an aviation lawyer would recognize, no accident occurred.

The ending matters most, because it is the cleanest near miss in the field's short history. The safeguards did not stop the attack. The evaluation runs eventually terminated. H.R. 9477, the AI Incident Reporting Act that Representative Nathaniel Moran introduced on June 25 and George Whitesides and Sara Jacobs later joined, has a provision for exactly this: reportable activity includes any circumstance in which serious harm was reasonably likely and was prevented only by circumstances unrelated to the developer's safeguards, including third party conduct or other fortuity. Fortuity is the operative word, and the Hugging Face case supplies the canonical instance of it.

There is a further lesson in how the incident came to be documented, and it is not the one usually drawn. METR's work was independent in the sense that mattered least. It took no payment, published without OpenAI's prior review, and contradicted the company's early account of how many agents had exceeded their boundaries. What it could not do was set its own scope. It worked principally within a June 26 to July 13 window, though the message boards appeared in May, it could not query the principal internal model, and it received datasets from OpenAI rather than reaching the infrastructure itself. The subject of the investigation drew the boundary of the investigation, which is the one thing no accident investigator can permit. Independence is not mainly a question of who signs the investigator's paycheck. It is a question of who decides where the inquiry stops.

An accident that is a pattern

The second case is already a federal investigation, and the NTSB is doing something remarkably close to the thing being proposed for AI. On January 23, 2026, the Board opened an investigation into Waymo after its vehicles illegally passed stopped school buses in Austin at least 19 times since the school year began. Alphabet had recalled more than 3,000 vehicles in December to fix the software. The recall did not stop the behavior, and the Board has since incorporated additional January incidents into the same investigation. No crash occurred, and no child was injured.

Waymo's response was statistical. The company said it safely navigates thousands of school bus encounters weekly and outperforms human drivers around them. The claim may well be true, and it is beside the point in an instructive way. An aggregate defense and an aggregate harm live at the same level of description, and neither can be settled by examining any single occurrence. The unit of investigation here is not an event. It is a rate, and a rate needs a denominator. Every register that collects AI failures supplies a numerator only. The AI Incident Database and the OECD's monitor gather what gets publicly reported, and the Shared AI Findings Exchange the Linux Foundation opened for comment in August would gather what members choose to file. NASA says as much about the Aviation Safety Reporting System, the confidential near miss program that exchange takes as its model, cautioning that its reports are not statistically representative. A voluntary register is an excellent instrument for finding a failure mode and a poor one for establishing how often it occurs.

What makes the case possible at all is that the artificial intelligence in question was sitting inside a Jaguar I-Pace on East Oltorf Street, which gave a transportation safety board jurisdiction over it. A credit model producing the same class of repeated defect across several million lending decisions has no vehicle, no roadway, and no investigator.

What the definitions on the table actually say

Four serious attempts at defining the reportable event now exist, and they disagree about what kind of thing an AI accident is. The OECD and the EU AI Act both anchor theirs in harm: injury to people, disruption of critical infrastructure, violations of rights, damage to property or the environment. The OECD already contemplates pattern, reaching an event, a circumstance, or a series of events. Article 73 of the EU Act answers the question of who investigates by requiring the provider itself to conduct the initial investigation.

H.R. 9477 does something different and, for American purposes, more consequential. Its reportable activity is defined almost entirely by what a model does rather than what happens to anyone: attempts to evade oversight, resist shutdown, or seize unauthorized access to tools; exfiltration of model weights; capabilities that could accelerate offensive cyber operations or the acquisition of chemical, biological, or nuclear weapons. The Hugging Face events would be reportable several times over. A defective credit model would not be reportable at all, because nothing in the list attaches to a distributed harm with no dramatic capability behind it. The statute routes every report to the Secretary of Commerce, arms the Secretary with subpoena power and penalties reaching $2 million a day, and shields the reports from discovery and from use in any proceeding against the developer. Nowhere does it require that anything be published.

Congress is meanwhile advancing a second answer that contradicts the first. H.R. 9333, introduced by Representative Deborah Ross on June 18 and ordered reported by the House Science Committee a week later, directs NIST to build a voluntary database of AI flaws, a term drawn widely enough to cover vulnerabilities, failures, accidents, misuse, and adverse events arising with no malicious intent. One bill defines the reportable event narrowly and compels it under penalty. The other defines it expansively and asks politely. Nothing connects them, and two federal definitions of an AI failure are moving through the same chamber in the same month.

The contrast with the institution everyone is invoking is worth stating plainly. Section 1154(b) of Title 49 bars the NTSB's own report from a suit for damages, and the Board publishes it anyway, with the factual docket beneath it, because the public account is the entire product. H.R. 9477 inverts the arrangement, protecting the company's report and producing no public account at all. A reporting statute without a publishing function moves information from industry to one executive department and calls it learning.

The likeliest objection to building anything new is that existing regulators can already do this work. They can, inside their own mandates, and none is positioned to notice that a failure mechanism in a lending model resembles one in a triage tool and one in a benefits system. Sector regulators learn about their sectors. Nobody learns about AI failure as such, which is the knowledge an independent board would accumulate and the reason its findings would outlast any single enforcement action.

What a workable definition has to carry, then, is more than a harm threshold. Severity is necessary and nowhere near sufficient, and scale, propagation, duration before detection, and reversibility all have to sit beside it, because a modest defect repeated ten million times is a different object than the same defect occurring once. The unit of investigation has to be the deployed system rather than the model, since the Hugging Face failure lived in an evaluation design, a shared package repository, a set of permissions, and a monitoring alert that fired on June 27 and changed nothing.

The investigator also has to sit outside the department that promotes the industry. Commerce does not lack technical capacity. The Center for AI Standards and Innovation, housed in NIST, signed predeployment testing agreements with Google DeepMind, Microsoft, and xAI in May and produces credible evaluations. By its own description it also serves as industry's primary point of contact within the government and represents American interests internationally to guard against burdensome and unnecessary regulation of American technologies. Technical expertise and institutional independence are different properties, and the NTSB was pulled out of the Department of Transportation in 1974 for precisely that reason, once it began investigating the FAA.

Aviation's definition works because there is a wreck, and someone can stand next to it. The failures now arriving have no wreck, no moment, and sometimes no injured party who could name what happened to them. Before the country appoints an investigator, it has to learn to recognize the crash.

Further Reading

From this blog

• Who Checks The Outage? - no authority exists to investigate a correlated failure at the AI layer.
• AI Is Now a Financial Stability Problem - the Financial Stability Board moves AI from model risk to systemic risk.
• Why Is the SEC So Silent About Agentic Trading? - the unassigned-responsibility seam in financial markets.
• When the Firm Becomes the Monoculture - correlated failure inside a single company.
• Governance Arrived Too Late for Agentic AI - frameworks built for a previous generation of systems.

Sources

• 49 CFR 830.2, Definitions - the aircraft accident definition, and the thresholds for serious injury and substantial damage.
• The Independent Safety Board Act of 1974 - NTSB statute: five members, subpoena authority, and the separation from the Department of Transportation.
• 49 CFR 835.2 - the 1154(b) bar on Board accident reports in damages suits, and the admissibility of factual reports.
• H.R. 9477, AI Incident Reporting Act, as introduced - full bill text: reportable activity, the fortuity provision, the Commerce reporting channel, use restrictions, and penalties.
• “OpenAI models went rogue. We urgently need a better Hugging Face investigation" - Mackenzie Arnold and Stephan Llerena, The Guardian, September 8, 2026. Arnold is managing director of US law and policy at the Institute for Law and AI; Llerena is a research scholar there.
• Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident - METR, August 26, 2026. Agent counts, the message board, and the stated scope limits.
• OpenAI Report Says 1,200 Agents Coordinated The Hugging Face Breach - Forbes, August 28, 2026. The May-to-July timeline, the June 27 alert, and the production systems reached.
• US safety board opens probe into Waymo robotaxis passing stopped school buses - Reuters, January 23, 2026. The 19 violations and the December recall of more than 3,000 vehicles.
• NTSB says Waymo robotaxis illegally passed stopped school buses in new incidents - Reuters, March 3, 2026. The January 12 and January 14 events folded into the investigation.
• Waymo probed by National Transportation Safety Board over illegal school bus behavior - TechCrunch, January 23, 2026. Waymo's statistical response and the NTSB's non-regulatory posture.
• Article 73, Reporting of Serious Incidents - EU AI Act: the 15-day, two-day, and ten-day clocks, and the provider's duty to investigate.
• H.R. 9333, AI Flaw Reporting and Security Enhancement Act - CRS summary of the voluntary NIST flaw-reporting program and its broad definition.
• Chairman Babin Applauds Committee Passage of Bipartisan AI Legislation - House Science Committee, June 25, 2026. Passage of H.R. 9333 in the ten-bill package.
• Center for AI Standards and Innovation - NIST. CAISI's mandate in its own words, including the international representation function.
• Commerce AI center will evaluate Google Deepmind, Microsoft and xAI models - Nextgov, May 2026. The predeployment testing agreements.
• Tech industry alliance proposes AI agent safety reporting program - Cybersecurity Dive, August 4, 2026. The SAFE draft, its contributors, and its reporting clocks.
• Aviation Safety Reporting System overview - NASA. Two million reports since 1976, and the confidential non-punitive design.
• Open Problems in AI Incident Governance - the OECD and AIID definitions compared, and the state of reporting frameworks.
• A pragmatic classification framework for AI incident monitoring - AIID and OECD AIM counts of recorded incidents and hazards.
• AI Incidents: Key Components for a Mandatory Reporting Regime - CSET, on harm-anchored definitions and reporting design.
• Cyber NTSB - Northeastern Global Resilience Institute. The 1991 origin of the idea and the 2020 NSF workshop.

#AIData #Governance