Causality in AI Incidents: Proving What Happened
Investigating an incident involving an artificial intelligence system is an exercise in applied epistemology under regulatory pressure. Teams must determine not only what occurred, but why it occurred in a way that satisfies technical standards and legal expectations of proof. The challenge is that AI systems are probabilistic, path-dependent, and often embedded in complex socio-technical environments. A model’s output may be the immediate trigger of harm, but the root cause could be a data pipeline change three months earlier, an ambiguous product requirement, or an over-ride by a human operator. In the European regulatory context, establishing causality is not merely a retrospective academic exercise; it is a prerequisite for meeting incident reporting duties, demonstrating risk management, and allocating liability. This article explains how technical and legal causality are approached in practice, how event reconstruction is performed, and what documentation creates defensible narratives under the EU’s AI Act and related frameworks.
From Probabilities to Proof: The Dual Nature of Causality in AI
When an AI system causes or contributes to an adverse outcome, two distinct but related forms of causality must be addressed: technical causality and legal causality. Technical causality is concerned with the mechanisms and conditions that produced the observed output or behavior. It asks: which inputs, model states, code paths, and environmental factors led to the result? Legal causality, by contrast, is about attributing responsibility and establishing whether the conduct or system design was a legally relevant cause of the harm. In practice, these domains interact: a technical finding (e.g., a data drift that degraded performance) may support a legal conclusion (e.g., failure to maintain adequate data governance).
European regulators do not demand metaphysical certainty. They expect a reasonable, evidence-based account that aligns with the state of the art and the risk profile of the system. The AI Act’s obligations for high-risk AI systems, for example, presuppose that providers can trace and explain system behavior throughout the lifecycle. Likewise, the GDPR’s right to an explanation (Article 22) and the accountability principle (Article 5) require meaningful information about the logic involved when automated decisions produce legal or similarly significant effects. In a post-incident context, these obligations translate into a need for reconstructable evidence trails.
Technical Causality: Mechanism, Path, and State
Technical causality in AI is multi-layered. It encompasses the data layer, the model layer, the software engineering layer, and the operational environment. A useful mental model is to think of causality as a chain of dependencies: data provenance → feature engineering → model training and validation → deployment configuration → runtime inputs → system outputs → human and system actions. A break or weakness in any link can be the proximate cause of an incident.
Consider a medical imaging model that misclassifies a tumor. A technical investigation might discover that the training dataset contained a disproportionate number of scans from a specific device vendor, and a recent firmware update changed the image normalization. The model’s performance degraded on images from the updated devices, but the monitoring system did not detect the distribution shift. Here, technical causality is not a single point; it is a combination of data bias, insufficient monitoring, and a change management gap. The investigation must reconstruct the timeline of changes, the versioned artifacts, and the runtime conditions to show how these factors jointly produced the misclassification.
Legal Causality: Foreseeability, Duty, and Proportionality
Legal causality in Europe is shaped by the principles of foreseeability, duty of care, and proportionality. Under the AI Act, providers of high-risk AI systems have explicit obligations to implement risk management, data governance, logging, and human oversight. If an incident occurs, regulators will ask whether the provider took those obligations seriously and whether the steps taken were reasonable given the risk level. For example, if a high-risk system is deployed in a critical infrastructure context, the expectation of robust monitoring and fail-safes is higher than for a low-risk application.
Legal causality also interacts with national tort regimes. In civil law jurisdictions, causation often involves an assessment of adequate condition and relevant condition (the “but-for” test and the “conditio sine qua non” with normative filtering). In common law jurisdictions, proximate cause and foreseeability play a central role. The AI Act does not harmonize tort liability, so cross-border incidents may require parallel analyses under national laws. The European Commission’s work on a possible AI liability directive indicates a trend toward shifting the burden of proof in certain cases, making it easier for victims to show causation when providers fail to meet regulatory obligations. Practically, this means that documentation demonstrating compliance can also serve as evidence of non-causation or mitigation.
Reconstructing AI Incidents: A Practical Methodology
Reconstruction is the process of assembling a coherent, evidence-backed narrative of what happened, when, and why. It should be structured, repeatable, and auditable. A robust methodology blends software forensics, data forensics, and operational inquiry.
Event Logging and Immutable Trails
Logging is the backbone of reconstruction. The AI Act’s logging obligation for high-risk systems (Article 12) requires automatic recording of events throughout the system’s lifecycle. In practice, this means capturing:
- Model versions and configuration at the time of the incident, including hyperparameters and feature sets.
- Input data identifiers and hashes, with timestamps and source metadata.
- Output values, confidence scores, and any human-in-the-loop decisions or overrides.
- System health metrics, latency, and resource utilization at the time of inference.
- Environmental context, such as user session identifiers, device types, or sensor readings.
Immutable logs are crucial. If logs can be altered post-incident, their evidentiary value collapses. Many organizations use append-only logs or cryptographic hashing to ensure integrity. In regulated environments, logs should be retained for periods aligned with sectoral obligations; for example, medical device regulations may require retention for the lifetime of the device plus a defined post-market period.
Data Lineage and Versioning
Data lineage answers the question: where did the data come from, and how was it transformed? A defensible reconstruction requires a map from raw data to features to training sets to production data. Tools like DVC, Pachyderm, or MLflow can track data and model versions. When an incident occurs, teams should be able to retrieve the exact training dataset, the preprocessing code, and the feature definitions that were active at the time of the event.
Lineage is particularly important in biotech and healthcare, where data may be subject to strict consent and provenance rules. If a model was trained on data collected under a specific consent framework, and a later incident involves a use case outside that framework, the legal and technical causal analysis will hinge on the documented lineage.
Model Forensics: Explainability and Counterfactuals
Explainability techniques help bridge the gap between technical mechanism and legal narrative. Local explanations (e.g., SHAP values, LIME) can show which features contributed to a specific output. Global explanations and partial dependence plots can reveal systematic biases or sensitivities. Counterfactual analysis asks: what minimal change to the input would have produced a different (and correct) output? This is not only a diagnostic tool; it can demonstrate whether the model’s behavior was reasonable given the input and context.
It is important to be cautious about over-interpretation. Explanations are approximations and can be misleading if not contextualized. A defensible approach combines multiple methods, documents their limitations, and anchors findings in domain expertise. For example, in a credit scoring incident, a SHAP analysis might show that a particular feature (e.g., “number of recent inquiries”) heavily influenced a rejection. The legal question is whether that feature is a permissible proxy under equal treatment rules, and whether the model’s reliance on it was disclosed and justified.
Human and Organizational Factors
AI incidents are rarely purely technical. Human decisions—such as overriding a recommendation, ignoring a monitoring alert, or approving a deployment under time pressure—often play a role. Organizational factors, like unclear ownership of model maintenance or insufficient training of operators, can be latent causes. A thorough reconstruction includes interviews, change management records, and access control logs. The goal is not to assign blame, but to understand the system as a socio-technical whole. In the EU’s safety culture approach for high-risk systems, human oversight must be meaningful; if operators lacked the information or time to intervene effectively, that is a causal factor.
Documentation for Defensible Narratives
Documentation is the medium through which technical findings are translated into legally persuasive evidence. It should be structured, versioned, and accessible to both technical and non-technical stakeholders, including regulators and courts.
Technical Documentation Under the AI Act
Providers of high-risk AI systems must prepare technical documentation that covers the system’s design, development, and testing (Article 13). In an incident context, this documentation becomes a baseline for assessing whether the system was built and operated in accordance with its intended purpose and risk management requirements. Key elements include:
- Purpose and scope: a clear statement of intended use and limitations.
- Risk management: the risk assessment, mitigation measures, and residual risk.
- Data governance: data sources, sampling strategies, labeling procedures, and bias mitigation.
- Testing and validation: performance metrics, stress tests, and results of conformity assessments.
- Human oversight: design of oversight features and instructions for use.
- Post-market monitoring: procedures for collecting and analyzing performance data after deployment.
When an incident occurs, the provider should be able to show how the documented measures were applied in practice, and how deviations were handled. If the system was updated post-deployment, the documentation should reflect version history and change impact analysis.
Incident Reports and Regulatory Notifications
The AI Act introduces a serious incident reporting obligation for high-risk AI systems (Article 73). Providers must notify national authorities of incidents that result in death or serious harm, or that otherwise meet the reporting criteria. The notification must include, among other things, a description of the incident, its causes, and the measures taken or proposed to address it. This is not a place for speculative narratives; it should be a factual account supported by logged evidence and preliminary forensic findings.
Reporting timelines are strict: a serious incident must be reported without undue delay and in any case within 15 days of becoming aware of it. In practice, teams should have a pre-defined incident response playbook that triggers immediate evidence preservation, triage, and drafting of the notification. The notification should be coordinated with any parallel obligations under sectoral rules, such as the Medical Device Regulation or the NIS2 Directive for cybersecurity incidents.
Traceability Artifacts: The Golden Thread
A defensible narrative relies on a “golden thread” of traceability artifacts that connect the incident to specific system states and decisions. These artifacts include:
- Model and data version identifiers with cryptographic hashes.
- Build and deployment logs, including CI/CD pipeline records.
- Configuration files and environment variables active at the time of the incident.
- Feature store snapshots and transformation code.
- Runtime request/response logs with correlation IDs.
- Monitoring dashboards and alert histories.
These artifacts should be stored in a manner that preserves integrity and supports audit. In multi-party ecosystems (e.g., cloud providers, data processors, and deployers), contractual agreements should specify logging responsibilities and data access rights to ensure that reconstruction is feasible.
EU Regulatory Context: How Causality Intersects with Obligations
The EU’s AI regulatory framework is layered. The AI Act sets horizontal requirements for AI systems; sectoral regulations (e.g., MDR, IVDR, NIS2, GDPR) impose additional duties; and national laws govern liability and enforcement. Causality sits at the intersection of these regimes.
AI Act: Risk Management and Post-Market Monitoring
High-risk AI systems must have a risk management system that identifies, analyzes, and mitigates risks throughout the lifecycle (Article 9). This includes iterative risk assessment, risk control measures, and testing. In an incident, regulators will examine whether the risk management process was adequate and whether it anticipated the type of failure that occurred. If not, the provider may be asked to explain why the risk was not identified and how the system was allowed to operate.
Post-market monitoring (Article 72) requires providers to collect and analyze data on system performance in the field. This is not optional; it is a continuous duty. A robust post-market monitoring plan enables early detection of drift, bias, or performance degradation. In a causality analysis, evidence of effective monitoring can demonstrate that the provider acted responsibly and that the incident was not foreseeable, or conversely, that warning signs were missed.
Logging and Traceability Obligations
Article 12’s logging obligation is central to causality. It ensures that high-risk AI systems are auditable and that regulators can reconstruct events. The logs must be automatically recorded and retained for a period determined by the provider, considering the system’s risk profile and sectoral rules. In practice, this means designing systems with observability as a first-class requirement, not an afterthought. For example, a hiring tool that produces automated rejections should log the specific inputs and model decisions, enabling an individual to exercise their right to an explanation under GDPR.
Sectoral Overlaps: MDR, IVDR, NIS2, and GDPR
In healthcare, the Medical Device Regulation (MDR) and In Vitro Diagnostic Regulation (IVDR) impose stringent obligations on clinical evaluation, performance monitoring, and vigilance. If an AI system is part of a medical device, an incident may trigger reporting under both the AI Act and the MDR. The causal analysis must align with the definitions of “incident” and “serious incident” in those regimes, which often focus on patient safety and device performance.
Cybersecurity incidents affecting AI systems may fall under NIS2, which requires incident reporting and risk management measures. Here, causality may involve threat actors, vulnerabilities, and compensating controls. The documentation should include security logs, patch histories, and risk assessments.
GDPR applies when personal data is processed. An AI incident involving personal data (e.g., unauthorized disclosure via model inversion) may require a DPIA, records of processing, and breach notification within 72 hours. The causality analysis should show how data protection measures failed and what safeguards were in place.
Liability and the Burden of Proof
While the AI Act harmonizes obligations, liability remains largely national. The European Commission has proposed an AI liability directive that would ease the burden of proof for victims in certain cases, particularly where a provider fails to meet regulatory duties. In practical terms, this means that robust compliance documentation can serve as a shield: it demonstrates that reasonable steps were taken and that the failure was not due to negligence. Conversely, gaps in documentation can be used to infer causation. Teams should therefore view documentation not as a bureaucratic task, but as a core risk management and liability defense tool.
Cross-Border Considerations and Regulatory Convergence
Europe is not a monolith. While the AI Act is a regulation (directly applicable), its implementation involves national authorities, conformity assessment bodies, and sectoral regulators. Enforcement practices and interpretations may vary.
National Competent Authorities and Enforcement
Each member state designates competent authorities for high-risk AI systems. For medical devices, these are often the same bodies that oversee MDR/IVDR. For other sectors, new authorities may be established. In an incident, the provider may face inquiries from multiple authorities, especially if the system is deployed across borders. Coordinated responses and consistent narratives are essential. Divergent national interpretations of “serious incident” or “adequate human oversight” can create compliance complexity. Maintaining a central repository of evidence and a single source of truth for incident documentation helps ensure consistency.
Harmonization vs. Local Nuance
Some countries have existing laws that interact with AI. For example, Germany’s Product Safety Act and its approach to automated decision-making under GDPR may influence how an incident is assessed. France’s CNIL has issued guidance on explainability and algorithmic transparency. Ireland’s Data Protection Commission has been active on GDPR enforcement for AI. Teams should be aware of local guidance and enforcement trends, and tailor their documentation accordingly. A defensible narrative in one jurisdiction may need augmentation to satisfy expectations in another.
Practical Steps for Multi-Jurisdiction Deployments
To navigate cross-border complexity, organizations should:
- Map the regulatory footprint of the AI system in each jurisdiction, including sectoral rules.
- Establish a unified incident response protocol that meets the strictest requirements (e.g., 15-day AI Act reporting, 72-hour GDPR breach notification).
- Use standardized templates for incident reports, with fields that can be adapted to local requirements.
- Ensure that contracts with processors and deployers specify evidence preservation and cooperation duties.
- Engage early with regulators through pre-submissions or sandbox programs when appropriate.
Operationalizing Causality: Governance, Tooling, and Culture
Establishing causality after an incident is easier if the groundwork is laid before it happens. This requires governance, tooling, and a culture that values traceability and continuous improvement.
Governance and Roles
Clear ownership of AI systems is essential. The AI Act assigns responsibilities to providers, but in practice, many systems involve multiple actors: data suppliers, model developers, integrators, and deployers. Governance frameworks should define:
- Who is responsible for model and data versioning.
- Who maintains the technical documentation.
- Who monitors performance and triggers retraining or rollback.
- Who is authorized to override model decisions and under what conditions.
- Who handles incident reporting and liaison with authorities.
For high-risk systems, a dedicated risk management function should be established, with direct reporting to senior management. This function should have the authority to halt deployments if risk controls are inadequate.