Security, Privacy, and Governance: How They Connect in AI

Artificial intelligence systems do not exist in a vacuum. They are built upon complex data pipelines, deployed on distributed infrastructure, and embedded in organisational decision-making structures. When these systems fail, the failure is rarely confined to a single technical domain. A vulnerability in a model’s training pipeline can trigger a privacy breach, which in turn exposes an organisation to regulatory sanctions and reputational damage. This interconnectedness is why security, privacy, and governance must be treated not as parallel workstreams but as a single, integrated discipline. For professionals working with AI, robotics, biotech, and data systems in Europe, understanding this integration is essential to designing and operating compliant, resilient, and trustworthy systems.

The European regulatory landscape is evolving rapidly to address these interdependencies. The General Data Protection Regulation (GDPR) established a robust baseline for data protection, while the Network and Information Security Directive (NIS2) strengthens cybersecurity requirements for a wide range of entities. The AI Act introduces a horizontal framework for the regulation of artificial intelligence, imposing obligations related to data quality, transparency, human oversight, and robustness. These instruments, together with national laws and sector-specific rules, create a complex web of obligations. Navigating this web requires a practical understanding of how security controls, privacy principles, and governance mechanisms interact in the day-to-day operation of AI systems.

The Interdependent Triad: Security, Privacy, and Governance

It is useful to begin with clear, functional definitions. In the context of AI, security refers to the protection of systems, networks, and data from unauthorised access, use, disclosure, alteration, or destruction. This encompasses classical cybersecurity (e.g., network hardening, access control) and AI-specific concerns (e.g., adversarial attacks, model inversion, data poisoning). Privacy, under the GDPR, is the protection of personal data and the rights of individuals. It is operationalised through principles like lawfulness, fairness, transparency, purpose limitation, data minimisation, accuracy, storage limitation, integrity and confidentiality, and accountability. Governance is the framework of policies, roles, responsibilities, and processes that ensures an organisation acts in accordance with its obligations and objectives. It is the mechanism through which strategic decisions are made and controlled.

These three domains are mutually reinforcing. Strong security is a prerequisite for confidentiality and integrity, which are core privacy principles. Effective governance ensures that security and privacy controls are defined, resourced, monitored, and audited. A failure in one domain inevitably cascades into the others. A governance failure, such as the absence of a clear policy for data labelling, can lead to the ingestion of improperly sourced data (a privacy failure). This data, if not secured, can be exfiltrated (a security failure). The resulting system may be non-compliant, unreliable, and harmful. The AI Act, GDPR, and NIS2 implicitly recognise this interplay by imposing overlapping and mutually dependent obligations.

Conceptualising the Cascade

Consider the lifecycle of a typical high-risk AI system used in a European public hospital for diagnostic imaging. The system is developed, deployed, and maintained over several years. Each phase presents unique risks that span security, privacy, and governance.

• Development: The system is trained on a vast corpus of medical images. The governance framework must ensure a lawful basis for processing this sensitive data under GDPR Article 9. It must also define data quality standards. A security failure here could be an insecure data transfer from the hospital to the developer, leading to a data breach. A privacy failure could be the use of data without proper patient consent or a valid legal basis. A governance failure would be the lack of a documented data provenance policy.
• Deployment: The model is integrated into the hospital’s IT infrastructure. Security requires robust authentication and network segmentation. Privacy requires ensuring that the model’s outputs do not inadvertently reveal personal data. Governance requires a process for clinical validation and risk assessment under the AI Act.
• Operation: The system provides diagnostic suggestions to clinicians. Security requires continuous monitoring for adversarial attacks that could manipulate inputs to produce incorrect diagnoses. Privacy requires logging and auditing to track who accessed the system and for what purpose. Governance requires a process for incident reporting, model drift monitoring, and managing updates.

A failure at any point can trigger a chain reaction. For instance, a failure to implement robust access controls (security) could allow an unauthorised user to access the training dataset (privacy). This breach would violate GDPR and could lead to a fine of up to 4% of global annual turnover. The incident would also trigger the AI Act’s obligations for reporting serious incidents, as the lack of data integrity could compromise the system’s performance and patient safety. The root cause would likely be traced back to a governance gap, such as inadequate security testing in the pre-deployment phase.

Regulatory Frameworks: An Integrated View

European regulation is not a monolith. It is a layered system of overlapping and complementary rules. Understanding how these rules interact is key to building a coherent compliance strategy.

GDPR: The Foundation of Data Protection

The GDPR is the cornerstone of privacy regulation in Europe. For AI systems, its relevance is profound. Article 5(1)(f) requires that personal data be processed in a manner that ensures appropriate security, including protection against unauthorised or unlawful processing and against accidental loss, destruction or damage, using appropriate technical or organisational measures. This directly links privacy and security. The “integrity and confidentiality” principle is not just about encryption; it extends to ensuring the reliability of AI systems that process personal data.

Article 22 of the GDPR grants individuals the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning them or similarly significantly affects them. This has direct implications for governance. Organisations must implement measures to ensure human oversight and the ability to contest decisions. This is not merely a procedural requirement; it necessitates technical and organisational controls that allow a human to understand the logic of the AI and intervene. This is a governance-driven requirement that has technical and security implications (e.g., logging, explainability interfaces, access controls for human reviewers).

Furthermore, the concept of Privacy by Design and by Default (Article 25) mandates that data protection measures are embedded into the design of systems. This is a core governance principle that forces a proactive approach. It means that security controls (e.g., anonymisation techniques, differential privacy) and privacy principles (e.g., data minimisation) must be considered at the architecture stage, not bolted on as an afterthought.

The AI Act: A Risk-Based Horizontal Framework

The AI Act introduces a risk-based approach, classifying AI systems into unacceptable, high, limited, and minimal risk categories. The obligations intensify with the risk level. For high-risk AI systems, the Act imposes extensive requirements that directly intersect with security, privacy, and governance.

Article 10: Data and Data Governance. This is a pivotal article. It requires that training, validation, and testing data sets be relevant, representative, free of errors, and complete. It also mandates that appropriate data governance and management practices be applied. This is a governance requirement with a technical and privacy dimension. It demands that organisations have policies for data sourcing, labelling, and cleaning. It implicitly requires that the data be processed lawfully under GDPR. A failure in data governance under the AI Act can be a GDPR violation if personal data is used improperly.

Article 14: Human Oversight. This requirement ensures that humans can oversee the system’s operation and intervene where necessary. This is a governance control that has security implications. For example, the oversight mechanism must be protected from tampering. The human overseer must be trained and have the authority to override the system, which requires robust access controls and a clear chain of command.

Article 15: Accuracy, Robustness, and Cybersecurity. The Act explicitly requires that high-risk AI systems be robust against errors and faults. They must be resilient to attempts to alter their use or performance by third parties. This is a direct cybersecurity requirement. It means that organisations must conduct adversarial testing and implement security measures to protect the model itself. A model that is not robust against adversarial attacks is not compliant with the AI Act. This links security directly to the legal obligation to ensure the system is fit for purpose.

Article 72: Serious Incidents. Providers of high-risk AI systems must report serious incidents to the national competent authorities. A serious incident is one that leads to the death or serious injury of a person, or causes significant damage to property or the environment. A security breach that results in the manipulation of a high-risk AI system (e.g., an autonomous vehicle’s perception system) could constitute a serious incident. This creates a direct link between cybersecurity incident response and regulatory reporting obligations.

NIS2 Directive: Strengthening Cybersecurity Posture

The NIS2 Directive (Directive (EU) 2022/2555) significantly expands the scope of entities subject to cybersecurity risk management measures and reporting obligations. It covers a wide range of sectors, including energy, transport, health, and digital infrastructure. Many organisations developing or deploying high-risk AI systems will fall under NIS2.

NIS2 requires entities to adopt a comprehensive set of security measures, including risk analysis, incident handling, business continuity, supply chain security, and the use of cryptography. These are foundational security controls that support both GDPR and AI Act compliance. For example, a robust incident handling process under NIS2 will help an organisation detect and respond to a personal data breach (GDPR) and a serious incident (AI Act). The directive also introduces stricter penalties and personal liability for management bodies, reinforcing the governance aspect of cybersecurity.

National Implementations and Sector-Specific Rules

While these frameworks are EU-wide, their implementation can vary. Member States designate national competent authorities for the AI Act and data protection authorities for GDPR. The interpretation of “high-risk” or “serious incident” may differ slightly in practice. For example, Germany’s approach to AI regulation builds upon its established data protection culture and the Federal Office for Information Security (BSI). France’s CNIL is known for its active enforcement of GDPR and its focus on data minimisation in AI projects.

Sector-specific rules add another layer. In the financial sector, the Digital Operational Resilience Act (DORA) imposes stringent cybersecurity and third-party risk management requirements on financial institutions, many of whom use AI. DORA’s focus on resilience and testing complements the AI Act’s robustness requirements. In the healthcare sector, national rules on medical devices, combined with GDPR for patient data, create a highly regulated environment for AI diagnostics.

Practical Examples of Failure Cascades

To illustrate the practical connection between security, privacy, and governance, let us examine three realistic failure scenarios in the European context.

Scenario 1: The Data Poisoning of a Hiring Algorithm

A multinational corporation uses an AI system to screen job applications. The goal is to improve efficiency and identify top candidates. The system is trained on historical hiring data and CVs.

The Governance Failure: The project was initiated by the HR department with a focus on speed-to-hire. The governance framework was weak. There was no dedicated AI ethics board, and the legal and compliance teams were consulted late in the process. The policy for data sourcing was vague, allowing the use of scraped data from public professional networking sites without proper vetting for bias or relevance.

The Security Failure: The data ingestion pipeline was not secured. An external actor (or even a disgruntled employee) could inject manipulated data into the training set. This is known as data poisoning. The attacker subtly alters the characteristics of successful candidates in the training data, for example, by associating certain keywords or demographic indicators with high scores. The security failure is the lack of integrity controls over the training data.

The Privacy Failure: The use of scraped data likely violates GDPR. There is no lawful basis for processing this personal data, as the individuals did not consent. Furthermore, the system may be making decisions based on special categories of data (e.g., inferred political opinions, health status from activity patterns) without a legal basis under Article 9. This is a significant privacy breach.

The Cascade: The poisoned model starts exhibiting biased behaviour, systematically downgrading qualified candidates from certain backgrounds. This leads to complaints and an investigation by the national data protection authority (DPA). The DPA finds GDPR violations (lack of lawful basis, potential for discriminatory processing). The AI Act would classify this as a high-risk system, and the provider would be required to demonstrate data governance and bias mitigation. The failure to do so would be a breach of the AI Act. The incident causes reputational damage, regulatory fines, and the need to completely re-engineer the system. The root cause was a governance failure to establish a secure and legally compliant data pipeline.

Scenario 2: The Model Inversion Attack on a Smart City Platform

A European city deploys an AI system to optimise energy consumption in public buildings. The system collects granular data from smart meters, which are linked to specific buildings (and thus, indirectly, to the people inside). The system uses this data to predict demand and adjust heating and lighting.

The Governance Failure: The city’s procurement process for the AI system did not adequately assess privacy and security risks. The contract with the vendor did not specify robust security testing requirements or a clear process for handling security vulnerabilities discovered post-deployment. There was no public-facing transparency report explaining how the data was used and protected.

The Security Failure: The model’s API is publicly accessible to allow for integration with other city services. However, it lacks rate limiting and proper input sanitisation. An attacker discovers that they can use the model’s predictions to infer sensitive information about the building’s occupants. This is a model inversion attack. By submitting carefully crafted queries and observing the model’s outputs (e.g., predicted energy usage), the attacker can reconstruct parts of the original training data, revealing patterns of life and potentially identifying individuals.

The Privacy Failure: The model inversion attack constitutes a data breach under GDPR. The attacker has gained unauthorised access to personal data. The city, as the data controller, is liable. The lack of technical measures to prevent such an attack (e.g., differential privacy, secure aggregation) can be seen as a failure to ensure the “integrity and confidentiality” of the data.

The Cascade: The attack is discovered and reported in the media. The city faces public backlash and a formal investigation by the DPA. The DPA issues an order to suspend the system and imposes a fine. The AI Act’s requirements for robustness and security (Article 15) are shown to be violated. The city must now invest in securing the model, notifying affected parties, and rebuilding public trust. The initial governance failure to prioritise security and privacy by design led directly to this crisis.

Scenario 3: The Insider Threat in a Biotech Research Project

A biotech company in Switzerland (which follows GDPR via its treaty with the EU) is using AI to analyse genomic data from clinical trial participants to identify potential drug targets. The data is highly sensitive and subject to strict confidentiality agreements.

The Governance Failure: The company’s internal access control policies are outdated. They do not follow the principle of least privilege. Researchers have broad access to the entire dataset, even if their specific project only requires a subset. There is no clear policy on the use of external cloud resources for model training.

The Security Failure: A researcher, intending to work on the project from home, copies the entire genomic dataset to a personal, unencrypted external hard drive. The hard drive is subsequently lost or stolen. This is a classic data breach resulting from a failure to enforce technical controls (e.g., data loss prevention, endpoint security) and a lack of secure remote access solutions.

The Privacy Failure: The loss of the hard drive is a personal data breach affecting thousands of clinical trial participants. The company must notify the DPA and the affected individuals under GDPR Article 33 and 34. The breach is severe due to the sensitivity of the data (genetic data is a special category). The company had a legal obligation to protect this data with appropriate technical and organisational measures.

The Cascade: The incident triggers a regulatory investigation, potential fines, and civil lawsuits from participants. The company’s reputation for handling sensitive data is damaged, potentially jeopardising future collaborations and recruitment for clinical trials. The AI Act would also be relevant if the AI system were to be used in a clinical context, as the lack of data integrity and confidentiality would undermine the system’s reliability. The core issue was a governance failure to define and enforce clear data handling and access policies, compounded by a security failure to protect the data outside the secure research environment.

Building an Integrated Compliance and Operational Framework

Avoiding these cascading failures requires a holistic approach that embeds security, privacy, and governance into the entire AI lifecycle. This is not about creating three separate checklists but about building a single, coherent framework.

From Silos to Synergy: The Role of the DPO, CISO, and AI Ethics Lead

Traditionally, the Data Protection Officer (DPO), the Chief Information Security Officer (CISO), and legal/compliance teams have operated in relative isolation. For AI, this is no longer viable. An effective governance structure requires these roles to collaborate closely. The DPO ensures GDPR compliance, the CISO ensures security, and the AI Ethics Lead (or equivalent) ensures the system aligns with broader ethical principles and the AI Act’s requirements. They should form a core review body for high-risk AI projects, assessing risks from their respective perspectives and ensuring a unified response.

Privacy and Security by Design in the AI Lifecycle

Integrating these principles means taking concrete actions at each stage:

1. Conception & Design: Conduct a combined Data Protection Impact Assessment (DPIA) under GDPR and a Risk Assessment under the AI Act. This should be a joint effort. The DPIA will identify privacy risks, which can be mitigated by security controls (e.g., anonymisation) and governance policies (e.g., data retention schedules). The AI Act risk assessment will identify risks to health, safety, and fundamental rights, which can be mitigated by robust design, human oversight, and security hardening.
2. Data Acquisition & Preparation: