Legal Definitions vs Technical Fairness Metrics
European legal frameworks and technical engineering practices increasingly intersect in the governance of artificial intelligence, yet they operate on fundamentally different planes of abstraction and validation. When a data scientist optimizes a model for ‘equalized odds’ or ‘demographic parity,’ they are engaging in a mathematical exercise to balance statistical outcomes. When a regulator assesses whether that same system constitutes discrimination, they are applying legal principles rooted in fundamental rights, causality, and intent. The gap between these two perspectives is not merely semantic; it is a critical compliance risk. A model that is technically ‘fair’ by one or more mathematical metrics can still be found to be discriminatory under EU law, leading to enforcement actions, civil liability, and reputational damage. This article dissects the divergence between legal definitions of discrimination and technical fairness metrics, explaining why a metric-only approach is insufficient for legal compliance and how professionals must bridge this gap.
The Foundational Disconnect: Statistical Parity vs. Legal Status
The core of the issue lies in the subject of analysis. Technical fairness metrics operate on data distributions and model outputs. They treat individuals as members of statistical groups defined by sensitive attributes, typically race, gender, or age. The goal is often to ensure that the model’s predictive performance or error rates are balanced across these groups. Legal analysis, particularly under EU non-discrimination law, starts with the individual. It asks whether a person has been treated less favourably than another in a comparable situation on grounds of a protected characteristic. The legal framework is concerned with the status of the person, not the statistical properties of the group they belong to.
Consider the concept of disparate impact, a term more commonly associated with US legal doctrine but with parallels in the EU concept of indirect discrimination. A model might have a disparate impact if it disproportionately disadvantages a protected group, even if the algorithm itself is ‘blind’ to the sensitive attribute. A technical practitioner might see this as a problem to be solved by re-weighting the data or adjusting the decision threshold to achieve statistical parity. A lawyer, however, looks at this through the lens of indirect discrimination as defined in the EU Race Equality Directive and Gender Equality Directive. Indirect discrimination occurs where an apparently neutral provision, criterion, or practice would put persons of a particular racial or ethnic origin, or persons of a particular sex, at a particular disadvantage compared with other persons.
Crucially, under EU law, indirect discrimination can be justified if the practice is objectively justified by a legitimate aim and the means of achieving that aim are appropriate and necessary. This justification test is a legal, not a technical, assessment.
A technical fairness metric can identify a potential disparate impact. It cannot, by itself, determine whether that impact is legally justifiable. For example, a bank using an AI model for credit scoring might find that the model has a higher rejection rate for applicants from a specific minority group. A fairness metric flags this as a problem. The legal inquiry, however, proceeds to ask: Is this difference caused by the protected characteristic itself, or is it a reflection of other, non-protected factors that correlate with that characteristic (e.g., income, employment history, residential postcode)? If the model is accurately predicting creditworthiness based on legitimate risk factors, the resulting statistical imbalance might be legally permissible. The legal test is one of causality and justification, not just statistical balance.
Legal Definitions of Discrimination in the European Union
To understand why technical metrics fall short, one must first grasp the legal architecture. EU law prohibits both direct and indirect discrimination on various protected grounds, including sex, racial or ethnic origin, religion or belief, disability, age, and sexual orientation. This is established in primary law (the Treaty on the Functioning of the European Union and the Charter of Fundamental Rights) and detailed in secondary legislation (the so-called ‘Equality Directives’).
Direct Discrimination
Direct discrimination is the less favourable treatment of a person on grounds of a protected characteristic. In the context of AI, this is often straightforward to spot if the system explicitly uses a sensitive attribute as an input for decision-making. For instance, a recruitment tool that automatically down-ranks female applicants for a technical role is engaging in direct sex discrimination. Most technical fairness frameworks would easily detect this if the sensitive attribute is available for auditing. However, direct discrimination can also arise from proxies. If a model uses ‘postcode’ as a key feature and that postcode is a highly accurate proxy for ethnicity, the outcome may be direct discrimination if the use of the proxy is a deliberate or knowingly discriminatory practice. Proving intent or knowledge can be difficult, but the legal standard for direct discrimination does not always require proving intent.
Indirect Discrimination
This is the area where technical and legal views most frequently collide. Indirect discrimination is about neutral criteria that have a discriminatory effect. The legal test involves three steps:
- Identify a neutral provision, criterion, or practice (PCP).
- Establish that the PCP puts a group with a protected characteristic at a particular disadvantage.
- Establish that the disadvantage is not a proportionate means of achieving a legitimate aim.
A technical fairness metric can help with step 2 by quantifying the disadvantage. It cannot, however, perform steps 1 and 3. Identifying the ‘PCP’ requires a legal interpretation of what the AI system is actually doing. Is the PCP the algorithm’s code, the training data, the specific feature selection, or the final decision threshold? The answer is not always clear. Furthermore, the justification test (step 3) is a complex balancing act between the rights of the individual and the interests of the data controller. A company might argue that using a biased dataset is necessary to achieve high predictive accuracy, which is a legitimate commercial aim. A regulator would counter-argue whether that aim is proportionate and whether less discriminatory alternatives exist. This is a debate about values and trade-offs, not a mathematical optimization.
Technical Fairness Metrics: A Taxonomy of Limitations
Data scientists have developed a rich vocabulary of fairness metrics. Each metric captures a different aspect of statistical fairness, but each also embeds a specific set of assumptions that may not align with legal principles. The most common metrics fall into two categories: group fairness and individual fairness.
Group Fairness Metrics
Group fairness metrics aim to ensure that different demographic groups receive similar outcomes or error rates. Common examples include:
- Demographic Parity (or Statistical Parity): The proportion of positive outcomes should be the same across groups. For example, the percentage of loan approvals for men and women should be equal. This is often the first metric people think of, but it is legally problematic. It implies that the underlying base rates of creditworthiness must be identical across groups, which may not be true. Enforcing demographic parity might mean granting loans to unqualified applicants from an underrepresented group to meet a quota, which could be seen as direct discrimination against other applicants.
- Equalized Odds: The model should have equal true positive rates and equal false positive rates across groups. This is a more sophisticated metric that looks at error rates. It ensures that the model is equally good at identifying qualified candidates and equally bad at incorrectly rejecting qualified candidates for each group. This is often more aligned with the idea of equal treatment, but it still requires access to sensitive attributes for auditing and may conflict with other goals like overall accuracy.
- Equal Opportunity: A relaxation of equalized odds that only requires equal true positive rates (sensitivity). This focuses on ensuring that qualified individuals from all groups have an equal chance of being correctly selected. This is often seen as a pragmatic compromise, but it ignores the harm of false positives, which can also be discriminatory.
The fundamental limitation of all group metrics is that they treat individuals as interchangeable members of a group. They do not account for intersectionality (e.g., the unique experience of a black woman, which is not simply the sum of being black and being a woman). Legally, discrimination can occur on the basis of a combination of characteristics, and a metric that only looks at one dimension may miss this. Moreover, these metrics often require the very sensitive data that privacy laws and ethical guidelines discourage using. The GDPR’s restrictions on processing special categories of data (Article 9) create a direct tension with the need to monitor for discrimination using those same categories.
Individual Fairness
The concept of individual fairness attempts to address the limitations of group fairness. The core idea is that ‘similarly situated individuals should be treated similarly.’ If two individuals are identical in all relevant respects (except for a protected characteristic), the model should produce the same outcome for both. This is legally very appealing, as it mirrors the principle of equal treatment. However, it is technically very difficult to implement. How does one define ‘relevant respects’? What is the appropriate distance metric to measure similarity between complex, high-dimensional data points? Defining this in a way that is both mathematically sound and legally meaningful is a major challenge. It often requires domain expertise and subjective judgment calls that move away from pure statistical measurement.
Why Metric-Only Approaches Fail Legally: A Scenario
Imagine a public university in France uses an AI system to screen applicants for its prestigious engineering program. The system is trained on ten years of historical admission data. The data reflects historical societal biases: male applicants from affluent backgrounds were admitted at a higher rate. The university, wanting to be fair, commissions a team to build a ‘fair’ model. The team optimizes the model to achieve equalized odds, ensuring that the true positive rate (correctly identifying qualified candidates) and false positive rate (incorrectly admitting unqualified candidates) are the same for male and female applicants. They achieve this by adjusting the decision threshold for each group.
From a technical fairness perspective, the project is a success. The metric is met. However, from a legal perspective, the system is deeply flawed and likely illegal.
Failure 1: The Justification Problem
By explicitly using gender to adjust the threshold, the university is engaging in differential treatment based on a protected characteristic. This is direct discrimination. To defend this in court, the university would have to prove it falls under one of the limited exceptions in the Equality Directive, such as positive action. Positive action is permissible to prevent or compensate for disadvantages linked to a protected characteristic. However, it must be proportionate and not lead to absolute quotas. A court would scrutinize whether the specific mathematical adjustment used is a legitimate and proportionate form of positive action or simply a rigid quota system by another name. The technical metric does not provide this justification; it only provides the mechanism.
Failure 2: The Causality Problem
The model still relies on historical data that is biased. The ‘features’ it uses (e.g., high school grades, extracurricular activities, teacher recommendations) may be proxies for gender or socio-economic status. Even if the final output is ‘fair’ according to equalized odds, the internal logic of the model may be discriminatory. The legal principle of causality requires that the less favourable treatment be ‘on the grounds of’ the protected characteristic. If the model uses a proxy like ‘participation in competitive sports’ which is correlated with gender, and this proxy is the driver of the outcome, a legal argument can be made that the discrimination is still present, just masked. A technical audit focused only on output metrics would miss this.
Failure 3: The Transparency and Explainability Problem
Under the GDPR and the incoming AI Act, individuals have a right to an explanation of decisions made by automated systems. A rejected applicant would be entitled to know why they were not admitted. Telling an applicant ‘the model adjusted your score to meet an equalized odds metric’ is not a meaningful explanation. It does not explain the substantive reasons for the decision. Legally, the university must be able to point to the specific, non-discriminatory factors that led to the rejection (e.g., ‘your physics grade was below the required threshold’). A metric-only approach obscures the causal chain, making it difficult to provide the legally required transparency.
Failure 4: The GDPR and Sensitive Data
To even calculate equalized odds, the university needs to know the gender of the applicants to segment them and measure the metric. Processing gender data for this purpose is processing a special category of data under GDPR Article 9. While there may be a legal basis for this (e.g., explicit consent or substantial public interest), it is not automatic. The university must have a lawful basis, a clear purpose limitation, and robust data protection measures in place. A technical team might simply assume the data is available for ‘fairness auditing’ without undertaking the necessary Data Protection Impact Assessment (DPIA) and legal analysis required to process it lawfully. Using this data to ‘tune’ the model could be an unlawful processing activity in itself.
The Regulatory Landscape: EU AI Act and Non-Discrimination
The EU AI Act, while not explicitly a non-discrimination law, codifies obligations that directly address these risks. It classifies AI systems based on risk, with ‘high-risk’ systems (like those used in hiring, education, and critical infrastructure) facing the strictest requirements. These requirements move beyond simple metric checks and demand a systemic approach to fairness.
Requirements for High-Risk Systems
For high-risk AI systems, the AI Act mandates:
- Risk Management System: Providers must implement a continuous process to identify, analyze, and mitigate risks. This includes the risk of discrimination. A risk management system that only looks at output metrics is inadequate. It must analyze the entire lifecycle of the AI system, from data collection to deployment.
- Data Governance: The Act requires high-quality training, validation, and testing data. This includes measures to detect and correct biases. This implies that simply having a ‘fair’ metric is not enough; the underlying data must be managed to minimize bias from the start. This is a proactive, not a reactive, requirement.
- Transparency and Provision of Information: Users must be informed that they are subject to an AI system and must be given instructions for use. This includes information on the system’s capabilities and limitations. A system that is ‘fair’ only under specific statistical conditions must have those conditions clearly documented and communicated.
- Human Oversight: High-risk systems must be designed to enable human oversight. This is a key safeguard. The human overseer is expected to monitor the system’s operation and be able to intervene. This implies that the system’s decision-making process must be understandable to a human, which goes beyond a black-box model with a fairness metric attached.
- Accuracy, Robustness, and Cybersecurity: The system must be accurate and robust against errors. A model that is ‘fair’ on a test dataset but performs poorly on a slightly different real-world dataset is not robust. Fairness must be maintained over time and across different contexts.
The AI Act also introduces the concept of ‘conformity assessments’ and ‘CE marking’ for high-risk systems. This means a provider must self-assess (or, for certain systems, have a third-party auditor assess) that their system complies with all the requirements before placing it on the market. This assessment would have to include a justification for the system’s fairness properties, not just a report of metrics. The regulatory burden is on the provider to prove the system is non-discriminatory by design and in practice.
National Implementations and Enforcement
While the AI Act is a Regulation (directly applicable in all Member States), its implementation will rely on national authorities. Furthermore, non-discrimination law itself is implemented through national transpositions of the EU Directives. This creates a patchwork of enforcement.
In Germany, the General Equal Treatment Act (AGG) provides strong protections and allows for compensation claims for discrimination. German courts have a well-developed jurisprudence on indirect discrimination. A German regulator or court would be very rigorous in examining the justification for any statistical disadvantage found in an AI system. They would look closely at the concept of ‘objective justification’ and whether the company could have achieved its aim through less discriminatory means.
In France, the National Consultative Commission on Human Rights (CNCDH) and the data protection authority (CNIL) are active in this space. The CNIL has published guidance on algorithmic bias and the need for ‘algorithmic audits’. The French approach emphasizes the need for external scrutiny and transparency, going beyond internal metrics. The concept of ‘discrimination by proxy’ is well-established in French law.
In the Netherlands
In the Netherlands, the Dutch Data Protection Authority (AP) and the non-discrimination service (College voor de Rechten van de Mens) are key players. The Dutch have a strong tradition of using data for public administration, which has led to high-profile cases on algorithmic bias in welfare and tax systems. This has created a political and legal environment that is highly sensitive to the social impact of AI and skeptical of purely technical fixes. These national differences mean that a ‘one-size-fits-all’ compliance strategy based on a universal fairness metric is unlikely to succeed. A provider must understand the specific legal and enforcement environment in each Member State where their system is deployed. The legal test for justification, the burden of proof, and the available remedies can all vary. Given the divergence between legal principles and technical metrics, how can organizations build and deploy AI systems that are both effective and legally compliant? The solution is not to abandon metrics, but to embed them within a broader, legally-informed governance framework. This requires a fusion of expertise from legal, ethical, data science, and business teams. The process should begin not with choosing a metric, but with a legal and ethical risk assessment. This involves:Bridging the Gap: A Multi-Disciplinary Approach
From Metrics to Mandates