Public Procurement for AI: What Vendors Must Prove

Public procurement for artificial intelligence systems in Europe is shifting from a purely technical evaluation to a rigorous legal and operational assessment. Public buyers are no longer simply purchasing software; they are acquiring socio-technical systems that must comply with a dense web of regulations, from data protection and fundamental rights to sector-specific safety rules. For vendors, this means the burden of proof has moved beyond functionality and cost. You must now demonstrate, through verifiable evidence and comprehensive documentation, that your AI solution is lawful, secure, and governable from the moment of contract signing through its entire lifecycle. This article analyzes the core requirements shaping these procurements, drawing on the evolving landscape of the EU AI Act, the GDPR, the NIS2 Directive, and national implementations, to provide a practical guide for vendors seeking to win and fulfill public contracts.

The Evolving Legal Basis for AI Procurement

Public procurement in the EU is governed by the Public Procurement Directives (2014/24/EU and 2014/23/EU), transposed into national law by Member States. These directives provide the procedural framework for awarding contracts and allow contracting authorities to select suppliers based on criteria such as the most economically advantageous tender (MEAT). Historically, this has often prioritized price and technical capability. However, the introduction of the EU AI Act (Regulation (EU) 2024/1689) fundamentally alters the technical specifications and award criteria that can be lawfully used for AI systems.

The AI Act establishes a risk-based framework, classifying AI systems into four categories: unacceptable, high, limited, and minimal risk. For the public sector, the most relevant are high-risk and limited-risk systems. High-risk AI systems, as defined in Annex III of the Act, include those used in critical infrastructure, education, employment, essential public services, and law enforcement. When a public authority procures a high-risk AI system, it effectively becomes a “deployer” under the AI Act. This status carries direct legal obligations, including conducting a Fundamental Rights Impact Assessment (FRIA) before deployment, ensuring human oversight, and maintaining logs of operation.

Consequently, a vendor’s ability to prove compliance is no longer a “nice-to-have” but a prerequisite for market access. The technical specifications in a tender document will increasingly demand evidence that the AI system is designed to comply with the AI Act’s requirements for data quality, transparency, and robustness. The burden of proof rests on the vendor to provide this evidence in a format that the public buyer can scrutinize and rely upon. This represents a significant departure from previous generations of software procurement, where the internal workings of a system were often treated as a “black box.”

From Technical Specifications to Conformity Evidence

Under the Public Procurement Directives, contracting authorities define their needs in the “technical specifications.” For AI, these specifications are evolving. Instead of simply requesting “an AI model for X,” authorities are now specifying requirements for how the AI system is developed, validated, and managed.

A key concept here is the Technical Documentation required by the AI Act. This is not a user manual. It is a comprehensive file intended for national market surveillance authorities, containing detailed information about the system’s design, development, and testing. For vendors, this documentation becomes a central asset in procurement. When responding to a tender, a vendor should be prepared to provide a summary or an audited version of this technical documentation to demonstrate compliance with key requirements.

Legal Definition (AI Act, Art. 3(2)): ‘conformity assessment’ means the process of verifying whether the requirements relating to an AI system have been fulfilled.

While a formal conformity assessment by a third-party notified body is only mandatory for high-risk AI systems that are not already subject to other sectoral legislation (e.g., medical devices, machinery), the *spirit* of this assessment is becoming the baseline for any serious AI procurement. Public buyers will expect vendors to have undergone, at a minimum, an internal conformity assessment and to provide the associated documentation. This includes a detailed description of the system’s intended purpose, its capabilities and limitations, and the underlying algorithms.

Core Pillars of Evidence: What Vendors Must Prove

To succeed in a modern public procurement for AI, vendors must build their bid around four pillars of evidence: Data Governance, Transparency and Explainability, Security and Resilience, and Governance and Oversight. These pillars map directly to the obligations of both the vendor (as provider) and the public buyer (as deployer).

Data Governance and Quality

The AI Act mandates that high-risk AI systems be trained, validated, and tested on data sets that are relevant, representative, free of errors, and complete. This is a significant challenge. Public buyers are acutely aware of the risks of biased data leading to discriminatory outcomes, particularly in areas like social benefits, policing, or recruitment.

Vendors must therefore provide robust evidence of their data handling practices. This goes beyond a simple statement of “we used a large dataset.” It requires a detailed data governance framework that documents:

• Data Provenance: Where did the data come from? What are the legal grounds for its use (e.g., consent, public task)? This is inextricably linked to GDPR compliance.
• Data Suitability: How was the data selected? How does the vendor ensure it is representative of the population the AI system will interact with? This is particularly critical for avoiding discrimination under the EU Charter of Fundamental Rights.
• Data Pre-processing: What steps were taken to clean, label, and augment the data? Any assumptions or transformations must be documented.
• Bias Mitigation: What technical and organizational measures were implemented to detect and mitigate biases in the data and the model? This could include techniques like re-weighting, adversarial debiasing, or diverse data sourcing.

In practice, a vendor might be asked to provide a Data Sheet or a Model Card, which are emerging industry standards for documenting AI systems. These documents provide a structured way to communicate the intended use, limitations, and performance metrics of an AI model, including its performance across different demographic groups. A public buyer will use these documents to assess whether the system is suitable and lawful for their specific context.

Transparency and Explainability

Transparency is a cornerstone of the AI Act and a fundamental principle of public administration. The public has a right to understand how decisions affecting them are made. This principle manifests in several ways in procurement.

First, the AI Act requires that AI systems be designed and developed in a way that allows operators to understand how they work and to interpret their outputs. For high-risk systems, this means providing “instructions for use” that are clear and accessible to the deployer. Vendors must prove that their system is not an inscrutable “black box.”

Second, for systems interacting with individuals (limited-risk AI, like chatbots), there is an obligation to inform those individuals that they are interacting with an AI system. In a public service context, this is straightforward. But for more complex systems, transparency means providing explanations for specific outputs. If an AI system is used to assess eligibility for a social benefit, the individual has a right to an explanation if their application is rejected.

Vendors must therefore demonstrate the explainability features of their system. This could involve:

• Providing feature importance scores (e.g., SHAP or LIME values) that show which factors most influenced a particular decision.
• Designing user interfaces that present the AI’s output alongside confidence scores and key contributing factors.
• Ensuring that the system’s decision-making logic is auditable, allowing a human expert to trace the path from input to output.

The level of explainability required will depend on the context. A system for predicting traffic flow may require less granular explainability than a system for flagging potential welfare fraud. Vendors should be prepared to discuss and demonstrate these capabilities during the procurement process.

Security, Robustness, and Resilience

AI systems introduce new security vulnerabilities beyond traditional software. Adversarial attacks, data poisoning, and model extraction are specific threats that public buyers are increasingly aware of. The obligation to ensure security is reinforced by the NIS2 Directive, which sets cybersecurity risk management requirements for essential and important entities, many of which are public sector bodies.

Vendors must provide evidence that their AI system is robust against both malicious attacks and unintentional errors. This includes:

• Adversarial Robustness: Testing the model against inputs designed to cause misclassification. For example, can a small, imperceptible change to a medical image cause the AI to give a wrong diagnosis?
• Data Integrity: Measures to prevent “data poisoning,” where an attacker injects malicious data during the training phase to corrupt the model’s behavior.
• Model Security: Protecting the model itself from being stolen or reverse-engineered.
• Operational Resilience: Ensuring the system has fallback mechanisms and can operate safely if it encounters data it was not trained on (out-of-distribution data). The system should be able to signal uncertainty rather than making a confident but wrong guess.

Procurement tenders may include specific requirements for security testing, such as penetration testing or red-teaming exercises, where the vendor’s system is subjected to simulated attacks. The results of these tests, and the subsequent remediation, become part of the evidence portfolio.

Governance and Human Oversight

The AI Act places a strong emphasis on human oversight. The goal is to ensure that a human, not the machine, has the final say, especially in high-stakes decisions. For public sector deployers, this is also a constitutional requirement in many Member States.

Vendors must design their systems to support effective human oversight. This is not just about a “human-in-the-loop” but about ensuring the human operator is properly equipped to understand the AI’s output and intervene effectively. Evidence to be provided includes:

• Interface Design: How does the user interface present information to the human operator? Does it clearly distinguish between AI-generated and human-entered data? Does it highlight uncertainty or potential errors?
• Override Functionality: Can the human operator easily override or ignore the AI’s recommendation? The system should be designed to make this simple, not cumbersome.
• Logging and Audit Trails: The system must log its operations, including inputs, outputs, and any human interventions. This is crucial for accountability and for investigating incidents. The vendor must specify what is logged and for how long.
• Training and Documentation: The vendor must provide comprehensive training materials for the public sector’s staff, explaining the system’s capabilities, limitations, and the correct procedures for human oversight.

During procurement, vendors may be asked to provide a “Governance Plan” or “Operational Playbook” that details how the system will be managed in a real-world public sector environment, including roles, responsibilities, and procedures for handling disputes or errors.

National Implementations and Divergences

While the EU AI Act provides a harmonized framework, its implementation and the broader procurement landscape will still show significant national variation. Public procurement is a shared competence, and Member States have discretion in how they transpose the directives and structure their public administration.

For example, Germany has a strong tradition of “Technikfolgenabschätzung” (technology assessment) and data protection. German public buyers, particularly at the federal level, are likely to be exceptionally rigorous in demanding evidence of bias mitigation and data protection impact assessments (DPIAs) under GDPR. The German procurement law (GWB) has specific provisions for electronic procurement and may favor vendors who can integrate seamlessly with government platforms.

In France, the CNIL (Commission nationale de l’informatique et des libertés) is a very active data protection authority. French public buyers will pay close attention to the legal basis for data processing and the minimization of data collection. There is also a strong emphasis on the “digital sovereignty” of the state, which may lead to preferences for solutions that can be hosted on-premise or within a trusted European cloud environment.

Spain and Italy have been actively developing national strategies for AI in the public sector. Spain’s “Plan Nacional de IA” promotes the use of AI for improving public services, and its procurement guidelines may reflect a focus on interoperability and accessibility. Italy’s data protection authority (Garante per la protezione dei dati personali) has shown a proactive stance in regulating AI, as seen in the temporary ban of ChatGPT. This signals that Italian public buyers will be particularly sensitive to issues of data privacy and the use of personal data in large language models.

These national differences mean that a one-size-fits-all procurement strategy is unlikely to succeed. Vendors must understand the specific regulatory culture and administrative priorities of the Member State they are targeting. A key part of the bidding process is demonstrating awareness of these national nuances and showing how the solution aligns with both EU-level and national-level requirements.

The Role of Standardization and Certification

To provide legal certainty and streamline compliance, the AI Act mandates the development of harmonized standards. The European standardization organizations (CEN, CENLEC) are already working on these standards, which will provide a presumption of conformity with the AI Act’s requirements.

For vendors, aligning with these future standards will be a powerful way to prove compliance. While the standards are still under development, they are expected to cover:

• Risk management systems for AI.
• Data quality and governance frameworks.
• Technical documentation requirements.
• Transparency and provision of information to users.
• Logging and auditability.
• Robustness, accuracy, and cybersecurity.

In the absence of final harmonized standards, vendors can refer to other relevant standards, such as those from ISO/IEC (e.g., the ISO/IEC 42001 for AI management systems) or sector-specific standards. Public buyers will look favorably on vendors who can demonstrate adherence to recognized best practices, even before the official EU standards are published. Certification schemes, once established by the European Commission, will offer an even stronger form of proof, allowing vendors to undergo a third-party assessment to certify their conformity.

Practical Steps for Vendors in a Procurement Process

Navigating an AI procurement requires a structured approach that integrates legal, technical, and commercial expertise. Vendors should consider the following steps to build a compelling and compliant bid.

1. Pre-Procurement Engagement and Scoping

Engage with the contracting authority before the tender is published. This is often done through market consultations or “pre-commercial procurement” (PCP) processes. Use this opportunity to understand the buyer’s specific needs, risk tolerance, and existing infrastructure. Crucially, help them understand the capabilities and limitations of your AI technology. This educational role can build trust and ensure the final tender specifications are realistic and well-informed.

2. Building a “Compliance-by-Design” Bid

Your tender response should not treat compliance as a separate section. It should be woven into every aspect of your technical and financial proposal.

• Technical Description: For each feature, explain how it addresses a specific legal or regulatory requirement (e.g., “Our explainability dashboard directly addresses the AI Act’s requirement for human oversight and the user’s right to an explanation”).
• Data Management: Provide a dedicated data governance annex. Include data sheets, provenance records, and bias mitigation reports.
• Security: Include a security annex detailing your SDLC (Secure Development Lifecycle), threat modeling, and results of any penetration testing. Align your description with the requirements of NIS2.
• Operational Plan: Detail the support, training, and logging capabilities you will provide. Include a draft of the “instructions for use” required by the AI Act.

3. The Role of the Conformity Assessment

Even if a formal third-party conformity assessment is not yet mandatory for your specific AI system, you should conduct an internal one. Document this process meticulously. This demonstrates maturity and foresight. If you are developing a high-risk AI system for which a notified body is required, you should be able to outline your timeline and strategy for achieving certification. This provides the public buyer with assurance that the system will be legally deployable by the time of contract completion.

4. Managing Liability and Contractual Clauses

Public procurement contracts for AI will need to address liability in a nuanced way. Who is responsible if the AI system makes a mistake that leads to a wrongful decision by a public official? The vendor, the deployer, or both?

Vendors must be prepared to negotiate clear contractual clauses that delineate responsibilities. These clauses should cover:

• The accuracy and performance levels the system is expected to maintain (Service Level Agreements or SLAs).
• Responsibility for monitoring the system’s performance post-deployment and retraining it if its accuracy degrades (“model drift”).
• Indemnification clauses in case of failures due to defects in the system’s design or training data.
• Data ownership and usage rights, especially regarding data generated by the system during operation.

It is important to note that under the AI Act, the ultimate responsibility for ensuring a high-risk AI system is used in compliance with the law rests with the deployer (the public authority). However, the vendor’s contractual obligations to provide a compliant system, along with comprehensive instructions and support, will be a key factor in determining liability in the event of a dispute.

Conclusion: A New Paradigm of Proof

The era of procuring AI as a simple utility is over. For vendors targeting the European public sector, success hinges on the ability to provide transparent, verifiable, and comprehensive evidence of their system’s