Human Factors in Clinical AI: Overreliance and Workflow Risk

The integration of artificial intelligence into clinical environments represents a paradigm shift in how medical data is processed, interpreted, and acted upon. While the technological capabilities of machine learning models, particularly in diagnostic imaging and predictive analytics, offer unprecedented potential for early detection and personalized treatment, they simultaneously introduce complex vulnerabilities at the intersection of human cognition and algorithmic output. The European regulatory landscape, anchored by the AI Act and the Medical Devices Regulation (MDR), is increasingly focused on these “human factors”—the cognitive and ergonomic variables that determine whether an AI system enhances patient safety or inadvertently introduces new risks. Understanding these dynamics is not merely a matter of user interface design; it is a fundamental requirement for regulatory compliance, clinical efficacy, and ethical deployment.

The Cognitive Ecology of Clinical AI

When a clinician interacts with an AI system, they are not simply using a tool; they are entering into a cognitive partnership. This partnership is often asymmetrical. The AI processes vast datasets with consistent logic, while the human operates under conditions of fatigue, time pressure, and cognitive load. The primary risks arising from this asymmetry—overreliance, under-reliance, and workflow friction—are often categorized under the umbrella of automation bias and alarm fatigue. These are not theoretical concerns; they are documented sources of adverse events in high-reliability industries, from aviation to nuclear power, and they manifest with distinct characteristics in healthcare.

From a regulatory perspective, the “state of the art” (as defined in the AI Act and MDR) increasingly requires that manufacturers demonstrate not just the algorithmic accuracy of their devices, but their usability in real-world clinical workflows. This necessitates a shift from viewing the AI as an isolated software component to viewing it as an active participant in a socio-technical system.

Automation Bias and the Erosion of Critical Review

Automation bias refers to the tendency of humans to favor suggestions from automated decision-making systems and to ignore contradictory information made without automation, even if it is correct. In a clinical setting, this can manifest when a radiologist, viewing an AI-generated overlay highlighting a potential nodule, focuses exclusively on the highlighted area and fails to scrutinize the surrounding tissue for subtle, unflagged anomalies. The AI acts as a cognitive anchor, narrowing the clinician’s field of attention.

Regulatory bodies are beginning to scrutinize how AI systems are designed to mitigate this bias. The AI Act classifies most clinical AI as “High-Risk” (Annex III), subjecting it to rigorous conformity assessments. A key requirement is the design of systems that ensure human oversight. However, “oversight” is not a passive state. It requires active engagement. If the AI interface is designed such that the path of least resistance is to accept the AI’s recommendation (e.g., a single-click approval for a treatment plan), the system is effectively encouraging automation bias.

Under the AI Act, high-risk AI systems must be designed and developed to allow oversight by humans, who must be able to intervene in the operation of the system or override decisions.

This legal obligation implies that the system architecture must support, rather than hinder, the user’s ability to maintain skepticism. For example, in Germany, the Digitale-Versorgung-Gesetz (DVG) emphasizes the “benefit to the patient” (Patientennutzen). Regulators evaluating such systems may ask: Does the interface present the AI’s confidence score? Does it visualize the features that led to the decision (explainability)? If the system is a “black box” that outputs a diagnosis without context, it increases the risk of uncritical acceptance.

Alert Fatigue and the Signal-to-Noise Ratio

While automation bias concerns the acceptance of AI output, alert fatigue concerns the rejection of it. Clinical decision support systems (CDSS) often rely on rule-based logic or predictive models to flag risks, such as sepsis onset or medication interactions. If the sensitivity is set too high to maximize the capture of true positives, the result is a deluge of false positives. Clinicians, bombarded with non-critical alerts, begin to habituate to the noise.

The consequence is the “cry wolf” effect, where a critical, true-positive alert is dismissed along with the false ones. This is a workflow risk that directly impacts patient safety. In the United Kingdom, the National Institute for Health and Care Excellence (NICE) provides evidence standards for AI and digital health technologies. These standards increasingly require manufacturers to provide evidence of clinical utility within specific workflows, not just diagnostic accuracy. A system that generates high accuracy but causes alert fatigue may fail to demonstrate utility because it disrupts the clinical process.

Designing for safety requires a nuanced approach to threshold management. However, a static threshold is rarely sufficient. A more sophisticated approach involves “adaptive alerting,” where the system adjusts the urgency and presentation of alerts based on the user’s current context and historical interaction patterns. However, this introduces new regulatory challenges regarding transparency and predictability. If the system changes its behavior dynamically, how can the clinician trust its output? This tension between customization and consistency is a central challenge in the regulatory evaluation of high-risk AI.

Workflow Mismatch: The Integration Gap

One of the most pervasive sources of risk is the “workflow mismatch”—the misalignment between the AI’s operational logic and the actual, often chaotic, reality of clinical practice. AI developers, often working in clean lab environments, tend to design for idealized workflows. They assume data availability, data quality, and user attention that rarely exist in a busy emergency department or a general practitioner’s office.

Data Entry and Interoperability Friction

Many clinical AI systems require structured data inputs to function correctly. Yet, much of European healthcare data remains unstructured or siloed in legacy Electronic Health Record (EHR) systems. If an AI tool for predicting stroke risk requires specific laboratory values that are not routinely entered or are stored in non-standard formats, the clinician must manually input the data. This adds time to their workflow and increases the likelihood of input errors.

The European interoperability standards (e.g., the European Health Data Space – EHDS proposal) aim to solve this by enforcing common data formats. However, until full harmonization is achieved, AI systems must be robust against “missing data” scenarios. A system that fails gracefully (e.g., by providing a prediction based on available data with a confidence warning) is safer than one that refuses to run or provides a confident prediction based on incomplete inputs. The MDR requires that devices be designed to minimize errors in the event of foreseeable misuse. “Foreseeable misuse” in this context includes attempting to use the system with incomplete data sets.

The “Double-Data Entry” Problem

A specific workflow risk arises when AI tools are not natively integrated into the EHR but exist as standalone applications. This forces clinicians to toggle between windows, re-enter patient identifiers, or copy-paste data. This fragmentation disrupts the “flow state” required for complex decision-making. In France, the Haute Autorité de Santé (HAS) evaluates the “amélioration du service rendu” (improvement in service provided) for digital health tools. A tool that disrupts workflow to the extent that it increases administrative burden is unlikely to receive a favorable assessment, regardless of its predictive power.

From a safety perspective, the friction caused by poor integration increases the probability of selection errors (applying the AI to the wrong patient) or interpretation errors (misreading data because it is presented out of context). Therefore, regulatory submissions for high-risk AI systems in Europe must increasingly address integration architecture. It is not enough to prove the algorithm works; one must prove it works in situ.

Regulatory Frameworks and the Human Factor

The European regulatory framework is evolving to address these human factors explicitly. While the AI Act provides the overarching horizontal legislation for AI, and the MDR/IVDR governs medical devices, their intersection creates a specific compliance environment for clinical AI.

The AI Act: General Purpose AI vs. High-Risk Systems

It is crucial to distinguish between AI models that are “General Purpose” (GPAI) and those that are defined as “High-Risk.” A large language model used for drafting clinical notes might be a GPAI, but when deployed for diagnostic summarization, it likely falls into the High-Risk category (if it is a safety component of a medical device). The obligations for High-Risk systems are stringent.

Specifically, Article 14 of the AI Act mandates “Human Oversight.” This is not merely a suggestion; it is a legal requirement. The provider must design the system so that:

• The human overseer can understand the capacities and limitations of the AI.
• The human overseer can interpret the output and decide to ignore it.
• The system is designed to be “fully transparent” to the user.

In practice, this means that “black box” algorithms face significant scrutiny. If a clinician cannot understand why the AI flagged a patient as high-risk for sepsis, they cannot effectively exercise human oversight. This drives the demand for Explainable AI (XAI) in clinical settings. However, XAI itself introduces human factors risks. If an explanation is too complex, it overwhelms the user. If it is too simplistic, it may be misleading. The regulatory challenge is to define what constitutes an “understandable” explanation for a busy clinician.

Usability Engineering under MDR

Prior to the AI Act, the Medical Device Regulation (MDR) 2017/745 already covered software as a medical device (SaMD). The MDR places heavy emphasis on “Usability Engineering” as detailed in the harmonized standard EN ISO 62366-1. This standard requires manufacturers to conduct a “Use Specification” defining the user profile, the intended use, and the characteristics of the use environment.

Under the MDR, a failure to account for human factors is a defect in design. For example, if an AI diagnostic tool is intended for use by general practitioners (who may have less specialized training than radiologists), the interface must be designed for that specific user profile. If the system requires expert-level knowledge to interpret the output, the manufacturer has failed to design for the intended user. This is a strict liability concept in European regulation.

Comparing this to the UK’s MHRA approach, we see a similar focus. The MHRA’s “Software and AI as a Medical Device” change program emphasizes “Better Regulation,” which includes ensuring that software is safe and effective. The UK approach is currently exploring a “regulatory sandbox” to allow for more agile updates to AI software, acknowledging that AI improves iteratively. However, the core principle remains: the burden of proof for safety lies with the manufacturer, and safety includes the mitigation of human error.

The Role of Post-Market Surveillance (PMS)

Human factors risks often only become apparent after deployment. A system that looks perfect in a clinical trial may cause alert fatigue in the wild. The AI Act and MDR both strengthen requirements for Post-Market Surveillance (PMS). Providers of high-risk AI systems are required to implement a system for “logging” events.

For AI, this means logging not just technical errors (system crashes), but “performance drift” and “user interaction patterns.” For instance, if the logs show that clinicians are overriding the AI’s recommendations in 80% of cases, this is a signal of a workflow mismatch or a lack of trust. Under the AI Act, providers have a duty to report “serious incidents” to the national competent authorities. A “serious incident” could be defined as a situation where the AI’s interaction with the user led to a delay in treatment or a wrong diagnosis due to automation bias. This creates a feedback loop where real-world human factors data must inform regulatory compliance.

Designing for Safety: Mitigation Strategies

To navigate these regulatory and safety challenges, developers and healthcare institutions must adopt a “Human-in-the-Loop” (HITL) design philosophy that goes beyond token gestures. It requires embedding safety mechanisms directly into the system architecture.

Calibrated Confidence and Uncertainty Quantification

One of the most effective ways to combat overreliance is to ensure the AI never presents its output as absolute fact. Systems should be designed to output calibrated confidence intervals. Instead of stating “Malignancy detected,” the system should state “Malignancy detected with 75% confidence. Low confidence due to image artifact.” This prompts the clinician to engage their critical faculties. It transforms the AI from an oracle into a consultant. This aligns with the AI Act’s transparency requirements.

Adaptive User Interfaces

Static interfaces are insufficient for dynamic clinical environments. Future-proof systems should employ adaptive interfaces that modulate the volume of information based on context. In a triage scenario (high stress, time pressure), the interface should present only critical alerts. In a review scenario (lower stress, more time), the interface can present deeper analytics and comparative data. This reduces cognitive load when it is highest and reduces alert fatigue by filtering noise.

Simulation-Based Validation

Traditional validation focuses on the algorithm’s output (e.g., sensitivity/specificity). A human-factors-informed validation process must include simulation-based testing. This involves placing the AI tool into a simulated clinical workflow with representative users (e.g., using high-fidelity mannequins or digital twins of patients) and measuring not just diagnostic accuracy, but time-to-action, error rates in data entry, and subjective workload scores (e.g., NASA-TLX). Regulatory submissions that include data from such simulations demonstrate a higher level of maturity and safety assurance.

Comparative European Approaches to AI Oversight

While the EU regulations provide a harmonized baseline, national implementation varies, creating a complex patchwork for developers.

Germany: As the largest market, Germany’s Digital Healthcare Act (DVG) has created a fast-track for digital health applications to be prescribed by doctors. However, the Federal Institute for Drugs and Medical Devices (BfArM) requires rigorous evidence of “positive healthcare effects.” This often translates to a requirement that the AI does not just diagnose, but improves the overall treatment pathway. German regulators are particularly sensitive to workflow integration; a standalone app that is not integrated into the certified Telematics Infrastructure may struggle to gain traction.

France: The Haute Autorité de Santé (HAS) is known for its rigorous evaluation of medical necessity. In the context of AI, HAS focuses heavily on the “clinical added value.” They scrutinize whether the AI reduces the workload of healthcare professionals or improves diagnostic precision. If an AI tool increases the cognitive load or disrupts workflow (e.g., by requiring excessive manual validation), HAS may deem it to have no added value, effectively blocking reimbursement.

The Netherlands: The Dutch approach is often characterized by a pragmatic focus on implementation. The National Health Care Institute (Zorginstituut Nederland) evaluates whether a technology fits within the basic insurance package. They often look at “real-world evidence” (RWE) earlier in the process than other jurisdictions. This means that post-market human factors data is crucial for long-term viability in the Dutch market.

These differences highlight that a “one-size-fits-all” deployment strategy is risky. A system designed for the French market must emphasize clinical precision and cost-effectiveness, while a system for the German market must emphasize integration into the national digital infrastructure and patient benefit.

The Future of Human-AI Collaboration in European Healthcare

The trajectory of clinical AI is moving toward “Augmented Intelligence”—systems designed to amplify human intelligence rather than replace it. This requires a fundamental rethinking of the user experience (UX) as a safety-critical component, not an aesthetic one.

As we look toward the full implementation of the AI Act (phased in over 2024-2026), the role of the “Human Factors Specialist” will become as critical as the Data Scientist in AI development teams. Regulatory audits will increasingly ask: “How have you tested this system with real users in real environments? How do you prevent the user from making mistakes? How do you ensure the user remains the decision-maker?”

For healthcare providers, the responsibility is twofold. First, they must select AI systems that demonstrate compliance with these human-centric design standards. Second, they must train their staff to be “AI-literate.” Clinicians must understand that AI systems have limitations and are prone to specific types of errors (such as brittleness when faced with out-of-distribution data). The concept of “trust” in AI must be replaced with “calibrated trust”—a healthy skepticism combined with an appreciation for the tool’s utility.

In conclusion, the safe deployment of clinical AI in Europe is not a problem that can be solved by better algorithms alone. It is a problem of socio-technical design. It requires aligning the cold logic of machine learning with the warm, messy, and variable reality of human clinical practice. By adhering to the principles of usability engineering, embracing the transparency requirements of the AI Act, and rigorously monitoring human-AI interactions in post-market surveillance, European stakeholders can ensure that these powerful tools fulfill their promise without introducing unacceptable risks to patient safety.