The deployment phase of FUTURE-AI framework focuses on implementing and maintaining AI tools in real-world clinical settings through eight key recommendations that emphasize local validation, quality control, monitoring, and governance. Table 6 provides detailed operations and examples for evaluating local clinical validity (universality 4), implementing quality control mechanisms (traceability 3-6), providing user training (usability 3), and ensuring regulatory compliance (general 5), with most recommendations being highly recommended (++) for deployable tools compared to research applications (+), reflecting the critical importance of robust implementation and oversight in real-world healthcare settings.
| Recommendations | Operations | Examples |
|---|---|---|
| Evaluate and demonstrate local clinical validity (universality 4) | Test AI model using local data | Data from local clinical registry |
| Identify factors that could affect AI tool’s local validity | Local operators, equipment, clinical workflows, acquisition protocols | |
| Assess AI tool’s integration within local clinical workflows | AI tool’s interface aligns with hospital IT system or disrupts routine practice | |
| Assess AI tool’s local practical utility and identify any operational challenges | Time to operate, clinician satisfaction, disruption of existing operations | |
| Implement adjustments for local validity | Model calibration, fine-tuning, transfer learning | |
| Compare performance of AI tool with that of local clinicians | Side-by-side comparison, in silico trial | |
| Define mechanisms for quality control of AI inputs and outputs (traceability 3) | Implement mechanisms to identify erroneous input data | Missing value or out-of-distribution detector, automated image quality assessment |
| Implement mechanisms to detect implausible AI outputs | Postprocessing sanity checks, anomaly detection algorithm | |
| Provide calibrated uncertainty estimates to inform on AI tool’s confidence | Calibrated uncertainty estimates per patient or data point | |
| Implement system for continuous quality monitoring | Real time dashboard tracking data quality and performance metrics | |
| Implement feedback mechanism for users to report issues | Feedback portal enabling clinicians to report discrepancies or anomalies | |
| Implement system for periodic auditing and updating (traceability 4) | Define schedule for periodic audits | Biannual or annual |
| Define audit criteria and metrics | Accuracy, consistency, fairness, data security | |
| Define datasets for periodic audits | Newly acquired prospective dataset from local hospital | |
| Implement mechanisms to detect data or concept drifts | Detecting shifts in input data distributions | |
| Assign role of auditor(s) for AI tool | Internal auditing team, third party company | |
| Update AI tool based on audit results | Updating AI model, re-evaluating AI model, adjusting operational protocols, continuous learning | |
| Implement reporting system from audits and subsequent updates | Automatic sharing of detailed reports to healthcare managers and clinicians | |
| Monitor impact of AI updates | Impact on system performance and user satisfaction | |
| Implement logging system for usage recording (traceability 5) | Implement logging framework capturing all interactions | User actions, AI inputs, AI outputs, clinical decisions |
| Define data to be logged | Timestamp, user ID, patient ID (anonymised), action details, results | |
| Implement mechanisms for data capture | Software to automatically record every data and operation | |
| Implement mechanisms for data security | Encrypted log files, privacy preserving techniques | |
| Provide access to logs for auditing and troubleshooting | By defining authorised personnel, eg, healthcare or IT managers | |
| Implement mechanism for end users to log any issues | A user interface to enter information about operational anomalies | |
| Implement log analysis | Time series statistics and visualisations to detect unusual activities and alert administrators | |
| Provide training (usability 3) | Create user manuals | User instructions, capabilities, limitations, troubleshooting steps, examples, and case studies |
| Develop training materials and activities | Online courses, workshops, hands-on sessions | |
| Use formats and languages accessible to intended end users | Multiple formats (text, video, audio) and languages (English, Chinese, Swahili) | |
| Customise training to all end user groups | Role specific modules for specialists, nurses, and patients | |
| Include training to enhance AI and health literacy | On application specific AI concepts (eg, radiomics, explainability), AI driven clinical decision making | |
| Identify and comply with applicable AI regulatory requirements (general 5) | Engage regulatory experts to investigate regulatory requirements | Regulatory consultants from intended local settings |
| Identify specific regulations based on AI tool’s intended markets | FDA’s SaMD in the United States, MDR and AI Act in the EU | |
| Identify specific requirements based on AI tool’s purpose | De Novo classification (Class III) | |
| Define list of milestones towards regulatory compliance | MDR certification: technical verification, pivotal clinical trial, risk and quality management, postmarket follow-up | |
| Establish mechanisms for AI governance (traceability 6) | Assign roles for AI tool’s governance | For periodic auditing, maintenance, supervision (eg, healthcare manager) |
| Define responsibilities for AI related errors | Responsibilities of clinicians, healthcare centres, AI developers, and manufacturers | |
| Define mechanisms for accountability | Individual v collective accountability/liability, compensations, support for patients |