The deployment phase of FUTURE-AI framework focuses on implementing and maintaining AI tools in real-world clinical settings through eight key recommendations that emphasize local validation, quality control, monitoring, and governance. The following table provides detailed operations and examples for evaluating local clinical validity (Universality 4), implementing quality control mechanisms (Traceability 3-6), providing user training (Usability 3), and ensuring regulatory compliance (General 5), with most recommendations being highly recommended for deployable tools compared to research applications, reflecting the critical importance of robust implementation and oversight in real-world healthcare settings.
Recommendations | Operations | Examples |
---|---|---|
Evaluate and demonstrate local clinical validity (universality 4) | Test AI model using local data | Data from local clinical registry |
Identify factors that could affect AI tool’s local validity | Local operators, equipment, clinical workflows, acquisition protocols | |
Assess AI tool’s integration within local clinical workflows | AI tool’s interface aligns with hospital IT system or disrupts routine practice | |
Assess AI tool’s local practical utility and identify any operational challenges | Time to operate, clinician satisfaction, disruption of existing operations | |
Implement adjustments for local validity | Model calibration, fine-tuning, transfer learning | |
Compare performance of AI tool with that of local clinicians | Side-by-side comparison, in silico trial | |
Define mechanisms for quality control of AI inputs and outputs (traceability 3) | Implement mechanisms to identify erroneous input data | Missing value or out-of-distribution detector, automated image quality assessment |
Implement mechanisms to detect implausible AI outputs | Postprocessing sanity checks, anomaly detection algorithm | |
Provide calibrated uncertainty estimates to inform on AI tool’s confidence | Calibrated uncertainty estimates per patient or data point | |
Implement system for continuous quality monitoring | Real time dashboard tracking data quality and performance metrics | |
Implement feedback mechanism for users to report issues | Feedback portal enabling clinicians to report discrepancies or anomalies | |
Implement system for periodic auditing and updating (traceability 4) | Define schedule for periodic audits | Biannual or annual |
Define audit criteria and metrics | Accuracy, consistency, fairness, data security | |
Define datasets for periodic audits | Newly acquired prospective dataset from local hospital | |
Implement mechanisms to detect data or concept drifts | Detecting shifts in input data distributions | |
Assign role of auditor(s) for AI tool | Internal auditing team, third party company | |
Update AI tool based on audit results | Updating AI model, re-evaluating AI model, adjusting operational protocols, continuous learning | |
Implement reporting system from audits and subsequent updates | Automatic sharing of detailed reports to healthcare managers and clinicians | |
Monitor impact of AI updates | Impact on system performance and user satisfaction | |
Implement logging system for usage recording (traceability 5) | Implement logging framework capturing all interactions | User actions, AI inputs, AI outputs, clinical decisions |
Define data to be logged | Timestamp, user ID, patient ID (anonymised), action details, results | |
Implement mechanisms for data capture | Software to automatically record every data and operation | |
Implement mechanisms for data security | Encrypted log files, privacy preserving techniques | |
Provide access to logs for auditing and troubleshooting | By defining authorised personnel, eg, healthcare or IT managers | |
Implement mechanism for end users to log any issues | A user interface to enter information about operational anomalies | |
Implement log analysis | Time series statistics and visualisations to detect unusual activities and alert administrators | |
Provide training (usability 3) | Create user manuals | User instructions, capabilities, limitations, troubleshooting steps, examples, and case studies |
Develop training materials and activities | Online courses, workshops, hands-on sessions | |
Use formats and languages accessible to intended end users | Multiple formats (text, video, audio) and languages (English, Chinese, Swahili) | |
Customise training to all end user groups | Role specific modules for specialists, nurses, and patients | |
Include training to enhance AI and health literacy | On application specific AI concepts (eg, radiomics, explainability), AI driven clinical decision making | |
Identify and comply with applicable AI regulatory requirements (general 5) | Engage regulatory experts to investigate regulatory requirements | Regulatory consultants from intended local settings |
Identify specific regulations based on AI tool’s intended markets | FDA’s SaMD in the United States, MDR and AI Act in the EU | |
Identify specific requirements based on AI tool’s purpose | De Novo classification (Class III) | |
Define list of milestones towards regulatory compliance | MDR certification: technical verification, pivotal clinical trial, risk and quality management, postmarket follow-up | |
Establish mechanisms for AI governance (traceability 6) | Assign roles for AI tool’s governance | For periodic auditing, maintenance, supervision (eg, healthcare manager) |
Define responsibilities for AI related errors | Responsibilities of clinicians, healthcare centres, AI developers, and manufacturers | |
Define mechanisms for accountability | Individual v collective accountability/liability, compensations, support for patients |