Availability of evidence for predictive machine learning algorithms in primary care: a systematic review
AbstractImportance. The aging and multimorbid population and health personnel shortages pose a substantial burden on primary health care. While predictive machine learning (ML) algorithms have the potential to address these challenges, concerns include transparency and insufficient reporting of model validation and effectiveness of the implementation in the clinical workflow.
Objectives. To systematically identify predictive ML algorithms implemented in primary care from peer-reviewed literature and US Food and Drug Administration (FDA) and Conformité Européene (CE) registration databases and to ascertain the public availability of evidence, including peer-reviewed literature, gray literature, and technical reports across the artificial intelligence (AI) life cycle.
Evidence review. PubMed, Embase, Web of Science, Cochrane Library, Emcare, Academic Search Premier, IEEE Xplore, ACM Digital Library, MathSciNet, AAAI.org (Association for the Advancement of Artificial Intelligence), arXiv, Epistemonikos, PsycINFO, and Google Scholar were searched for studies published between January 2000 and July 2023, with search terms that were related to AI, primary care, and implementation. The search extended to CE-marked or FDA-approved predictive ML algorithms obtained from relevant registration databases. Three reviewers gathered subsequent evidence involving strategies such as product searches, exploration of references, manufacturer website visits, and direct inquiries to authors and product owners. The extent to which the evidence for each predictive ML algorithm aligned with the Dutch AI predictive algorithm (AIPA) guideline requirements was assessed per AI life cycle phase, producing evidence availability scores.
Findings. The systematic search identified 43 predictive ML algorithms, of which 25 were commercially available and CE-marked or FDA-approved. The predictive ML algorithms spanned multiple clinical domains, but most [27 (63%)] focused on cardiovascular diseases and diabetes. Most [35 (81%)] were published within the past 5 years. The availability of evidence varied across different phases of the predictive ML algorithm life cycle, with evidence being reported the least for phase 1 (preparation) and phase 5 (impact assessment) (19 and 30%, respectively). Twelve (28%) predictive ML algorithms achieved approximately half of their maximum individual evidence availability score. Overall, predictive ML algorithms from peer-reviewed literature showed higher evidence availability compared with those from FDA-approved or CE-marked databases (45 vs 29%).
Conclusions and relevance. The findings indicate an urgent need to improve the availability of evidence regarding the predictive ML algorithms’ quality criteria. Adopting the Dutch AIPA guideline could facilitate transparent and consistent reporting of the quality criteria that could foster trust among end users and facilitating large-scale implementation.
Key points
Question. Which machine learning (ML) predictive algorithms have been implemented in primary care, and what evidence is publicly available for supporting their quality?
Findings. In this systematic review of 43 predictive ML algorithms in primary care from scientific literature and the registration databases of the US Food and Drug Administration and Conformité Européene, there was limited publicly available evidence across all artificial intelligence life cycle phases from development to implementation. While the development phase (phase 2) was most frequently reported, most predictive ML algorithms did not meet half of the predefined requirements of the Dutch artificial intelligence predictive algorithm guideline.
Meaning. Findings of this study underscore the urgent need to facilitate transparent and consistent reporting of the quality criteria in literature, which could build trust among end users and facilitate large-scale implementation.
Rakers M.M., van Buchem M.M., Kucenko S., de Hond A., Kant I., van Smeden M., Moons K.G.M., Leeuwenberg A.M., Chavannes N., Villalobos-Quesada M., van Os H.J.A. Availability of evidence for predictive machine learning algorithms in primary care: a systematic review. JAMA Netw Open. 2024; 7 (9): e2432990. DOI: 10.1001/jamanetworkopen.2024.32990 PMID: 39264624. PMCID: PMC11393722.
Литература/References
- Boerma W., Bourgueil Y., Cartier T., et al. Overview and future challenges for primary care. 2015. URL: https://www.ncbi.nlm.nih.gov/books/NBK458729/ (date of access October 20, 2023).
- Smeets H.M., Kortekaas M.F., Rutten F.H., et al. Routine primary care data for scientific research, quality of care programs and educational purposes: the Julius General Practitioners’ Network (JGPN). BMC Health Serv Res. 2018; 18 (1): 735. DOI: https://doi.org/10.1186/s12913-018-3528-5
- Kuiper J.G., Bakker M., Penning-van Beest F.J.A., Herings R.M.C. Existing data sources for clinical epidemiology: the PHARMO Database Network. Clin Epidemiol. 2020; 12: 415–22. DOI: https://doi.org/10.2147/CLEP.S247575
- Shilo S., Rossman H., Segal E. Axes of a revolution: challenges and promises of big data in healthcare. Nat Med. 2020; 26 (1): 29–38. DOI: https://doi.org/10.1038/s41591-019-0727-5
- Moons K.G.M., Kengne A.P., Woodward M., et al. Risk prediction models: I. development, internal validation, and assessing the incremental value of a new (bio)marker. Heart. 2012; 98 (9): 683–90. DOI: https://doi.org/10.1136/heartjnl-2011-301246
- Babel A., Taneja R., Mondello Malvestiti F., Monaco A., Donde S. Artificial intelligence solutions to increase medication adherence in patients with non-communicable diseases. Front Digit Health. 2021; 3: 669869. DOI: https://doi.org/10.3389/fdgth.2021.669869
- Hazarika I. Artificial intelligence: opportunities and implications for the health workforce. Int Health. 2020; 12 (4): 241–5. DOI: https://doi.org/10.1093/inthealth/ihaa007
- Liyanage H., Liaw S.T., Jonnagaddala J., et al. Artificial intelligence in primary health care: perceptions, issues, and challenges. Yearb Med Inform. 2019; 28 (1): 41–6. DOI: https://doi.org/10.1055/s-0039-1677901
- Andaur Navarro C.L., Damen J.A.A., Takada T., et al. Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review. BMJ. 2021; 375: n2281. DOI: https://doi.org/10.1136/bmj.n2281
- Shaw J., Rudzicz F., Jamieson T., Goldfarb A. Artificial intelligence and the implementation challenge. J Med Internet Res. 2019; 21 (7): e13659. DOI: https://doi.org/10.2196/13659
- Norori N., Hu Q., Aellen F.M., Faraci F.D., Tzovara A. Addressing bias in big data and AI for health care: a call for open science. Patterns (N Y). 2021; 2 (10): 100347. DOI: https://doi.org/10.1016/j.patter.2021.100347
- van Leeuwen K.G., Schalekamp S., Rutten M.J.C.M., van Ginneken B., de Rooij M. Artificial intelligence in radiology: 100 commercially available products and their scientific evidence. Eur Radiol. 2021; 31 (6): 3797–804. DOI: https://doi.org/10.1007/s00330-021-07892-z
- Andaur Navarro C.L., Damen J.A.A., Takada T., et al. Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review. BMC Med Res Methodol. 2022; 22 (1): 12. DOI: https://doi.org/10.1186/s12874-021-01469-6
- Daneshjou R., Smith M.P., Sun M.D., Rotemberg V., Zou J. Lack of transparency and potential bias in artificial intelligence data sets and algorithms: a scoping review. JAMA Dermatol. 2021; 157 (11): 1362–9. DOI: https://doi.org/10.1001/jamadermatol.2021.3129
- de Hond A.A.H., Leeuwenberg A.M., Hooft L., et al. Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review. NPJ Digit Med. 2022; 5 (1): 2. DOI: https://doi.org/10.1038/s41746-021-00549-7
- van Smeden M., Moons K.G., Hooft L., Chavannes N.H., van Os H.J., Kant I. Guideline for high-quality diagnostic and prognostic applications of AI in healthcare. OSFHome, November 14, 2022. URL: http://OSF.IO/TNRJZ (date of access August 6, 2024).
- Page M.J., McKenzie J.E., Bossuyt P.M., et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Syst Rev. 2021; 10 (1) :89. DOI: https://doi.org/10.1186/s13643-021-01626-4
- Muehlematter U.J., Daniore P., Vokinger K.N. Approval of artificial intelligence and machine learning-based medical devices in the USA and Europe (2015–20): a comparative analysis. Lancet Digit Health. 2021; 3 (3): e195–203. DOI: https://doi.org/10.1016/S2589-7500(20)30292-2
- Zhu S., Gilbert M., Chetty I., Siddiqui F. The 2021 landscape of FDA-approved artificial intelligence/machine learning-enabled medical devices: an analysis of the characteristics and intended use. Int J Med Inform. 2022; 165: 104828. DOI: https://doi.org/10.1016/j.ijmedinf.2022.104828
- US Food and Drug Administration. Artificial intelligence and machine learning (AI/ML)-enabled medical devices. URL: https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices#resources (date of access August 23, 2023).
- Rifkin S.B. Alma Ata after 40 years: primary health care and health for all-from consensus to complexity. BMJ Glob Health. 2018; 3 (suppl 3): e001188. DOI: https://doi.org/10.1136/bmjgh-2018-001188
- Gama F., Tyskbo D., Nygren J., Barlow J., Reed J., Svedberg P. Implementation frameworks for artificial intelligence translation into health care practice: scoping review. J Med Internet Res. 2022; 24 (1): e32215. DOI: https://doi.org/10.2196/32215
- Tenhunen H., Hirvonen P., Linna M., Halminen O., Hörhammer I. Intelligent patient flow management system at a primary healthcare center – the effect on service use and costs. Stud Health Technol Inform. 2018; 255: 142–6.
- Liu J., Gibson E., Ramchal S., et al. Diabetic retinopathy screening with automated retinal image analysis in a primary care setting improves adherence to ophthalmic care. Ophthalmol Retina. 2021; 5 (1): 71–7. DOI: https://doi.org/10.1016/j.oret.2020.06.016
- Eyenuk, Inc. Harnessing deep learning to prevent blindness. URL: https://www.eyenuk.com/en/ (date of access August 5, 2024).
- Bachtiger P., Petri C.F., Scott F.E., et al. Point-of-care screening for heart failure with reduced ejection fraction using artificial intelligence during ECG-enabled stethoscope examination in London, UK: a prospective, observational, multicentre study. Lancet Digit Health. 2022; 4 (2): e117–25. DOI: https://doi.org/10.1016/S2589-7500(21)00256-9
- EKO. Unlock AI Murmur & AFib Detection with Eko+. URL: https://www.ekohealth.com/ (date of access August 5, 2024).
- Hill N.R., Arden C., Beresford-Hulme L., et al. Identification of undiagnosed atrial fibrillation patients using a machine learning risk prediction algorithm and diagnostic testing (PULsE-AI): study protocol fora randomised controlled trial. Contemp Clin Trials. 2020; 99: 106191. DOI: https://doi.org/10.1016/j.cct.2020.106191
- Herter W.E., Khuc J., Cinà G., et al. Impact of a machine learning-based decision support system for urinary tract infections: prospective observational study in 36 primary care practices. JMIR Med Inform. 2022; 10 (5): e27795. DOI: https://doi.org/10.2196/27795
- Bhatt S., Cohon A., Rose J., et al. Interpretable machine learning models for clinical decision-making in a high- need, value-based primary care setting. NEJM Catal Innov Care Deliv. 2021; 2 (4). DOI: https://doi.org/10.1056/CAT.21.0008
- Herman B., Sirichokchatchawan W., Nantasenamat C., Pongpanich S. Artificial intelligence in overcoming rifampicin resistant-screening challenges in Indonesia: a qualitative study on the user experience of CUHAS-ROBUST. J Health Res. 2021; 36 (6): 1018–27. DOI: https://doi.org/10.1108/JHR-11-2020-0535
- Wang S.V., Rogers J.R., Jin Y., et al. Stepped-wedge randomised trial to evaluate population health intervention designed to increase appropriate anticoagulation in patients with atrial fibrillation. BMJ Qual Saf. 2019; 28 (10): 835–42. DOI: https://doi.org/10.1136/bmjqs-2019-009367
- Chiang P.H., Wong M., Dey S. Using wearables and machine learning to enable personalized lifestyle recommendations to improve blood pressure. IEEE J Transl Eng Health Med. 2021; 9: 2700513. DOI: https://doi.org/10.1109/JTEHM.2021.3098173
- Yao X., Rushlow D.R., Inselman J.W., et al. Artificial intelligence-enabled electrocardiograms for identification of patients with low ejection fraction: a pragmatic, randomized clinical trial. Nat Med. 2021; 27 (5): 815–9. DOI: https://doi.org/10. 1038/s41591-021-01335-4
- Jaremko J.L., Hareendranathan A., Ehsan S., et al. AI aided workflow for hip dysplasia screening using ultrasound in primary care clinics. Sci Rep. 2023; 13 (1): 9224. DOI: https://doi.org/10.1038/s41598-023-35603-9
- Escalé-Besa A., Fuster-Casanovas A., Börve A., et al. Using artificial intelligence as a diagnostic decision support tool in skin disease: protocol for an observational prospective cohort study. JMIR Res Protoc. 2022; 11 (8): e37531. DOI: https://doi.org/10.2196/37531
- TytoCare. URL: https://www.tytocare.com/ (date of access August 5, 2024).
- Peerbridge Health. Home. URL: https://peerbridgehealth.com/ (date of access August 5, 2024).
- Rooti Labs Limited. RootiCare: dependable, continuous montioring. URL: https://www.rootilabs.com/doctor (date of access August 5, 2024).
- Digital Diagnostics. LumineticsCore. URL: https://www.digitaldiagnostics.com/products/eye-disease/idx-dr/ (date of access August 5, 2024).
- FibriCheck. Advanced monitoring of your heart rhythm for detection and treatment of atrial fibrillation. URL: https://www.fibricheck.com/nl/ (date of access August 5, 2024).
- Cardio-Phoenix. Cardio-HART. URL: https://www.cardiophoenix.com/ (date of access August 5, 2024).
- eMURMUR. Join the world’s first enterprise-level, open platform for advanced digital auscultation. URL: https://emurmur.com/ (date of access August 5, 2024).
- Minttihealth. Home. URL: https://minttihealth.com/ (date of access August 5, 2024).
- BioIntelliSense, Inc. BioIntelliSence. URL: https://biointellisense.com/ (date of access August 5, 2024).
- EchoNous Inc. EchoNous. URL: https://echonous.com/ (date of access August 5, 2024).
- Coala. COALA heart monitoring system. URL: https://www.coalalife.com/us/ (date of access August 5, 2024).
- My mhealth. Empowering patients to manage their asthma fora lifetime. URL: https://mymhealth.com/myasthma (date of access August 5, 2024).
- eMed. eMed weight loss programme. URL: https://www.emed.com/uk (date of access August 5, 2024).
- Huma. Longer, fuller lives with digtal-first care and research. URL: https://medopad.com/ (date of access August 8, 2024).
- Skin Analytics. Skin analytics. URL: https://skin-analytics.com/ (date of access August 5, 2024).
- ResApp Health. ResAppDx-EU. URL: https://digitalhealth.org.au/wp-content/uploads/2020/06/ResAppDx-EU-flyer.pdf (date of access August 8, 2024).
- MobileODT. Automated Visual Evaluation (AVE) explained: everything you need to know about the new AI for cervical cancer screening. January 14, 2019. URL: https://www.mobileodt.com/blog/everything-you-need-to-know-about-ave-automated-visual-examination-for-cervical-cancer-screening/ (date of access June 29, 2022).
- Kata. Inhale correctly, live better. URL: https://kata-inhalation.com/en/ (date of access August 5, 2024).
- SkinVision. Skin cancer melanoma tracking app. URL: https://www.skinvision.com/nl/ (date of access August 5, 2024).
- Medicalgorithmics. The most effective technology solutions for cardiology. URL: https://www.medicalgorithmics.com/ (date of access August 25, 2023).
- Apple. IRN Global 2.0. instructions for use. 2021. URL: https://www.apple.com/legal/ifu/irnf/2-0/irn-2-0-en_US.pdf (date of access August 25, 2023).
- Benrimoh D., Tanguay-Sela M., Perlman K., et al. Using a simulation centre to evaluate preliminary acceptability and impact of an artificial intelligence-powered clinical decision support system for depression treatment on the physician-patient interaction. BJPsych Open. 2021; 7 (1): e22. DOI: https://doi.org/10.1192/bjo.2020.127
- Healthy.io Ltd. Increase ACR testing by up to 50%. URL: https://healthy.io/services/kidney/ (date of access August 5, 2024).
- Zio by iRhythm Technologies, Inc. iRhythm gains FDA clearance for its clinically integrated ZEUS system. July 22, 2022. URL: https://www.irhythmtech.com/company/news/irhythm-gains-fda-clearance-for-its-clinically-integrated-zeus-system (date of access August 25, 2023).
- Breitbart E.W., Choudhury K., Andersen A.D., et al. Improved patient satisfaction and diagnostic accuracy in skin diseases with a visual clinical decision support system-a feasibility study with general practitioners. PLoS One. 2020; 15 (7): e0235410. DOI: https://doi.org/10.1371/journal.pone.0235410
- Kanagasingam Y., Xiao D., Vignarajan J., Preetham A., Tay-Kearney M.L., Mehrotra A. Evaluation of artificial intelligence-based grading of diabetic retinopathy in primary care. JAMA Netw Open. 2018; 1 (5): e182665. DOI: https://doi.org/10.1001/jamanetworkopen.2018.2665
- Long J., Yuan M.J., Poonawala R. An observational study to evaluate the usability and intent to adopt an artificial intelligence-powered medication reconciliation tool. Interact J Med Res. 2016; 5 (2): e14. DOI: https://doi.org/10.2196/ijmr.5462
- Romero-Brufau S., Wyatt K.D., Boyum P., Mickelson M., Moore M., Cognetta-Rieke C. A lesson in implementation: a pre-post study of providers’ experience with artificial intelligence-based clinical decision support. Int J Med Inform. 2020; 137: 104072. DOI: https://doi.org/10.1016/j.ijmedinf.2019.104072
- Seol H.Y., Shrestha P., Muth J.F., et al. Artificial intelligence-assisted clinical decision support for childhood asthma management: a randomized clinical trial. PLoS One. 2021; 16 (8): e0255261. DOI: https://doi.org/10.1371/journal.pone.0255261
- Frontoni E., Romeo L., Bernardini M., et al. A decision support system for diabetes chronic care models based on general practitioner engagement and EHR data sharing. IEEE J Transl Eng Health Med. 2020; 8: 3000112. DOI: https://doi.org/10. 1109/JTEHM.2020.3031107
- Escalé-Besa A., Yélamos O., Vidal-Alaball J., et al. Exploring the potential of artificial intelligence in improving skin lesion diagnosis in primary care. Sci Rep. 2023; 13 (1): 4293. DOI: https://doi.org/10.1038/s41598-023-31340-1
- VivaQuant. Introducing the world’s smallest one-piece MCT: RX-1 mini. URL: https://rhythmexpressecg.com/ (date of access August 5, 2024).
- Kaia Health. Digitale therapien bei COPD und rückenschmerzen. URL: https://kaiahealth.de/ (date of access June 29, 2022).
- Zuckerman D., Brown P., Das A. Lack of publicly available scientific evidence on the safety and effectiveness of implanted medical devices. JAMA Intern Med. 2014; 174 (11): 1781–7. DOI: https://doi.org/10.1001/jamainternmed.2014.4193
- Andaur Navarro C.L., Damen J.A.A., van Smeden M., et al. Systematic review identifies the design and methodological conduct of studies on machine learning-based prediction models. J Clin Epidemiol. 2023; 154: 8–22. DOI: https://doi.org/10.1016/j.jclinepi.2022.11.015
- Lu J.H., Callahan A., Patel B.S., et al. Assessment of adherence to reporting guidelines by commonly used clinical prediction models from a single vendor: a systematic review. JAMA Netw Open. 2022; 5 (8): e2227779. DOI: https://doi.org/10.1001/jamanetworkopen.2022.27779
- Lin S.Y., Mahoney M.R., Sinsky C.A. Ten ways artificial intelligence will transform primary care. J Gen Intern Med. 2019; 34 (8): 1626–30. DOI: https://doi.org/10.1007/s11606-019-05035-1
- Gerke S., Minssen T., Cohen G. Ethical and legal challenges of artificial intelligence-driven healthcare. In: Artificial Intelligence in Healthcare. 2020: 295–336. DOI: https://doi.org/10.1016/B978-0-12-818438-7.00012-5
- Fraser A.G., Nelissen R.G.H.H., Kjaersgaard-Andersen P., Szymański P., Melvin T., Piscoi P.; CORE-MD Investigators. Improved clinical investigation and evaluation of high-risk medical devices: the rationale and objectives of CORE-MD (Coordinating Research and Evidence for Medical Devices). EFORT Open Rev. 2021; 6 (10): 839–49. DOI: https://doi.org/10.1302/2058-5241.6.210081
- US Food and Drug Administration. Fostering transparency to improve public health. URL: https://www.fda.gov/news-events/speeches-fda-officials/fostering-transparency-improve-public-health (date of access May 15, 2023).
- US Food and Drug Administration. Public access to results of FDA-funded research. URL: https://www.fda.gov/science-research/about-science-research-fda/public-access-results-fda-funded-scientific-research (date of access August 8, 2024).
- Wu E., Wu K., Daneshjou R., Ouyang D., Ho D.E., Zou J. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nat Med. 2021; 27 (4): 582–4. DOI: https://doi.org/10.1038/s41591-021-01312-x
- MDR-Eudamed. Welcome to EUDAMED. URL: https://webgate.ec.europa.eu/eudamed/landing-page#/ (date of access September 7, 2023).
- Gasser U. An EU landmark for AI governance. Science. 2023; 380 (6651): 1203. DOI: https://doi.org/10.1126/science. adj1627
- Markus A.F., Kors J.A., Rijnbeek P.R. The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies. J Biomed Inform. 2021; 113: 103655. DOI: https://doi.org/10.1016/j.jbi.2020.103655
- Zerilli J., Bhatt U., Weller A. How transparency modulates trust in artificial intelligence. Patterns (N Y). 2022; 3 (4): 100455. DOI: https://doi.org/10.1016/j.patter.2022.100455
- Kordzadeh N., Ghasemaghaei M. Algorithmic bias: review, synthesis, and future research directions. Eur J Inf Syst. 2022; 31 (3): 388–409. DOI: https://doi.org/10.1080/0960085X.2021.1927212
- Smale N., Unsworth K., Denyer G., Barr D. A review of the history, advocacy and efficacy of data management plans. Int J Digit Curation. 2020; 15 (1): 30–58. DOI: https://doi.org/10.2218/ijdc.v15i1.525
- Michener W.K. Ten simple rules for creating a good data management plan. PLoS Comput Biol. 2015; 11 (10): e1004525. DOI: https://doi.org/10.1371/journal.pcbi.1004525
- Williams M., Bagwell J., Nahm Zozus M. Data management plans: the missing perspective. J Biomed Inform. 2017; 71: 130–42. DOI: https://doi.org/10.1016/j.jbi.2017.05.004
- Wilkinson M.D., Dumontier M., Aalbersberg I.J., et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data. 2016; 3: 160018. DOI: https://doi.org/10.1038/sdata.2016.18
- Kanza S., Knight N.J. Behind every great research project is great data management. BMC Res Notes. 2022; 15 (1): 20. DOI: https://doi.org/10.1186/s13104-022-05908-5
- European Commission. H2020 Programme Guidelines on FAIR Data Management in Horizon 2020. 2016. URL: https://ec.europa.eu/research/participants/data/ref/h2020/grants_manual/hi/oa_pilot/h2020-hi-oa-data-mgt_en.pdf (date of access March 20, 2023).
- Matheny M., Israni S.T., Ahmed M., Whicher D. (eds). Artificial Intelligence in Health Care: The Hope, the Hype, the Promise, the Peril. The National Academies Press, 2019. DOI: https://doi.org/10.17226/27111
- Terry A.L., Kueper J.K., Beleno R., et al. Is primary health care ready for artificial intelligence? What do primary health care stakeholders say? BMC Med Inform Decis Mak. 2022; 22 (1): 237. DOI: https://doi.org/10.1186/s12911-022-01984-6
- Morrison A., Polisena J., Husereau D., et al. The effect of English-language restriction on systematic review- based meta-analyses: a systematic review of empirical studies. Int J Technol Assess Health Care. 2012; 28 (2): 138–44. DOI: https://doi.org/10.1017/S0266462312000086
- Collins G.S., Moons K.G.M. Reporting of artificial intelligence prediction models. Lancet. 2019; 393 (10 181): 1577–9. DOI: https://doi.org/10.1016/S0140-6736(19)30037-6
- Vasey B., Nagendran M., Campbell B., et al.; DECIDE-AI Expert Group. Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. BMJ. 2022; 377: e070904. DOI: https://doi.org/10.1136/bmj-2022-070904
- Liu X., Cruz Rivera S., Moher D., Calvert M.J., Denniston A.K.; SPIRIT-AI and CONSORT-AI Working Group. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. BMJ. 2020; 370: m3164. DOI: https://doi.org/10.1136/bmj.m3164
- Eldridge S.M., Chan C.L., Campbell M.J., et al.; PAFS Consensus Group. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. BMJ. 2016; 355: i5239. DOI: https://doi.org/10.1136/bmj.i5239
- Collins G.S., Dhiman P., Andaur Navarro C.L., et al. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open. 2021; 11 (7): e048008. DOI: https://doi.org/10.1136/bmjopen-2020-048008