DECIDE-AI

Developing new reporting guidelines for the early-stage clinical evaluation of decision support systems driven by artificial intelligence.

“Developing new reporting guidelines for the early-stage clinical evaluation of decision support systems driven by artificial intelligence.”

Much like IDEAL 2a/2b studies, comprehensive early formative evaluation, with an emphasis on the human users, is a necessary bridge from the development of clinical decision support systems driven by artificial intelligence (AI) to their clinical implementation. Early-stage studies are also important steppingstones toward the large-scale comparative trials needed to generate robust evidence about these systems’ effectiveness. The objective of DECIDE-AI is to improve reporting of clinical AI studies along four main axes:

  • the performance of the AI systems when first used with humans in small-scale, actual clinical settings,
  • the safety profile of the AI systems prior to large-scale utilisation,
  • the human factors (ergonomic) evaluation of the AI systems,
  • the preparatory steps towards large-scale (randomised controlled) clinical trials.

You can read more about the project in correspondence to the editors of Nature Medicine: https://www.nature.com/articles/s41591-021-01229-5

Access this article on Nature Medicine

Access this article in the British Medical Journal

Link to the systematic review performed as part of the preliminary investigations: https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2777403

 

Contact: Baptiste Vasey, baptiste.vasey@nds.ox.ac.uk

 

People:

From the IDEAL Collaboration: Baptiste Vasey (University of Oxford), Dr Mudathir Ibrahim (Maimonides Medical Center), Prof Bruce Campbell (University of Exeter), Prof Peter McCulloch (University of Oxford)

External collaborators: Myura Nagendran (Imperial College London), Prof David Clifton (University of Oxford), Prof Gary Collins (University of Oxford), Prof Alastair Denniston (University of Birmingham), Dr Livia Faes (Moorfields Eye Hospital), Dr Bart Geerts (healthplus.ai), Dr Xiaoxuan Liu (University of Birmingham), Dr Piyush Mathur (Cleveland Clinic, Ohio), Dr Lauren Morgan (Morgan Human Systems Ltd), Dr Suchi Saria (Johns Hopkins University), Dr Peter Watkinson (University of Oxford)

DECIDE-AI expert group: Aaron Y. Lee32; Alan G. Fraser33; Alastair K. Denniston12,13,14; Ali Connell34; Alykhan Vira35; Andre Esteva36; Andrew D. Althouse37; Andrew L. Beam38; Anne de Hond39; Anne-Laure Boulesteix40; Anthony Bradlow41; Ari Ercole42; Arsenio Paez43; Athanasios Tsanas44; Baptiste Vasey1,2,3; Barry Kirby45; Bart Geerts15; Ben Glocker46; Bilal A. Mateen8,17,18; Bruce Campbell5,6; Campbell Rogers24; Carmelo Velardo2,47; Chang Min Park48; Charisma Hehakaya49; Chris Baber50; Chris Paton51; Christian Johner52; Christopher J. Kelly34; Christopher J. Vincent53; Christopher Yau54; Clare McGenity55; Constantine Gatsonis56; Corinne Faivre-Finn57; Crispin Simon58; Daniel S.W. Ting27,28; Danielle Sent59; Danilo Bzdok60; Darren Treanor61; David A. Clifton2; David C. Wong62; David F. Steiner63; David Higgins64; Dawn Benson65; Declan P. O’Regan66; Dinesh V. Gunasekaran27; Dominic Danks67; Emanuele Neri68; Evangelia Kyrimi69; Falk Schwendicke70; Farah Magrabi71; Frances Ives72; Frank E. Rademakers73; Gary S. Collins7; George E. Fowler74; Giuseppe Frau75; H. D. Jeffry Hogg76; Hani J. Marcus77; Heang-Ping Chan78; Henry Xiang79; Hugh F. McIntyre80; Hugh Harvey81; Hyungjin Kim82; Ibrahim Habli83; James C. Fackler84; James Shaw85; Janet Higham86; Jared M. Wohlgemut87; Jaron Chong88; Jean-Emmanuel Bibault89; Jérémie F. Cohen90; Jesper Kers91; Jessica Morley92; Joachim Krois93; Joao Monteiro94; Joel Horovitz16; Johan Ordish23; John Fletcher30; Jonathan Taylor95; Jung Hyun Yoon96; Karandeep Singh97; Karel G.M. Moons98; Kassandra Karpathakis99; Ken Catchpole100; Kerenza Hood101; Konstantinos Balaskas102; Konstantinos Kamnitsas50; Laura Militello103; Laure Wynants104; Lauren Morgan22; Lauren Oakden-Rayner105; Laurence B. Lovat106; Livia Faes14; Luc J.M. Smits107; Ludwig C. Hinske108; M. Khair ElZarrad109,§; Maarten van Smeden110; Mara Giavina-Bianchi111; Mark Daley112; Mark P. Sendak113; Mark Sujan114; Maroeska Rovers115; Matthew DeCamp116; Matthew Woodward117; Matthieu Komorowski118; Max Marsden87; Maxine Mackintosh119; Melissa D. McCradden20,21; Michael D. Abramoff120; Miguel Ángel Armengol de la Hoz121; Myura Nagendran4; Neale Hambidge122; Neil Daly123; Niels Peek124; Oliver Redfern125; Omer F. Ahmad126; Patrick M. Bossuyt127; Pearse A. Keane128; Pedro N.P. Ferreira129; Peter McCulloch1; Peter Watkinson3,29; Peter Wheatstone31; Petra Schnell-Inderst130; Pietro Mascagni131; Piyush Mathur19; Prokar Dasgupta132; Pujun Guan133; Rachel Barnett15; Rawen Kader134; Reena Chopra34; Ritse M. Mann135; Rupa Sarkar136; Saana M. Mäenpää137; Samuel G. Finlayson138; Sarah Vollam3; Sebastian J. Vollmer139; Seong Ho Park140; Shakir Laher141; Shalmali Joshi142; Siri L. van der Meijden15,143; Spiros Denaxas8,9,10,11; Suchi Saria25,26; Susan C. Shelmerdine144; Tien-En Tan27; Tom J.W. Stocker145; Valentina Giannini146; Vince I. Madai147; Virginia Newcombe148; Wei Yan Ng27; Wendy A. Rogers149; William Ogallo150; Wim Weber30; Xiaoxuan Liu12,13; Yoonyoung Park151; Zane B. Perkins87.

 

1 Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK

2 Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK

3 Critical Care Research Group, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK

4 UKRI Centre for Doctoral Training in AI for Healthcare, Imperial College London, London, UK

5 University of Exeter Medical School, Exeter, UK

6 Royal Devon and Exeter Hospital, Exeter, UK

7 Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology & Musculoskeletal Sciences, University of Oxford, Oxford, UK

8 Institute of Health Informatics, University College London, London, UK

9 British Heart Foundation Data Science Centre, London, UK

10 Health Data Research UK, London, UK

11 UCL Hospitals Biomedical Research Centre, London, UK

12 University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK

13 Academic Unit of Ophthalmology, Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, UK

14 Moorfields Eye Hospital NHS Foundation Trust, London, UK

15 Healthplus.ai-R&D B.V., Amsterdam, the Netherlands

16 Department of Surgery, Maimonides Medical Center, Brooklyn, NY, USA

17 The Wellcome Trust, London, UK

18 The Alan Turing Institute, London, UK

19 Department of General Anesthesiology, Anesthesiology Institute, Cleveland Clinic, OH, USA

20 The Hospital for Sick Children, Toronto, Canada

21 Dalla Lana School of Public Health, University of Toronto, Canada

22 Morgan Human Systems Ltd, Shrewsbury, UK

23 The Medicines and Healthcare products Regulatory Agency, London, UK

24 HeartFlow Inc., Redwood City, CA, USA

25 Departments of Computer Science, Statistics, and Health Policy, and Division of Informatics, Johns Hopkins University, Baltimore, MD; USA

26 Bayesian Health, New York, NY, USA

27 Singapore National Eye Center, Singapore Eye Research Institute, Singapore, SG

28 Duke-NUS Medical School, National University of Singapore, Singapore, SG

29 NIHR Biomedical Research Centre Oxford, Oxford University Hospitals NHS Trust, Oxford, UK

30 The BMJ, London, UK

31 School of Medicine, University of Leeds, Leeds, UK

32 Department of Ophthalmology, School of Medicine, University of Washington, Seattle, WA, USA;

33 School of Medicine, Cardiff University, Cardiff, UK;

34 Google Health, London, UK;

35 Quantium Health, Johannesburg, SA;

36 Artera Research, Artera, Mountain View, CA, USA;

37 University of Pittsburgh, Pittsburgh, PA, USA;

38 Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, MA, USA;

39 CAIRElab, Leiden University Medical Centre, Leiden, NL;

40 Institute for Medical Information Processing, Biometry and Epidemiology, Ludwig Maximilian University, Munich, DE;

41 Rheumatology Department, Royal Berkshire Hospital, Reading, UK;

42 Cambridge Centre for AI in Medicine, University of Cambridge, Cambridge, UK;

43 Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK;

44 Usher Institute, Edinburgh Medical School, University of Edinburgh, Edinburgh, UK;

45 K Sharp, Llanelli, UK;

46 Department of Computing, Imperial College London, London, UK;

47 Sensyne Health, UK;

48 Seoul National University College of Medicine, Seoul, KR;

49 Division of Imaging & Oncology, University Medical Center Utrecht, Utrecht, NL;

50 School of Computer Science, University of Birmingham, Birmingham, UK;

51 Nuffield Department of Medicine, University of Oxford, Oxford, UK;

52 Johner Institute, Konstanz, DE;

53 PDD Group Ltd, London, UK;

54 University of Manchester, Manchester, UK;

55 Pathology and Data Analytics, University of Leeds, Leeds, UK;

56 Department of Biostatistics, Brown University School of Public Health, Providence, RI, USA;

57 The Christie NHS Foundation Trust, Manchester, UK;

58 London School of Economics, London, UK;

59 Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam, NL;

60 Mila, Quebec Artificial Intelligence Institute, Montreal, CA;

61 Leeds Teaching Hospitals NHS Trust, Leeds, UK;

62 Department of Computer Science and Centre for Health Informatics, University of Manchester, Manchester, UK;

63 Google Health, Palo Alto, USA;

64 Berlin Institute of Health, Berlin, DE;

65 Healthcare Safety Investigation Branch, Farnborough, UK

66 MRC London Institute of Medical Sciences, Imperial College London, London, UK;

67 Institute of Cancer and Genomic Sciences, University of Birmingham, Birmingham, UK;

68 University of Pisa, Pisa, IT;

69 School of Electronic Engineering and Computer Science (EECS), Queen Mary University of London, London, UK;

70 Charité Universitätsmedizin Berlin, Berlin, DE;

71Australian Institute of Health Innovation, Macquarie University, Sydney, AU;

72 West Midlands Academic Health Science Network, Birmingham, UK;

73 Department Cardiovascular sciences, KU Leuven, Leuven, BE;

74 Bristol Centre for Surgical Research, Department of Population Health Sciences, Bristol Medical School, Bristol, UK;

75 Deep Blue, Rome, IT;

76 Population Health Science Institute, Newcastle University, Newcastle upon Tyne, UK;

77 Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, Queen Square, London, UK;

78 Department of Radiology, University of Michigan, Ann Arbor, MI, USA;

79 The Abigail Wexner Research Institute, Nationwide Children’s Hospital, The Ohio State University, Columbus, OH, USA;

80 Department of Medicine, East Sussex Healthcare Trust, Hastings, UK;

81 Hardian Health, UK;

82 Department of Radiology, Seoul National University Hospital, Seoul, KR;

83 Department of Computer Science, University of York, York, UK;

84 Department of Anesthesiology and Critical Care Medicine, The Johns Hopkins University School of Medicine, Baltimore, MD, USA;

85 Joint Centre for Bioethics, University of Toronto, Toronto, CA;

86 University of Oxford, Oxford, UK;

87 Centre for Trauma Sciences, Blizard Institute, Queen Mary University of London, London, UK;

88 Medical Imaging, Western University, London, CA;

89 Radiation Oncology Department, Hôpital Européen Georges Pompidou, AP-HP, Paris, FR;

90 Center of Research in Epidemiology and Statistics (Inserm 1153), Université de Paris, Paris, FR;

91 Department of Pathology, Amsterdam UMC, University of Amsterdam, Amsterdam, NL;

92 Oxford Internet Institute, University of Oxford, Oxford, UK;

93 Oral Diagnostics & Digital Health & Health Services Research, Charité Universitätsmedizin Berlin, Berlin, Germany;

94 Nature Medicine, New York, NY, USA;

95 Nuclear Medicine / 3DLab, Sheffield Teaching Hospitals, Sheffield, UK;

96 Department of Radiology, Severance Hospital, Yonsei University College of Medicine, Seoul, KR;

97 Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, MI, USA;

98 Julius Center, UMC Utrecht, Utrecht University, Utrecht, NL;

99 Harvard TH Chan School of Public Health, Boston, MA, USA;

100 Medical University of South Carolina, Charleston, SC, USA;

101 Centre for Trials Research, Cardiff University, Cardiff, UK;

102 Moorfields Ophthalmic Reading Centre and Clinical AI Hub, Moorfields Eye Hospital, London, UK;

103 Applied Decision Science LLC, USA;

104 Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Maastricht, NL;

105 Australian Institute of Machine Learning, University of Adelaide, Adelaide, AU;

106 University College London, London, UK;

107 Department of Epidemiology, Maastricht University, Maastricht, NL;

108 Institute for Biomedical Information Processing, Biometry and Epidemiology, Ludwig-Maximilians-University Munich, Munich, DE;

109 U.S. Food and Drug Administration;

110 Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, NL;

111 Hospital Israelita Albert Einstein, São Paulo, BR;

112 The University of Western Ontario, London, CA;

113 Duke Institute for Health Innovation, Durham, USA;

114 Human Factors Everywhere Ltd, Woking, UK;

115 Department of Operating rooms, Radboudumc, Nijmegen, NL;

116 University of Colorado, Boulder, CO, USA;

117 THIS Institute, School of Clinical Medicine, University of Cambridge, Cambridge, UK;

118 Dept of Surgery and Cancer, Imperial College London, London, UK;

119 Genomics England, Queen Mary University of London, London, United Kingdom;

120 University of Iowa, Iowa City, IA, USA;

121 Big Data Department, FPS, Regional Ministry of Health of Southern Spain, ES;

122 National Hospital for Neurology and Neurosurgery, Queen Square, London, UK;

123 Skin Analytics, London, UK;

124 Division of Informatics, Imaging and Data Science, The University of Manchester, Manchester, UK;

125 Kadoorie Centre for Critical Care Research and Education, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK;

126 Wellcome/EPSRC centre for Interventional & Surgical Sciences (WEISS), University College London, London, UK;

127 Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, NL;

128 Institute of Ophthalmology, University College London, London, UK;

129 CENTEC – IST, University of Lisbon, Lisbon, PT;

130 Institut of Public Health, Medical Decision Making and HTA, UMIT – University for Health Sciences, Medical Informatics and Technology, Hall i. T., AT;

131 Gastrointestinal Endoscopic Surgery, Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, IT;

132 King’s Health Partners Academic Surgery, King’s College London, London, UK;

133 Graduate School of Biomedical Sciences, University of Texas MD Anderson Cancer Center and UTHealth, Houston, USA;

134 Division of Surgery and Interventional Sciences, University College London, London, UK;

135 Department of medical imaging, Radboud University Medical Center, Nijmegen, NL;

136 The Lancet Digital Health, The Lancet Group, London, UK;

137 Department of Neurosurgery, Helsinki University Hospital, Helsinki, FI;

138 Harvard Medical School, Boston, MA, USA;

139 Data Science and its Application, DFKI, Kaiserslautern, DE;

140Department of Radiology, Asan Medical Center, Seoul, KR;

141 University of York, York, UK;

142 SEAS, Harvard University, Cambridge, MA, USA;

143 Leiden University Medical Center, Leiden, NL;

144 Department of Clinical Radiology, Great Ormond Street Hospital for Children NHS Foundation Trust, London, UK;

145 PUBLIC ltd, Oxford, UK;

146 University of Turin, Turin, IT;

147 QUEST Centre for Responsible Research, Berlin Institute of Health, Charité Universitätsmedizin Berlin, Berlin, Germany;

148 University Division of Anaesthesia, Department of Medicine, University of Cambridge, Cambridge, UK;

149 Philosophy Department and School of Medicine, Macquarie University, Sydney, AU;

150 IBM Research Africa, Nairobi, KE;

151 Center for Computational Health, IBM Research, Cambridge, MA USA;

 

  • Participation represents personal views and perspectives that may not necessarily reflect the positions and opinions of the U.S. FDA.