Hirst A, Philippou Y, Blazeby J, Campbell B, Campbell M, Feinberg J, Rovers M,  Blencowe N, Pennell C, Quinn T, Rogers W, Cook J, Kolias AG, Agha R, Dahm P, Sedrakyan A, McCulloch P. No Surgical Innovation Without Evaluation: Evolution  and Further Development of the IDEAL Framework and Recommendations.  Ann Surg.  2018 Apr 24. doi: 10.1097/SLA.0000000000002794. [Epub ahead of print] PubMed  PMID: 29697448.

A writing group from the IDEAL Collaboration including surgeons, research trialists, methodologists, statisticians, ethicists, and experts in device evaluation and HTA have developed an update to the original IDEAL Framework papers published in the Lancet in 2009.

OBJECTIVE: To update, clarify, and extend IDEAL concepts and recommendations.

BACKGROUND: New surgical procedures, devices, and other complex interventions need robust evaluation for safety, efficacy, and effectiveness. Unlike new medicines, there is no internationally agreed evaluation pathway for generating and analyzing data throughout the life cycle of surgical innovations. The IDEAL Framework and  Recommendations were designed to provide this pathway and they have been used increasingly since their introduction in 2009. Based on a Delphi survey, expert workshop and major discussions during IDEAL conferences held in Oxford (2016) and New York (2017), this article updates and extends the IDEAL Recommendations, identifies areas for future research, and discusses the ethical problems faced by investigators at each IDEAL stage.

METHODS: The IDEAL Framework describes 5 stages of evolution for new surgical therapeutic interventions-Idea, Development, Exploration, Assessment, and Long-term Study. This comprehensive update proposes several modifications.

1. A “Pre-IDEAL” stage describing preclinical studies has been added.
2. We discuss potential adaptations to expand the scope of IDEAL (originally designed for surgical procedures) to accommodate therapeutic devices, through an IDEAL-D variant.
3. We explicitly recognise the value of comprehensive data collection through registries at all stages in the Framework.
4. We examine the ethical issues that arise at each stage of IDEAL and underpin the recommendations.

The Recommendations for each stage are reviewed, clarified and additional detail added. We use a PICO format to outline the key features of each stage, outline reporting guidance for each stage and describe how to identify when an endpoint to a stage has been reached.

CONCLUSIONS: The intention of this article is to widen the practical use of IDEAL by clarifying the rationale for and practical details of the Recommendations. Additional research based on the experience of implementing these Recommendations is needed to further improve them.

We look forward to developing these Recommendations further by working with surgical researchers developing innovative surgical therapies and medical devices.

Peter McCulloch, Chair of IDEAL and co-authors examined progress in surgical research and their findings are now published in The Lancet Online First

Joshua Feinberg, surgeon, researcher and member of the IDEAL Collaboration at Maimonides Medical Center, Brooklyn, New York describes the research undertaken to examine how surgical research has improved in the last two decades and whether the IDEAL Framework and Recommendations have contributed.

“The quality of clinical research in surgery has long attracted criticism. High-quality randomised trials have proved difficult to undertake in surgery, and many surgical treatments have therefore been adopted without adequate supporting evidence of efficacy and safety. To address this, we conducted a review of the progress of methodology in clinical studies of surgery, and whether it is moving towards compliance with the IDEAL Recommendations.

To evaluate progress, we examined the surgical literature, focusing specifically on studies of surgical outcomes, in two periods 10 years apart, one before and one after the publications of the IDEAL Recommendations (2000-2004 and 2010-2014). Compliance with the following IDEAL recommendations were compared between the two time periods:

• Use of standardised terminology
• Definition and description of procedure
• Prospective data collection
• Explanation of modifications during early studies (2a)
• Previous analysis of learning curves in pre-RCT studies
• Use of quality control measures
• Use of qualitative research to define RCT questions
• Use of prior prospective cohort study to prepare for RCTs
• Mention of pilot of feasibility studies to prepare for RCT
• Masking reported in RCTs

The main findings of the study include:

IMPROVEMENT in the use of standard outcome measures, adoption of Consolidated Standards of Reporting Trials (CONSORT) standards, and assessment of the quality of surgery and of learning curves.

NO PROGRESS in the use of qualitative research or reporting of modifications during procedure development.

When discussing visions for the future, the paper concludes

“Surgical research is getting better, although it still has a long way to go…The IDEAL Framework and Recommendations have probably only contributed in a minor way to the improvements seen so far, but their influence is growing, they are useful as a tool to measure progress, and they represent a serious attempt to create a new framework for surgical research methodology.”

We are pleased to announce that the China IDEAL Centre was opened in Chengdu, Schizuan province, on 28th October 2017.

The Director, Professor Xin Sun, is also Director of the Chinese Cochrane Centre, and of a large statistics and clinical epidemiology department at the West China Hospital in Chengdu. Funded by the Chinese government through the West China hospital and university, the centre will educate clinicians about IDEAL and research methodology and help groups to develop their clinical research using IDEAL principles.

At the opening event of the Centre Professor Xin Sun and the Emeritus Director, Professor Louping Yi, emphasised the value of IDEAL for the evaluation of new medical devices and procedures in the Chinese health system.  They referred to recent new guidance from the FDA on the use of “real world evidence” to evaluate devices, and stressed how IDEAL could provide a guiding template for interpreting FDA proposals.  This theme was supported by the representative of the Chinese FDA who spoke at the meeting, who explained how China aimed to develop a distinctive approach which is nevertheless compatible with US FDA guidance.

Professor Peter McCulloch (IDEAL Chair) presented two talks, one on the principles of IDEAL and the other on its use in device evaluation, and how it could assist regulators and healthcare purchasers.

The meeting was generously sponsored by Medtronic China, whose Vice-President for Clinical Affairs, Xioajing Chen, made a presentation on Medtronic’s plans for device surveillance.

We look forward to supporting and collaborating with IDEAL China. Future plans include development of a post-doctoral Fellowship in Oxford for one of the IDEAL China staff and co-operation on research projects. It was agreed that one important area where IDEAL needed to develop a clear position was the guidelines for policy on what types of evidence should be acceptable for regulation or purchasing in which stages of evolution of a device.  An annual IDEAL China conference is planned and it is hoped next year’s event will be able to report progress based on this year’s meeting.

We are delighted to welcome Dr Arsenio Paez to the IDEAL Collaboration in the role of Specialty Lead for Physiotherapy.

Arsenio is a physiotherapist with a pediatric private practice in New York, NY. He is a Professor in the Doctoral Program in Physical Therapy in the Dept of Physical Therapy, Movement, and Rehab Sciences at Northeastern University, Boston, MA, and former Lend Fellow in Neurodevelopmental Disabilities at the Children’s Hospital, Boston. 

Arsenio recently completed the MSc in Evidence-Based Health Care at the University of Oxford, and has applied to continue to a DPhil in Evidence-Health Care for 2018.

His current area of research is in neurodevelopmental outcomes in childhood epilepsy, innovative practice in physiotherapy and clinical trial methods in complex interventions.

Arsenio states ” I am very enthusiastic about IDEAL and what IDEAL-Physio can do to improve the quality of practice and research in physiotherapy. We have had a great surge in the amount of evidence being designed and produced in physiotherapy, with over 18,500 clinical trials and 5600 systematic reviews added to the body of evidence in the last decade alone. Ideal-Physio offers us an invaluable and practical framework to help guide the innovation of new practices and improve the quality of evidence. The unique, complex nature of our interventions requires a dynamic and practicable approach to innovation and evidence gathering, and I believe IDEAL-Physio draws on these to create something truly representative of the nature of complex interventions. It also has great potential to aid us in preparing the evidence-makers and innovators of the future, with very promising applications in academic settings and clinical education”.

Arensio has led work on extending IDEAL to physiotherapy (IDEAL-Physio) recently accepted for publication in the Physical Therapy Journal:

Beard D, Hamilton D, Davies L, Cook J, Hirst A, McCulloch P and Paez A. Evidence-based evaluation of practice and innovation in Physical Therapy using the IDEAL-Physio framework. Physical Therapy Journal (In Press) DOI 10.1093/ptj/pzx103 

(Link to follow when available)

“There is a clear need for surgical registry data to improve patient safety and help regulate surgical practices.The IDEAL collaborative, Department of Health (DOH), the National Institute for Health and Care Excellence (NICE), policymakers and commissioning groups have called for surgical registries that can collect prospective outcome and safety data, promote transparency as well as provide patients and the public with information on their care. However, developing and maintaining a registry faces considerable challenges with the majority of registries failing.

To help address this, we conducted a systematic review and narrative synthesis, learning from existing and previous registries, to identify the fundamentals to successful registry development. We found that the key factors for successful registry development include:  steering committee to lead and oversee the registry; clear registry objectives; planning for initial and long-term funding; strategic national collaborations among key stakeholders; dedicated registry management team; consensus meetings to agree registry dataset; established data processing systems; anticipating challenges; and implementing strategies to increase data completion. Patient involvement and awareness of legal factors should occur throughout the development process.

Our work provides robust knowledge that can be used to inform the successful development of any UK surgical registry. It also provides a methodological framework for international surgical registry development”

Mr Rishi Mandavia , NIHR Academic Clinical Fellow ENT Surgery and NICE Specialist Advisor, evidENT Team, Ear Institute, University College London

Link to the paper here: http://bmjopen.bmj.com/content/7/9/e017373.info

“For many years I saw articles compliant with the STROBE guidelines but still missing large amounts of key information relevant to surgeons.  Such key information centred around the intervention itself is paramount to understanding what’s been done in a study, critically appraising and replicating it. Readers need complete, clear and transparent reporting and we hope the STROCSS guideline goes some way towards this.”

Dr Riaz Agha, Managing and Executive Editor, International Journal of Surgery

The full guideline and checklist for reporting surgical cohort, cross-sectional and case-control studies is available on the STROCSS website: http://www.strocssguideline.com/

Agha RA, Borrelli MR, Vella-Baldacchino M, Thavayogan R, Orgill DP; STROCSS Group. The STROCSS statement: Strengthening the Reporting of Cohort Studies in Surgery. Int J Surg. 2017;46: 198–202

The STROCSS checklist can be downloaded here: the_strocss_statement_checklist

Make a note to yourself now to save 13th-14th September 2018 to participate in #IDEAL2018. Our 3rd International Conference will be hosted by the Bristol Centre for Surgical Research 

Keep an eye on this site for further information and follow updates on Twitter @IDEALCollab and @CSR_Bris

The IDEAL team Claudia Ashton, Allison Hirst, Claire Thomson and Peter McCulloch  presented a workshop on using IDEAL to appraise the surgical literature at #EvidenceLive on 21st June 2017 http://evidencelive.org/

Following the workshop Peter McCulloch, Chair of the IDEAL Collaboration discussed some of the challenges of surgical research with Douglas Badenoch @Mental_Elf

As part of our 2017 Conference in New York, 4th-5th May 2017, we are holding a half day educational workshop. Please select this at the time of registration for the conference at www.idealconference2017.com

Early registration discount available until April 1st   Registration fees include breakfast and lunch each day of the conference, and CME credits will be available for physicians. For further information, contact: information@ideal-conference2017.com or visit www.ideal-collaboration.net.

Workshop 4th May 2017

Critical appraisal using the IDEAL Framework (provisional timetable)

8.30—9.15AM (JH)  History of EBM; Inherent difficulties of surgical evaluation; lack of formal education among trainees in US

9.15—10.00AM (JF)  Group workshop – Systematic approach to the Critical Appraisal of a Surgical Paper

• Short lecture explaining the approach to appraising a surgical paper using GATE framework
• Using an RCT – group(s) will appraise the paper

10.00—10.30AM               Coffee/Tea

10.30—11.00AM (CP)  IDEAL: Framework and How to utilize it

• Describe the purpose of IDEAL and why it is necessary

• Explain each stage and the relevant questions at each stage

11.00AM – 11.30AM  Group workshop – Identifying the IDEAL stage of innovation

• Group given a surgical article and asked to identify the IDEAL stage described in the paper

11.30 – 12.00PM (PM)  Using IDEAL: Progressing through the life-cycle of surgical Innovation

• As a large group, we take a surgical procedure through the IDEAL stages of innovation and display how each stage would be designed

12.00—12.15PM (JF)  IDEAL: Current status and future

• Update since conference last year
• Applications to devices and other sectors

12:15 – 12.30 (JF)  Assessment of participants’ comprehension of IDEAL

• Short questionnaire asking participants about Framework & Recommendations
• Will provide handouts on IDEAL and critical appraisal

Teaching Faculty

JH – Joel Horovitz, Vice Chair of Surgery, Maimonides Medical Center

JF – Joshua Feinberg, Maimonides Medical Center

CP – Christopher Pennell, Maimonides Medical Center

PM – Peter McCulloch, Chair IDEAL Collaboration, University of Oxford

This question was recently addressed and the findings published in the British Journal of Surgery

• Chapman SJ. Shelton B. Maruthappu M. Singh P. McCulloch P. Bhangu A. Cross-sectional observational study of the availability of evidence supporting novel implantable devices used in gastrointestinal surgery. Br J Surg 2017 [Epub ahead of print]. doi: 10.1002/bjs.10485

Two of the authors describe what they did and what they found;

Stephen J Chapman, Leeds Institute of Biomedical and Clinical Sciences, University of Leeds, Leeds, UK

Aneel Bhangu, Dept of ColorectalSurgery, University of Birmingham, Birmingham, UK

Innovation in surgery is important to ensure the treatment of surgical disease continues to advance. The role of industry is paramount in driving this forward, but must be matched by robust clinical evidence and dedicated regulation.

Whilst regulatory bodies in the U.K. work to ensure safe introduction of surgical devices, many are supported by minimal or no evidence of safety or effectiveness. We are acutely aware of this problem; new surgical devices are introduced every day yet these may be unfamiliar or unproven. Whilst regulatory bodies are often satisfied with evidence of “equivalence”, the true clinical impacts of these are unclear.

Our study, published in the British Journal of Surgery, demonstrated the extent of this problem (1). We systematically compiled a list of implantable devices used in gastrointestinal surgery and set out to find evidence to support their use. Alarmingly, only 1 in 10 of these devices was supported by evidence from high quality randomised controlled trials (RCT).

Whilst RCTs are perceived to slow the rate of innovation, dedicated networks of surgeons have proven their ability to evaluate surgical interventions quickly and effectively (2). Additionally, whilst RCTs are not always the most appropriate design to test novel interventions, they provide high quality evidence when undertaken by dedicated clinical and academic networks.

In line with proposals from the IDEAL Collaboration, we encourage a standardised, step-wise approach to the assessment of novel surgical technology. This should begin with first in man studies, progress through stages of randomised assessment, and culminate in long term surveillance in dedicated devices registries.

References:

1. Chapman SJ. Shelton B. Maruthappu M. Singh P. McCulloch P. Bhangu A. Cross-sectional observational study of the availability of evidence supporting novel implantable devices used in gastrointestinal surgery. Br J Surg 2017 [Epub ahead of print]. doi: 10.1002/bjs.10485
2. Bhangu A. Kolias AG. Pinkney T. Hall NJ. Fitzgerald JE. Surgical research collaboratives in the UK. Lancet 2013;382:1091-2.