Peter McCulloch talks about IDEAL at Evidence Live 2017

The IDEAL team Claudia Ashton, Allison Hirst, Claire Thomson and Peter McCulloch  presented a workshop on using IDEAL to appraise the surgical literature at #EvidenceLive on 21st June 2017

Following the workshop Peter McCulloch, Chair of the IDEAL Collaboration discussed some of the challenges of surgical research with Douglas Badenoch @Mental_Elf




Educational Workshop 4th May 2017, NY: Critical appraisal using IDEAL

As part of our 2017 Conference in New York, 4th-5th May 2017, we are holding a half day educational workshop. Please select this at the time of registration for the conference at

Early registration discount available until April 1st   Registration fees include breakfast and lunch each day of the conference, and CME credits will be available for physicians. For further information, contact: or visit

Workshop 4th May 2017

Critical appraisal using the IDEAL Framework (provisional timetable)

8.30—9.15AM (JH)  History of EBM; Inherent difficulties of surgical evaluation; lack of formal education among trainees in US

9.15—10.00AM (JF)  Group workshop – Systematic approach to the Critical Appraisal of a Surgical Paper

  • Short lecture explaining the approach to appraising a surgical paper using GATE framework
  • Using an RCT – group(s) will appraise the paper

10.00—10.30AM               Coffee/Tea

10.30—11.00AM (CP)  IDEAL: Framework and How to utilize it

  • Describe the purpose of IDEAL and why it is necessary
  • Explain each stage and the relevant questions at each stage

11.00AM – 11.30AM  Group workshop – Identifying the IDEAL stage of innovation

  • Group given a surgical article and asked to identify the IDEAL stage described in the paper

11.30 – 12.00PM (PM)  Using IDEAL: Progressing through the life-cycle of surgical Innovation

  • As a large group, we take a surgical procedure through the IDEAL stages of innovation and display how each stage would be designed

12.00—12.15PM (JF)  IDEAL: Current status and future

  • Update since conference last year
  • Applications to devices and other sectors

12:15 – 12.30 (JF)  Assessment of participants’ comprehension of IDEAL

  • Short questionnaire asking participants about Framework & Recommendations
  • Will provide handouts on IDEAL and critical appraisal

Teaching Faculty

JH – Joel Horovitz, Vice Chair of Surgery, Maimonides Medical Center

JF – Joshua Feinberg, Maimonides Medical Center

CP – Christopher Pennell, Maimonides Medical Center

PM – Peter McCulloch, Chair IDEAL Collaboration, University of Oxford

Where is the evidence for novel surgical devices?

This question was recently addressed and the findings published in the British Journal of Surgery

Two of the authors describe what they did and what they found;


Stephen J Chapman, Leeds Institute of Biomedical and Clinical Sciences, University of Leeds, Leeds, UK




Aneel Bhangu, Dept of ColorectalSurgery, University of Birmingham, Birmingham, UK


Innovation in surgery is important to ensure the treatment of surgical disease continues to advance. The role of industry is paramount in driving this forward, but must be matched by robust clinical evidence and dedicated regulation.

Whilst regulatory bodies in the U.K. work to ensure safe introduction of surgical devices, many are supported by minimal or no evidence of safety or effectiveness. We are acutely aware of this problem; new surgical devices are introduced every day yet these may be unfamiliar or unproven. Whilst regulatory bodies are often satisfied with evidence of “equivalence”, the true clinical impacts of these are unclear.

Our study, published in the British Journal of Surgery, demonstrated the extent of this problem (1). We systematically compiled a list of implantable devices used in gastrointestinal surgery and set out to find evidence to support their use. Alarmingly, only 1 in 10 of these devices was supported by evidence from high quality randomised controlled trials (RCT).

Whilst RCTs are perceived to slow the rate of innovation, dedicated networks of surgeons have proven their ability to evaluate surgical interventions quickly and effectively (2). Additionally, whilst RCTs are not always the most appropriate design to test novel interventions, they provide high quality evidence when undertaken by dedicated clinical and academic networks.

In line with proposals from the IDEAL Collaboration, we encourage a standardised, step-wise approach to the assessment of novel surgical technology. This should begin with first in man studies, progress through stages of randomised assessment, and culminate in long term surveillance in dedicated devices registries.


  1. Chapman SJ. Shelton B. Maruthappu M. Singh P. McCulloch P. Bhangu A. Cross-sectional observational study of the availability of evidence supporting novel implantable devices used in gastrointestinal surgery. Br J Surg 2017 [Epub ahead of print]. doi: 10.1002/bjs.10485
  2. Bhangu A. Kolias AG. Pinkney T. Hall NJ. Fitzgerald JE. Surgical research collaboratives in the UK. Lancet 2013;382:1091-2.

Opportunities to develop collaborative surgical projects at IDEAL 2017 Conference



Would you like to participate in the development of major new scientific studies to evaluate your work?

If so, the 2017 IDEAL MEETING in New York offers a unique opportunity.  During the SURGICAL SPECIALITY THINK TANKS ON 4TH MAY IDEAL 2017 Provisional Agenda  we’ll be seeking to initiate multi-centre prospective collaborative studies of one new cutting edge technique in each workshop.  These will use the IDEAL 2b PROSPECTIVE COHORT format and the intention is that the initial study will form the basis for a subsequent major RCT.  If you would like to get in on the ground floor of one of these collaborative studies by surgical innovators, register for the meeting and send your ideas to:

Registration is now open at – early registration discount available until April 1st 2017! Registration fees include breakfast and lunch each day of the conference, and CME credits will be available for physicians. For further information, contact: or visit

This is an opportunity to help plan important studies which will shape the direction of future practice in your speciality.  The IDEAL Collaboration contains expert methodologists, statisticians and others who can help guide the development of research groups and reduce the burden of participation on clinicians to a minimum. 

We look forward to meeting you and working together on these exciting projects.

Prof Peter McCulloch, IDEAL Chair, Oxford (left)


  Prof Art Sedrakyan, IDEAL    Vice-Chair, New York (right)

Abstracts from our 2016 Conference now published in International Journal of Surgery

Abstracts from our 2016 Conference are now available in  the International Journal of Surgery

December 2016 Volume 36, Supplement 2, S133-S144

Access to innovative treatments and device support during IDEAL stages 2b-4 K. Hutchison

Advancing the cause of research registration: The first 500 registrations of the R. Agha, A.J. Fowler, C. Limb, Y. Al Omran, H. Sagoo, K. Koshy, D.J. Jafree, M.O. Anwar, Peter McCulloch, D.P. Orgill

Surgical trainee research collaboratives in the United Kingdom A.G. Kolias, A.A.B. Jamjoom, P.N.H. Phan, P.J.A. Hutchinson

The merits of decision modeling in the earliest stages of the IDEAL framework – The case of innovative bilateral DIEP flap surgery J. Grutters, J. Gerrits, L. Schultze Kool, D. Ulrich, M. Rovers, S. Hummelink

Evaluation of innovation in radiation oncology: R-IDEAL H. Verkooijen, L. Kerkmeijer, D. Fuller, Robbert Huddart, C. Faivre-Finn, M. Verheij, A. Sahgal, E. Hall, M. van Vulpen

Small simple trials: A strategy to study rare surgical condition J.G. Wright

Short-term cost-effectiveness of colonic stenting is lost at 90-day follow-up in patients with malignant large bowel obstruction  H. Yeo, J. Abelson, J. Milsom, S. Sharma, A. Sedrakyan

3-Year outcomes and cost-savings of combined endoscopic laparoscopic surgery (CELS) for benign colon polyps M. Kiely, S. Sharma, A. Sedrakyan, J. Yoo, H. Yeo, J. Abelson, J. Milsom

Stereotactic body radiotherapy followed by surgery for unstable spinal metastases: Technical feasibility and safety study according to the IDEAL stages 1 and 2a.  A. Versteeg, J. van der Velden, H. Verkooijen, W. Eppinga, N. Kasperts, S. Gerlich, C. Oner, M. van Vulpen, J.-J. Verlaan

Perioperative outcomes, health care costs and survival after robotic-assisted versus open radical cystectomy: A national comparative effectiveness study  J. Hu, B. Chughtai, P. O’Malley, J. Halpern, J. Mao, D. Scherr, D. Hershman, J. Wright, A. Sedrakyan

Comparative effectiveness of cancer control and survival after robotic assisted versus open radical prostatectomy J. Hu, B. Chughtai, P. O’Malley, A. Isaacs, J. Wright, D. Hershman, A. Sedrakyan

Reconstruction of bladder defects with amniotic membrane – IDEAL-D Stage 0-1  D. Barski, H. Gerullis, A. Winter, I. Pintelon, J.-P. Timmermans, A. Ramon, M. Boros, G. Varga, T. Otto

Is more evidence always better? The value of adding decision analytical modeling to the IDEAL framework C. Tax, P.H.M. Govaert, M. Stommel, M.G.H. Besselink, H.G. Gooszen, J. Grutters, M.M. Rovers

Progressing through IDEAL: When is the right time to move from observational to randomised studies? – A case study of REBOA J.O. Jansen, M.K. Campbell

MiCollar – A novel iPhone application to analyse cervical spine motion restriction with different size and type of cervical orthoses R. Ingleton, C. Ashton, J. Bull, V.N. Vakhari

The X-Bolt dynamic hip plating system: Evaluating a novel surgical device for hip fracture surgery  M. Fernandez, J. Achten, N. Parsons, X.L. Griffin, M.L. Costa

Radical cystectomy in epidural anaesthesia – Feasibility analysis using a new reporting method following the IDEAL recommendations H. Gerullis, T.H. Ecke, C. Bantel, A. Weyland, D. Barski, T. Jansen, J. Uphoff, F. Wawroschek, A. Winter

Applying IDEAL: Early stage surgical innovation of a novel bio-wrap-assisted vasectomy reversal technique  A. Gudeloglu, J. Brahmbhatt, S. Parekattil

Matching trial design decisions to the needs of those you hope will use the results: The PRECIS-2 tool  K. Loudon, M. Zwarenstein, F. Sullivan, P. Donnan, S. Treweek

The GASTROS Study: Standardising outcome reporting in gastric cancer surgery research  Bilal Alkhaffaf, Anne-Marie Glenny, Jane Blazeby, Paula Williamson, Iain Bruce

Support for reporting guidelines in surgical journals needs improvement: A systematic review  R. Agha, I. Barai, S. Rajmohan, S.Y. Lee, M. Anwar, A.J. Fowler, D. Orgill, D. Altman

Compliance of systematic reviews in plastic surgery with the PRISMA statement: A systematic review  S.Y. Lee, H. Sagoo, K. Whitehurst, G. Wellstead, A.J. Fowler, R. Agha, D. Orgill

A systematic review of the methodological and reporting quality of case series in surgery  R. Agha, A.J. Fowler, S.Y. Lee, B. Gundogan, K. Whitehurst, H. Sagoo, K.J.L. Jeong, D. Altman, D. Orgill

An assessment of the compliance of systematic review articles published in craniofacial surgery with the PRISMA statement guidelines: A systematic review  T.E. Pidgeon, G. Wellstead, H. Sagoo, D.J. Jafree, A.J. Fowler, R. Agha

The use of study registration and protocols in plastic surgery research: A systematic review  T.E. Pidgeon, C. Limb, R. Agha, K. Whitehurst, C. Chandrakumar, G. Wellstead, A.J. Fowler, D. Orgill

Twist-drill craniostomy with hollow screws for evacuation of chronic subdural haematoma  A.G. Kolias, A. Chari, S.J. Bond, P.J.A. Hutchinson

Intra-operative hyperspectral imaging for brain tumour detection and delineation: Current progress on the HELICoid project S. Kabwama, D. Bulters, H. Bulstrode, H. Fabelo, S. Ortega, G.M. Callico, B. Stanciulescu, R. Kiran, D. Ravi, A. Szolna, J.F. Piñeiro

Developing a patient and public involvement intervention to enhance recruitment and retention in UK surgical trials (PIRRIST)  J.C. Crocker, S. Rees, L. Locock, S. Petit-Zeman, A. Chant, S. Treweek, J.A. Cook, N. Farrar, K. Woolfall, J. Bostock, L. Bowman, R. Bulbulia

Osseointegrated joint replacement connected to a lower limb prosthesis: An IDEAL stage 1 proof of concept study with 7 cases  A. Khemka, S. Lord, M. Al Muderis

The East Grinstead Consent Collaborative (EGCC) online consent creator J. Dhanda, M. Dungarwalla

Development of an online platform for registration and outcome measurement of urogynecological implants according to IDEAL-system D. Barski, H. Gerullis, T. Ecke, R. Joukhadar, J. Kranz, R. Tahbaz, F. Queissert, L. Schneidewind, S. Mühlstädt, M. Grabbert, N. Huppertz, A.E. Pelzer, U. Klinge, M. Boros, W. Bader, F. Puppe, T. Otto

Bringing laparoscopy to your own home M. Vella-Baldacchino, M. Vella Baldacchino, M. Schembri, R. Bugeja

Autologous plasma coating improves the biocompatibility of mesh implants. On the IDEAL way from bench to bedside  H. Gerullis, D. Barski, T.H. Ecke, C. Eimer, M. Boros, B. Klosterhalfen, A. Ramon, T. Otto

Percutaneous nephrolithotomy (PCNL) under local infiltrative anesthesia with and without stand-by anesthesia – Complication rates and clinical outcome for a method at the exploration stage according to IDEAL T.H. Ecke, G. Weingart, C. Lange, S. Hallmann, F. Wawroschek, D. Barski, J. Ruttloff, H. Gerullis

The landscape of surgical innovation in robotic microsurgery A. Gudeloglu, B. Ozdemir, S.R. Fleischman, P. Dahm

Awareness of the ideal recommendations among the members of the Robotic-Assisted Microsurgery and Endoscopic Society (RAMSES)  A. Gudeloglu, S.R. Fleischman, P. Dahm

DPhil place available at NDORMS, Oxford: Big data methods and evaluation of surgical interventions

primary-logoProject outline

Surgical interventions can have profound effects on people’s lives, both in terms of benefits and (potentially) in terms of risks or complications. It is therefore essential that surgical interventions should be evaluated appropriately and accurately to inform clinical decisions.

Although randomized controlled trials (RCT) provide gold standard evidence, surgical RCTs are expensive, challenging (given pragmatic, ethical, and other considerations), and hence scarce in the scientific literature. In addition, RCTs tend to use restrictive eligibility criteria therefore limiting the external validity (generalizability) of their findings to relatively healthy populations.

Conversely, routinely collected data (i.e. anonymized data from national registries, audits, and computerized NHS records) provide a unique opportunity to evaluate surgical interventions in potentially any NHS user, widening our ability to gather information on the performance of surgical and medical alternatives in different patient groups. However, given the observational nature of such studies, confounding (by indication) is a major challenge that needs to be accounted for to avoid biased results. Methods exist to minimize such confounding and other biases, which are widely used in pharmaco-epidemiology (drug safety studies), but these have been seldom tested for the study of surgical treatments.

The project will focus on two clinical scenarios for the risk-benefit evaluation of 1.two different surgical alternatives (unicompartmental vs total knee replacement) for the treatment of knee arthritis, and 2.a surgical procedure (bariatric surgery) compared to no surgery for the management of severe obesity. A number of different methodological approaches will be examined and compared (including feature selection methods to identify predictive factors) and propensity score, disease risk scores, instrumental variables, and case-only designs to study the risks and benefits of these treatments. These findings will be supplemented with simulation studies to investigate the performance and robustness of the different design parameters and statistical methods proposed in different scenarios, and to offer solutions to the challenges encountered while analyzing big data.

The project will use routinely collected data from the UK Clinical Practice Research Datalink, Hospital Episode Statistics (HES), and National Joint Registry, as well as bespoke (created ad-hoc) simulated datasets.

Research Group

The DPhil will be jointly supervised by Associate Prof Prieto-Alhambra, Dr M Sanni Ali, and Dr Sara Khalid, (all members of the NDORMS ‘Musculoskeletal Pharmaco- and Device epidemiology’ research group), and Prof Gary Collins, Professor of Medical Statistics and Deputy Director of the Oxford Centre for Statistics in Medicine (CSM).

Associate Prof Daniel Prieto-Alhambra has published extensively in the field of pharmaco-epidemiology, and is recognised internationally as an authority on use of routine data for musculoskeletal pharmaco- and device epidemiology.

Dr M Sanni Ali is a Senior Research Associate in Pharmaco-epidemiology. He has extensive expertise in the use, validation and development of pharmaco-epidemiological methods, both for the analysis of routinely collected data as well as in simulated datasets.

Dr Sara Khalid is lead analyst for the Health Services Delivery and Musculoskeletal Pharmaco-epidemiology group. She has an Oxford DPhil in Engineering Science, and has an excellent track record and experience in the use of big data methods including machine learning and similar methods.

Prof Gary Collins‘ research interests are focused on methodological aspects surrounding the development and validation of multivariable prediction (prognostic) models and has published widely in this area. He has a particular focus on the role that big data has in evaluating prediction models.

Current DPhil Students within the pharmaco-epidemiology group: 3


The Botnar Research Centre plays host to the University of Oxford’s Institute of Musculoskeletal Sciences and Centre for Statistics in Medicine.

Training will be provided in relevant related research methodology, including the handling and analysis of large datasets, and advanced statistical techniques. Attendance at formal training courses will be encouraged, and will include the “Real world epidemiology Oxford summer school” and the “Advanced musculoskeletal epidemiology UK-RIME summer school”.

In addition, courses from the Oxford Learning Institute and the Oxford University Computer Sciences on key skills for the completion of a successful DPhil thesis will be available. Additional on the job training opportunities will arise, and the supervisors will encourage the student to pursue such opportunities.

A core curriculum of lectures organized departmentally will be taken in the first term to provide a solid foundation in a broad range of subjects including epidemiology, health economics, and data analysis.

Students will attend weekly seminars within the department and those relevant in the wider University.

Students will be expected to present data regularly to the department, the research group and to attend external conferences to present their research globally.

Further information

Associate Prof D Prieto-Alhambra:

New reporting guidance for surgical case reports (SCARE) & case series (PROCESS)

To address a lack of guidance for authors of papers reporting case reports and case series in surgical research Riaz Agha, Department of Plastic Surgery, Guy’s and St. Thomas’ NHS Foundation Trust, London, UK and colleagues have developed 2 consensus based guidelines.

For Case Reports

  • CARE Guideline supports transparency and accuracy in publication of case-reports.
  • However, the CARE guidelines are not tailored to surgery.
  • Delphi-consensus exercise was used to develop Surgical CAse REport (SCARE) Guidelines.
  • Journals will be encouraged to endorse the SCARE guideline.

To cite the guidelines:
Agha RA, Fowler AJ, Saetta A, Barai I, Rajmohan S, Orgill DP, for the SCARE  Group. The SCARE Statement: Consensus-based surgical case report guidelines.  International Journal of Surgery 2016;34:180-186. 

Download the paper and checklist from the SCARE Statement website




Agha R, Fowler AJ, Saetta A, Barai I, Rajmohan S, Orgill DP on behalf of the SCARE Steering Group and the Academic Surgical Collaborative.  A Protocol for the Development of Reporting Criteria for Surgical Case Reports: the SCARE Statement.  International Journal of Surgery 2016;27:187-189.  PMID: 26828281.

Preferred reporting of case series in surgery (PROCESS) 

To cite the guidelines:
Agha RA, Fowler AJ, Rajmohan S, Barai I, Orgill DP and the PROCESS Group.  The PROCESS Statement: Preferred Reporting of Case Series in Surgery. International Journal of Surgery 2016; (article in press).

Download the paper and checklist from the PROCESS Statement website


  1. Agha R, Fowler AJ, Lee SY, Gundogan B, Whitehurst K, Sagoo H, Jeong KJL, Altman DG and Orgill DP. A Systematic Review Protocol for Reporting Deficiencies within Surgical Case Series.  BMJ Open 2015;5:e008007.  PMID:26438134.
  2. Agha R, Fowler A, Lee S, Gundogan B, Whitehurst B, Sagoo H, Jeong KJL, Altman D and Orgill D. A Systematic Review of the Methodological and Reporting Quality of Care Series in Surgery.  British Journal of Surgery 2016 (article in press). PMID: 27511619.


Special Surgical Theme issue of The Lancet features IDEAL


“Today’s surgical-themed issue of The Lancet, published ahead of the American College of Surgeons’ Clinical Congress in Washington, DC, USA, (Oct 16–20, 2016), contains three diverse surgical randomised trials (IDEAL Stage 3): early mobilisation compared with standard care for patients in the surgical ICU; lens extraction compared with laser peripheral iridotomy in atients with primary angle-closure glaucoma; and vertebroplasty compared with sham procedure in patients with osteoporotic fractures. All three are success stories of surgical research, supporting changes in practice and offering the possibility to improve the lives of thousands of patients every year.”

“Laurent Lantieri and colleagues report the prospective results of a programme of facial allotransplantation including long-term follow-up of seven transplanted patients (IDEAL Stage 2a) and they ask readers “should we continue?”. As J Rodrigo Diaz-Siso and Eduardo D Rodriguez discuss in their accompanying Comment, the promising outcomes reported in this small case series mean that the focus should now shift to outcome reporting and comparison with the standard(s) of care. This transition would be described as a move from the development and exploration stages to the assessment stage of evaluation by Peter McCulloch and the Balliol Collaboration in the Idea Development Exploration Assessment Long-term study (IDEAL) recommendations for surgical innovation.”

 A workshop on the future of surgery held at the UK Royal College of Surgeons, London on Sept 20, explored the reasons why the transition from development (2a) and exploration (2b) to assessment (3) of surgical innovations now required by the technique of facial allotransplantation so often proves difficult. The personal and professional investment of individual surgeons in innovations seems to play a key part. In comparison to drug company research, individual surgeons play a greater part in the evaluation of techniques or devices, effectively functioning as gatekeepers for innovation. Recruiting patients to stage 1 or 2 trials requires a familiarity with and a belief in the innovation, such that surgeons might begin to feel equipoise no longer exists, or could continue to iteratively refine the technique, sometimes unnecessarily.”

“Failure to make a timely transition to innovation assessment has many implications: an ethical failure of responsibility to trial participants, a high likelihood of research waste, the risk that a superior technique is ignored, and the possibility that rapid uptake of an attractive new technique—recommended by an eminent surgeon, for example—is such that a randomised comparison is no longer possible.”

ACook “In these circumstances emphasising uncertainty rather than equipoise, which suggests the clinician has equal levels of doubt about the efficacy of the two approaches, can be a way to get even the most polarised surgeons to collaborate on a trial”, said Andrew Cook, lead on surgery and interventional procedures for the UK National Institute of Health Research Health Technology Assessment programme at the Royal College meeting.

Weill Cornell Medical College Art Sedrakyan July 15, 2014

Weill Cornell Medical College
Art Sedrakyan
July 15, 2014

“This theme of improving the quality of surgery is also picked up by Art Sedraykan (Vice-Chair, IDEAL Collaboration) and colleagues, who discuss the importance of surgical registries. Rigorous registries are clearly vital for the transparent, long-term study of all new innovations but Sedraykan and colleagues go further, suggesting that almost all surgical patients should be entered into registries. Major national all-inclusive registries provide high-quality feedback to surgeons on their own outcomes, and have a proven role in facilitating the improvement they suggest.”

(Emphasis and specific addition of IDEAL Stages added by IDEAL)