2019 Update of the IDEAL Framework

The Collaboration held its first international annual conference in 2016 at St Catherine’s College, Oxford which included an additional day meeting to address updating the IDEAL Framework. A writing group worked to produce the update.

This comprehensive update proposes several modifications. First, a “Pre-IDEAL” stage describing preclinical studies has been added. Second, it discusses potential adaptations to expand the scope of IDEAL (originally designed for surgical procedures) to accommodate therapeutic devices, through an IDEAL-D variant. Third, we explicitly recognise the value of comprehensive data collection through registries at all stages in the Framework and fourth, we examine the ethical issues that arise at each stage of IDEAL and underpin the recommendations. The Recommendations for each stage are reviewed, clarified and additional detail added.

Hirst A, Philippou Y, Blazeby J, Campbell B, Campbell M, Feinberg J, Rovers M, Blencowe N, Pennell C, Quinn T, Rogers W, Cook J, Kolias AG, Agha R, Dahm P, Sedrakyan A, McCulloch P. No Surgical Innovation Without Evaluation: Evolution and Further Development of the IDEAL Framework and Recommendations. Ann Surg. 2019 Feb;269(2):211-220. doi: 10.1097/SLA.0000000000002794. PMID: 29697448.

Summary of updates

The table below provides details of the updates to the IDEAL Framework in 2019.

Download this table as a PDF

Jump to:
Stage 0 | Stage 1 | Stage 2a | Stage 2b | Stage 3 | Stage 4

Stage of innovation Updated IDEAL Framework  Recommendations for researchers Research environment

Stage 0

Pre-clinical

Pre-IDEAL was not described in original IDEAL framework

 Purpose: Feasibility and definition of procedure
Number & Types of Patients: Pre-clinical
Number & Types of Surgeons: Very few; innovators
Output: Description addressing:

  • Whether intended goal of procedure is accomplished
  • Level of difficulty of performing procedure or using device as compared to standard of care
  • Safety risks
  • Desirability of intervention

Method: Various, including simulator, cadaver, animal, modelling and cost-effectiveness studies
Stage Endpoint: Any studies that could avoid predictable risks of failure or harm to the first human should have been conducted.

 All predictable risks to patients should be investigated before human studies begin

Guidelines on best scientific practice and ethics specific to the types of study should be followed where available

A minimum dataset describing technical consistency should be made public before first-in-human testing.

 Journals
Editors should require publication of the Pre-IDEAL minimum dataset before or together with first-in-human reportsRegulatory/legal
Regulators should develop a definition of the minimum publically available dataset required to allow First-in Human studies of new devices to proceed.Ethical aspects
General standards of research integrity apply

Stage 1 Idea

First in human

 Purpose: Proof of concept
Number & Types of Patients: Single digit; highly selective.
Number & Types of Surgeons: Very few; innovators
Output: Description
Intervention: Evolving; procedure inception
Methods: Structured case reports
Outcomes: Proof of concept; technical achievement; dramatic success; adverse events, surgeon views of the procedure
Stage Endpoint: Outcomes will determine whether to proceed to stage 2a.		 Provide full details of patient selection, technique and outcomes and patients not selected during the time frame, and why.

Use standard well-defined measures for reporting outcome and patient characteristics

Use structured reporting system eg, SCARE checklist.

Make the above information available to peers regardless of outcome

 Journals:
Encourage or require registration of the innovation when considering for publication (E.g. IJS: Case Reports and www.researchregistry.com)Regulatory/legal:
Provide public interest defence from legal discovery for registries specifically for first-in-human studies.
Ensure local hospital policy on innovative procedures groups foster innovation (i.e., IRB or new procedure committee)Ethical aspects:
multiple strategies required to minimise harms to patients, including formal human research ethics approval for selected planned interventions

Stage 2a Development

Single centre/single intervention; case series/prospective cohort

 Purpose: Development of procedure
Number & Types of Patients: Few; Selected
Number & Types of Surgeons: Few; innovators and some early adopters
Output: Technical description of procedure and its development with reasons for changes
Intervention: Evolving; procedure development
Methods: Prospective development studies
Outcomes: Mainly safety; technical and procedural success
Stage Endpoint: Procedure should be refined enough to allow replication in Stage 2b and there should be no intent to make further major modifications		 Make protocol for study available

Use standard well-defined measures for reporting outcome and patient characteristics

Report and explain all exclusions

Report all cases sequentially with annotation and explanation of when and why changes to indication or procedure took place.

Display main outcomes graphically to illustrate the above.

 Journals: Support for publication of Development study formats and protocols
Regulatory/legal: Ensure that patient consent includes information about known outcomes from Stage 1**,about unknown risks and –inform the patient that the surgeon has carried out few of the procedures previously
Ethical aspects: formal human research ethics approval required

Stage 2b – Exploration

Bridge from observational to comparative evaluation.
Purpose is to gain data to decide if and how to test in a robust RCT or other appropriate pivotal design.

 Purpose: Achieving consensus between surgeons and centres
Number & Types of Patients: Many; broadening indication to include all potential beneficiaries
Number & Types of Surgeons: Many; innovators, early adopters, early majority
Output: Effect estimate based on large sample; Analysis of learning curves; estimate of influence of pre-specified technical variants and patient subgroups on outcome.
Intervention: Stable; acceptable variants definedMethod: Prospective multi-centre exploration cohort study (disease or treatment based); pilot/feasibility multicentre RCTs.Outcomes: Safety; clinical outcomes (specific/graded); short-term outcomes; patient centred/reported outcomes; feasibility outcomesStage Endpoints: fall in to two main groups; Demonstrate that technique can be more widely adopted; and, Demonstrate that progression to RCT is desirable and feasible		 Make protocol for study available

Use standard well-defined measures for reporting outcome and patient characteristics

Participate in collaborative multi-centre co-operative data collection, incorporating feasibility issues such as:

  • estimating effect size,
  • defining intervention quality standards,
  • evaluating learning curves,
  • exploring subgroup differences,
  • eliciting key stakeholder values and preferences,
  • analysis of adverse events:

Pre-planned consensus meeting prior to progressing to an RCT to identify feasibility and ability to recruit, intervention and comparator definitions, appropriate patient selection criteria, primary endpoint.

 Funders:
Support Stage 2b Exploratory cohort studies as preliminary “pilot/feasibility” phases for RCT proposals.Journals:
Support publication of IDEAL Exploration studies and protocolsEthical aspects:Formal human research ethics approval required. Ensure that potential harms from the learning curve are minimised by training and mentoring prior to progressing to Stage 3

Stage 3 – Assessment

Definitive comparative evaluation of main efficacy and safety aspects of new technique against current best treatment.

 Purpose: Comparative effectiveness testing
Number & Types of Patients: Many; expanded indications (well-defined)
Number & Types of Surgeons: Many; early majority
Output: Comparison with current standard therapy
Intervention: StableMethod: RCT with or without additions/modifications; alternative designs (cluster, preference RCTs, stepped wedge, adaptive designs)Outcomes: Clinical outcomes (specific and graded); potentially Patient Reported outcomes, Health Economic outcomesStage Endpoints: two main endpoints; Clear valid evidence on relative effectiveness of innovation; and, identification of issues requiring long term monitoring.		 Register on an appropriate international register (e.g., clinicaltrials.gov)

Use standard well-defined measures for reporting outcome and patient characteristics

Incorporate information about patient and clinician values and preferences in consent information and outcome measure design

Reporting guidelines:

  • CONSORT update of 2010 with extension for non-pharmacological treatments
  • COMET
  • TIDieR
  • SPIRIT (for RCT protocol design)
 Funders: Support trial proposals incorporating preparatory Stage 2b work
Journals: Encourage authors to refer to work on innovation in prior IDEAL stages preceding RCT. Support use of appropriate reporting guidelines. Mandate registration of RCT in trials register prior to publication.
Ethical aspects: formal human research ethics approval required

Stage 4

Long term monitoring

 Purpose: Surveillance
Number & Types of Patients: All eligible
Number & Types of Surgeons: All eligible
Output: Description; audit; regional variation; quality assurance; risk adjustment
Intervention: Stable
Method: Registry; routine database; rare-case reports
Outcomes: Rare events; long-term outcomes; quality assurance
Additions:

  • Registries for devices – IDEAL-D
  • Registries at earlier stages of IDEAL
  • Registries may begin from the earliest stages of human use
  • Registry datasets should be defined by the clinical community with patient input
  • Datasets should be simple, cheap and easy to collect
  • Curation of registries by clinical community is desirable
  • Funding of registries should be agreed between government and commercial interests but kept separate from curation
  • Consent for use of registry data in research should be broad and where possible automatic
 Funders: Link funding for purchasing treatment to delivery of adequate long term follow up
Ethical aspects: resolve issues of consent for data use and especially for nested RCTs

 

Table provided With permission Wolters Kluwer Permissions Team (SF: 01700255)