Prospective development studies for novel procedures: Call for ideas

Do you know of a surgical or interventional procedure in development that needs better evidence to support its use? Would you be interested in working with IDEAL in developing and carrying out a prospective development study?

The IDEAL recommendation for procedures in the Development stage (2a) is the prospective development study (PDS).  The Development stage is when a technique or device is still being modified iteratively as a result of lessons learned on a case by case basis in the first cohort of patients.  Traditionally, published research on procedures at this stage has been in the form of the retrospective case series, which is criticised for its significant selection bias and other flaws. The prospective development study reduces the risk of bias but also provides more complete information about the process of innovation which allows the  reader to understand how the technique evolved to its current state and how applicable it may be to other settings. There are few procedures that have been published as prospective development studies, but two recent examples include minimally invasive oesophagectomy[1], published by Jane Blazeby and colleagues in 2011, and focal ablation of prostate cancer[2], published by Mark Emberton and colleagues in 2012.

No specific formal outline of a PDS exists, but IDEAL recommends that it includes the following:

  • Registered study protocol
  • Clear description of patient selection criteria
  • Recording of proportion of eligible patients selected
  • Ethical consideration for reasonable precautions against harm
  • Clear description of procedure
  • Sequential reporting of ALL cases over time, showing outcome in each case
  • Description of changes to procedure and indication,showing their  timings on the sequential display of cases
  • (Where appropriate) Use of statistical process control (SPC) methods
  • Relevant outcomes described, including unexpected outcomes
  • Use of standardised outcome measures

What other features should be included in a prospective development study? In order to improve the quality of surgical evidence and propagate the use of prospective development studies, ultimately phasing retrospective case series out of the literature, IDEAL needs your input.  We would be interested in working with suitable partners who are developing surgical innovations at this early stage,  to develop and report your work in the way we describe, and in interviewing you afterwards to develop a qualitative analysis of the strengths and weaknesses of the IDEAL Recommendations for this type of study. If you would like to explore this possibility, please contact the IDEAL Collaboration.

To get in touch, you can leave a reply on this post below, use our contact form (scroll to bottom of page) or tweet/ direct-message @IDEALCollab on Twitter.


[1] Blazeby JM, Blencowe NS, Titcomb DR, Metcalfe C, Hollowood AD, Barham CP. Demonstration of the IDEAL recommendations for evaluating and reporting surgical innovation in minimally invasive oesophagectomy. Br J Surg. 2011 Apr;98(4):544-51. doi: 10.1002/bjs.7387. Epub 2011 Jan 18.

[2] Ahmed HU, Hindley RG, Dickinson L, Freeman A, Kirkham AP, Sahu M, Scott R, Allen C, Van der Meulen J, Emberton M. Focal therapy for localised unifocal and multifocal prostate cancer: a prospective development study. Lancet Oncol. 2012 Jun;13(6):622-32. doi: 10.1016/S1470-2045(12)70121-3. Epub 2012 Apr 17.

Sham surgery: Is it inherently unethical?

One of the general criticisms of surgical randomised controlled trials is the limitation of blinding. It is impossible to blind the surgeon performing the procedure, and most of the time it is also impossible, impractical or arguably unethical to blind the patient. Sham surgery has been suggested and in fact attempted as a potential method for addressing the placebo effect of surgery.

The first instance of a trial involving a sham surgery was published in the New England Journal of Medicine in 1959 by Cobb and colleagues[1]. Prior to the study, it was theorised that angina pectoris from coronary artery disease would be alleviated by ligating the internal mammary artery, presumably by increasing coronary flow through collateral arteries. In the trial, patients were taken to the operating room and put under local anaesthesia, at which time the operating surgeon would open an envelope containing a piece of paper randomising the patient to either internal mammary ligation (involving bilateral parasternal incisions and isolation and ligation of the internal mammary arteries), or a sham procedure, which was identical at every step except after the arteries were isolated they were not ligated. The study showed that there was no difference in angina symptoms between the groups, and the procedure soon fell out of favour as a result.

More recently, studies involving placebo operations have been published sporadically in the surgical literature. In 2002, Moseley et al[2] randomised patients with osteoarthritis to one of two arthroscopic procedures (lavage or débridement), or to a sham surgery (see a video interview with Moseley here). They found that there was no difference in outcomes (pain) amongst the groups. A number of trials[3],[4] in the early 2000s utilised sham surgery as a control in treating patients with Parkinson’s disease with fetal neural tissue transplantation, which sparked debates surrounding the ethics of sham surgery. Macklin argues that placebo drugs used in pharmaceutical trials must have no known adverse effects[5], so why should surgical trials be different? Non-maleficence is, after all, a central tenet of medical ethics. According to Macklin, even though placebo surgery does have benefits, it does not outweigh the risks, and is inappropriate because the sham surgery itself is not used therapeutically as a result of the trial.

Others are more forgiving of the rationale behind placebo surgery. In the now classic 1961 JAMA paper ‘Surgery as Placebo’[6], Beecher describes a quantitative analysis of the placebo effect in surgery, acknowledging that it exists and may even have some benefit, but urging caution in its’ use, to “question the moral or ethical right to continue with casual or unplanned new surgical procedures”. One might take it a step further and ask: If it is true that the placebo effect exists and shows some benefit, and the patient consents to a possible placebo surgery, then what’s the problem?

Bioethicist Franklin Miller describes three misconceptions in the argument of sham surgery ethics[7]:

  1. Confusion over clinical research ethics versus clinical medicine ethics
  2. Taking one highly publicised, controversial case, such as fetal tissue transplantation for Parkinson’s disease, to be paradigmatic of all sham surgery (not taking into account the possibility that less risky sham procedures might present a more favourable risk-benefit ratio in terms of the research)
  3. Misinterpretation of the ethical requirement for clinical research to minimize patient risk

As with many discussions centered around ethics, this one tends to raise more questions than answers. Interestingly, in a similar trial[8] to Moseley’s, published by Kirkley and others six years later, outcomes were similar, with no difference between patients randomised to surgery and patients who underwent medical treatment and physical therapy only. In Kirkley’s trial, however, no sham surgery was used. The Kirkley study also addressed some of the methodological concerns of the Moseley paper, by using validated outcome measures and a more diverse patient population to increase external validity. From this, one might argue that a placebo study was not necessary in this case, since the same conclusion was reached in a trial that was not placebo-controlled.

What do you think? Is sham surgery inherently unethical?


[1] Cobb LA, Thomas GI, Dillard DH, Merendino KA, Bruce RA. An evaluation of internal-mammary-artery ligation by a double-blind technic. N Engl J Med. 1959 May 28;260(22):1115-8.

[2] Moseley JB, O’Malley K, Petersen NJ, Menke TJ, Brody BA, Kuykendall DH, Hollingsworth JC, Ashton CM, Wray NP. A controlled trial of arthroscopic surgery for osteoarthritis of the knee. N Engl J Med. 2002 Jul 11;347(2):81-8.

[3] Olanow CW, Goetz CG, Kordower JH, Stoessl AJ, Sossi V, Brin MF, Shannon KM, Nauert GM, Perl DP, Godbold J, Freeman TB. A double-blind controlled trial of bilateral fetal nigral transplantation in Parkinson’s disease. Ann Neurol. 2003 Sep;54(3):403-14.

[4] Freed CR, Greene PE, Breeze RE, Tsai WY, DuMouchel W, Kao R, Dillon S, Winfield H, Culver S, Trojanowski JQ, Eidelberg D, Fahn S. Transplantation of embryonic dopamine neurons for severe Parkinson’s disease. N Engl J Med. 2001 Mar 8;344(10):710-9.

[5] Macklin R. The ethical problems with sham surgery in clinical research. N Engl J Med. 1999 Sep 23;341(13):992-6.

[6] Beecher HK. “Surgery as placebo. A quantitative study of bias”. JAMA 1961 176 (13): 1102–7.

[7] Miller FG. Sham surgery: an ethical analysis. Sci Eng Ethics. 2004 Jan;10(1):157-66.

[8] Kirkley A, Birmingham TB, Litchfield RB, et al. A randomized trial of arthroscopic surgery for osteoarthritis of the knee. N Engl J Med. 2008;359:1097-1107.

Recent RCS conference offers advice for surgeons new to research

Last week, the Royal College of Surgeons of England held a conference at their headquarters in London, entitled “Surgical Research in the 21st Century: New Opportunities and Challenges“, to encourage young surgical trainees to get involved wtih research. IDEAL chair Peter McCulloch was invited to speak at the event among other influential figures in surgical research, listed below.

In an electronic vote held on the day, 83% of attendees said they would be more likely to embark on surgical research if they had a framework to guide them about research study design and reporting and 89% indicated they would be more likely to do so if there was a network of potential collaborators to work with.  Not coincidentally, IDEAL aims to help in both areas, and accordingly, Peter’s talk, “Problems in Clinical Research in Surgery, and IDEAL solutions”, outlined what IDEAL can offer at each stage of surgical research (slides available for download here).

Following talks on available funding for surgical research and how best to apply for it by utilising a wealth of specialist methodologists and others, the conference then focused on answering relevant questions from young research-oriented surgeons.

Below is a summary of some of the key questions asked and the advice offered by the main speakers:

Q. “I can’t get funding when I have no track record – how do I start from nothing?”
A.  You need to collaborate with a proven team – IDEAL hopes to gather a list of potential collaborators to put people in contact with each other to develop high quality applications (PMcC)

Q. “How do I become a PI?”
A. You need to join research active teams and collaborate – look for specific funding calls in your area and express interest (TT)

Q. “If I collaborate with others, I’m concerned they will steal my ideas”
A. You won’t get funded as an inexperienced individual – you could be a PI using your idea if you seek funding in conjunction with a well-supported team, especially in clinical trials (DM)

Q. “I have previously applied for funding for a complex study with help of the Research Design Service (RDS) but the funding review said it was too complex to do”
A. You MUST include a feasibility study demonstrating your team’s ability to recruit and randomise and progress the study and show how you will do it (DM)

In sum: “Opportunities for surgical research are huge now – but they must be high quality. This means they must be collaborative” (FH)

Q. “Is there an upper limit to an application to the Health Services and Delivery Research (HSDR) funding stream?”
A. There is no upper limit – typically applications are around £500K – surgeons could consider applications in this area more. Tips – focus on relevance, clarity, people and future action. Look out for HSDR briefing events when calls are out. (TL)

Q. “Does the RDS charge for services and how long does an application take?”
A. There is no charge (it is NIHR-funded) and can help with whatever funding source is being applied for not just NIHR. Time varies. Leave plenty of time for developing high quality application. Tips – ensure early involvement of multi-disciplinary team members (AB) (More information at the RDS section on the NIHR website)

Q. “I am unclear on the roles of various advisory organisations for applying for grants RDS, Clinical Trial Units (CTUs), etc. Who should be approached first?”
A. They are complementary – it depends on where you are located and the capacity of each organisation in your area. In all cases allow plenty of time to develop a good proposal (DM) (Useful links at the NIHR website)

Q. “Surgeons are interested in costs but we don’t know how to incorporate in research. Are there health economists we can ask to help?”
A. Yes, there will be health economists in most universities – just make contact at early stage (GB)

Q. “Should all complex intervention studies (including surgery) include a qualitative aspect?”
A. This can be valuable in most cases (GM).

Feasibility studies should incorporate qualitative elements in examining delivery and evaluation of the intervention (DM)

And finally, a tip from Sir Muir Gray (Chief Knowledge Officer, NHS) in his closing remarks: Use casebooks to show how you are contributing to the processes of service development.

PMcC: Mr Peter McCulloch (Chair, IDEAL Collaboration)
DM: Professor Dion Morton (Clinical Director of Research, RCS)
TT: Professor Tom Treasure (Clinical Operational Research Unit, UCL)
TL: Tara Lamont (Scientific Advisor, NIHR HS&DR programme)
AB: Mr Andrew Barton (Senior lecturer in Healthcare methodology, Research Design Service)
FH: Professor Freddie Hamdy (Director, RCS Surgical Trials Unit, Oxford)
NB: Mr Nigel Beasley (Deputy Medical Director, Nottingham University Hospital)
GB: Professor Gwyn Bevan (Professor of Policy Analysis, London School of Economics)
GM: Professor Graham Martin (Professor of Health Organisation and Policy, University of Leicester)

by Allison Hirst with Rachel Kwon

IDEAL’s presence at Evidence Live 13: Medical device regulation talk by Peter McCulloch

IDEAL chair Peter McCulloch was an invited speaker at Evidence Live 13, an international conference dedicated to the practice of evidence-based medicine held in Oxford in late March. Peter’s talk was focused on the relevance of IDEAL to the future of medical device regulation and specific applications of the framework to that domain.

Medical devices, especially high-risk implantable devices such as artificial joints, and cardiovascular stents, grafts and valves have a lot in common with surgical operations when it comes to innovation and evaluation.  Like operations, these devices need an early “tinkering” phase of rapid iterative modification before they are ready for definitive trials, and implanting and using them successfully often requires an operator learning curve similar to that for an operation.  And like surgery, devices create strong user preferences, and need long term surveillance to detect rare or late problems, and to detect whether the indications for use are not “creeping” with time. Furthermore, both surgical procedures and devices are currently subjected to a much lower evidence bar than drugs before they can be marketed – they only have to demonstrate safety rather than efficacy – hence the incentive to provide good evidence of efficacy is weak.

Since the development processes and the regulatory environments for both surgery and devices are so similar, it makes sense to apply the IDEAL Framework and Recommendations to the evaluation of devices, and doing so can give us a guide as to how a more rational regulatory system might demand evidence on which to base decisions.  In fact this idea has already been explored in a meeting between IDEAL and the FDA in Washington DC in December 2011 and some of the recent guidance from the FDA closely resembles some of the IDEAL Recommendations.  In his talk, Peter suggested three ways in which IDEAL recommendations and proposals could be taken further, and so help in the development of a more logical regulatory system based on ethical principles.

The first of these is the recommendation that all “first in man” studies be registered and available to those with a legitimate interest in accessing the details.  This is appropriate on ethical grounds, as we should ensure others benefit from both our successes and our failures in experimenting on patients.  To avoid the registry from becoming simply a feeding ground for the medico-legal profession, however, it would need a facility for confidential reporting of adverse outcomes backed by a powerful body able to resist claims for discovery.  Such systems exist in aviation and other industries, justified by the need to share information for safety reasons.  A registry might prove useful in other ways.  By collating reports from many sources, it might allow an expert panel charged with confidential analysis of the reports the opportunity to identify themes and trends, point to possible causes of risk and harm and identify unexpected connections between studies, leading to new ideas.  Registering a new device would be a powerful piece of evidence of primacy in its development, which might prove useful in future disputes around intellectual property.  Over time, it might well become an ethical obligation on innovators to search the registry for similar devices before trying one out on humans for the first time, to avoid repeating adverse outcomes.

The second IDEAL principle which might advance regulations of devices is the proposal that regulators should provide systems to allow rapid, flexible and expert evaluation of Development stage devices.  This is necessary because of the rapidity with which devices alter during this stage, and the impossibility of conducting definitive trials.  Any human studies at this stage must therefore accept an element of risk, and be based on two principles: (1) that subjects are as fully informed and warned as possible, and (2) that regulatory panels have the expertise and the integrity to decide whether a proposed intervention is a reasonable risk, and give permission for it to proceed.  The logical consequences of this idea are interesting.  The rapidity required could only occur if regulatory decisions were delegated to a low level: a district or a hospital, for instance.  Likewise, the current multiple regulatory jeopardy would need to be replaced by a single body acting for all the agencies with an interest in the experiment.  Such local panels would be able to convene and decide rapidly, but they would not have universal expertise, so they would need to be able to call on a central fund of specialist advice for specific questions.  Finally, there would need to be a system for oversight of these local Committees, to avoid the development of heterogeneity in the decisions made in different areas, which left unchecked could become partial, obstructive or reckless.

The third IDEAL principle highlighted in Peter’s talk has the most far-reaching implications of all.  The proposal that provisional approval of a device for marketing should be contingent on the demonstration of appropriate evidence of efficacy and safety would, if developed logically, lead to the abolition of the current distinction between pre-market and post-market devices.  Instead of a simple watershed, a system of graded market licensing would develop, based on the IDEAL framework stages.  At each stage the appropriate type of study would be required of the manufacturer, and the device could be marketed only within the confines of this study type, whose results would need to be reported to the appropriate regulator.  Thus Development stage devices could only be used in prospective development studies (a few dozen cases at most) whereas Assessment stage devices could be marketed within large randomised trials.  Once a randomised trial against conventional treatment had confirmed efficacy, a device could be marketed generally, but information to allow surveillance using large international registries would still be required.  To allow this degree of evaluation, the only feasible funding method would be a tax on the manufacturers of devices proportional to their expected profit.  Where no profit could be expected, the government might have to bear the costs.  To avoid the problems of data configuration which often prevent full access to and comparison of  commercial data at present, the registries concerned would be run by public or collaborative bodies but funded by commercial ones who were profiting from the devices studied in them.

Two unexpected benefits of this system might reward companies willing to invest in innovation.  First, any company wishing to produce a “me too” device would be required to develop its evidence in exactly the same way as the originator.  This would give important commercial advantages to the first player in the market, similar to those currently enjoyed via the patents system by drugs.  It would also mean that small companies with good ideas which they could not exploit would sell themselves at a profit to larger ones who could, again in a similar way to drug developers.  Although this would make it more difficult for smaller firms to grow, it would ensure that devices were produced with a solid scientific evidence base.  Second, the need for flexible international registries funded by a levy on commercial companies would provide an incentive for cooperation on a unique device identifier programme, which would allow devices to be followed across their life history regardless of manufacturer or country.

The changes suggested here are far-reaching and present major challenges to the status quo, but they represent the logical consequences of recommendations based only on a consistent view of medical ethics and a framework which simply describes the realities of innovation in surgical, interventional and device therapy.

IDEAL extends official support to AllTrials

Around half of all clinical trials have not been published; some trials have not even been registered.

One of the barriers to generating a high-quality evidence base is the wealth of unreported, unpublished data. Additionally, clinical trials are often undertaken without being registered in a centralised database, leading to unnecessary duplication of research. All Trials Registered, All Results Reported, a recently-launched joint initiative by the British Medical Journal, Bad Science author and whistleblower Ben Goldacre, IDEAL collaborators Sir Iain Chalmers for the James Lind Initiative and Carl Heneghan of Oxford’s Centre for Evidence-Based Medicine, and others, highlights both of these barriers and calls for registration and reporting of all clinical trials.

The IDEAL collaboration has been a proponent for prospective registries for surgical procedures, particularly in the Innovation phase where similar novel interventions may be developing in parallel to each other, but also in the other stages of the IDEAL framework through the Long-term study phase when quality assurance and monitoring are issues.

The call to report all results and methods is also vital to promoting comprehensive evidence of the highest possible quality. Suppression of negative trials is a well-known source of publication bias, in both the medical and surgical literature. Surgical interventions are particularly vulnerable, as new procedures or new methods of doing established procedures often rely on tinkering or making small modifications, which are seldom formally reported, and even more rarely when they fail. By making such reporting mandatory, in conjunction with a non-punitive approach such as anonymous and/or confidential reporting, this bias could be significantly reduced or even eliminated.

The AllTrials campaign represents a prime example of multiple stakeholders collaborating in order to effect change in the development of reliable clinical evidence. The BMJ in particular has been instrumental in promoting open data.

Even industry is showing its support: pharmaceutical giant GlaxoSmithKline made headlines yesterday by officially announcing its support for the campaign, in a move that many are hoping will be a precedent for others in industry to follow and improve data transparency.

Supporting parties can sign the petition and find more information at

Have a comment? Use the form below. IDEAL members can also join the discussion in our forum.

IDEAL commentary on reform of device regulation in latest BMJ

“Now is the time for radical change” for regulating medical devices in the EU, says IDEAL founder Peter McCulloch in an editorial published in the latest issue of the British Medical Journal.

Members of the IDEAL collaboration have been instrumental in the proposed reform to device regulation in both the US, as seen by the recent publication of the Food and Drug Administration’s report on postmarket surveillance, as well as in the EU with recent commentary in The Lancet and now the BMJ. Continue reading

IDEAL involvement in new US Food and Drug Administration plan for device surveillance

The US Food and Drug Administration has published a new plan for postmarket surveillance of medical devices in accordance with IDEAL recommendations.

The plan is outlined in a 26-page report (available for download from the FDA website) entitled Strengthening Our National System for Medical Device Postmarket Surveillance and was the result of a public meeting held by the FDA’s Center for Devices and Radiological Health on September 10, 2012.

The report was also based in part on discussion from an earlier FDA workshop held in December 2011 on application of IDEAL recommendations to device regulation and evidence development. Several key collaborators in the IDEAL initiative were present at both the initial meeting as well as last month’s workshop.

Among the four proposed actions outlined in the report are the promotion of national and international device registries and the development of new methodology for generating and appraising the evidence. In an editorial published in the latest issue of the British Medical Journal in response to the report, Harvard professor of healthcare policy Sharon-Lise Normand and others point out the challenge of the latter:

Surveillance… involves observation of
outcomes of medical devices used by clinicians in routine
practice. It is difficult to agree on what constitutes sufficient
evidence to warrant public health concern.

The editorial was published in response to the FDA report and summarises the four changes detailed in the plan.

Though the IDEAL collaboration originally arose from the need for better evaluation of surgical innovation, it soon became clear that many of the same challenges and limitations were applicable to devices and implants used in procedures. This new report by the FDA is an important step forward in the larger debate concerning device regulation and patient safety.


  1. Normand SL, Hatfield L, Drozda J, Resnic FS. Postmarket surveillance for medical devices: America’s new strategy. BMJ 2012;345:e6848.
  2. FDA. Strengthening Our National System for Medical Device Postmarket Surveillance, September 2012.
  3. FDA. Transcript for Public Workshop – Bridging the IDEAL and TPLC Approaches for Evidence Development for Surgical Medical Devices and Procedures, December 2, 2011; accessed 17/10/12.

UPDATE: Lancet publishes IDEAL collaboration’s response letter to President of Royal College of Surgeons

The debate on regulation of surgically-implanted devices continues to draw attention, as seen by the Lancet‘s recent publication of the IDEAL collaboration’s response letter to Professor Norman Williams, President of the Royal College of Surgeons.

Professor Williams’ statement, originally published in the Lancet on 12 May 2012, was a response to an editorial by Richard Horton, editor-in-chief of the Lancet, criticising the current system of device regulation. He suggests a mandatory database for both devices and techniques while also acknowledging that regulation must balance innovation, which could arguably be stifled with over-regulation.

…unlike the rigorous regulatory system in place for drugs, the assessment and regulation of medical devices is failing patients…we should, as a minimum, have mandatory databases for all surgical implants and associated techniques which would provide ongoing patient safety data, while enabling trusts to make evidence-based procurement decisions.

The collaboration agrees with Professor Williams, and recommends that IDEAL methodology be applied to device regulation. As outlined in the index paper on IDEAL recommendations, it is not enough for manufacturers of novel devices to simply show that their product will not fail; they must also prove efficacy, and demonstrate to what level the device is effective.

We support the call for adequate mandatory clinical outcome databases for existing implants, and for regulatory evaluation for new ones which strikes a balance between fostering innovation and protecting patients.

This response letter was initially posted here on the IDEAL website prior to publication in the Lancet. You can view the story here.