Research methodology in surgery: issues in study design

Study design in surgery is challenged by many problems inherent in the nature of surgery compared with pharmaceutical interventions.

The articles listed below describe these problems and offer some solutions.

IDEAL publications:

McCulloch P, Cook JA, Altman DG, Heneghan C, Diener MK; IDEAL Group. IDEAL framework for surgical innovation 1: the idea and development stages. BMJ. 2013 Jun 18;346:f3012. doi: 10.1136/bmj.f3012. PubMed PMID: 23778427

Ergina PL, Barkun JS, McCulloch P, Cook JA, Altman DG; IDEAL Group. IDEAL framework for surgical innovation 2: observational studies in the exploration and assessment stages. BMJ. 2013 Jun 18;346:f3011. doi: 10.1136/bmj.f3011. PubMed PMID: 23778426

Cook JA, McCulloch P, Blazeby JM, Beard DJ, Marinac-Dabic D, Sedrakyan A; IDEAL Group. IDEAL framework for surgical innovation 3: randomised controlled trials in the assessment stage and evaluations in the long term study stage. BMJ. 2013 Jun 18;346:f2820. doi: 10.1136/bmj.f2820. PubMed PMID: 23778425

McCulloch P, Altman DG, Campbell WB, Flum DR, Glasziou P, Marshall JC, Nicholl J; Balliol Collaboration, Aronson JK, Barkun JS, Blazeby JM, Boutron IC, Campbell WB, Clavien PA, Cook JA, Ergina PL, Feldman LS, Flum DR, Maddern GJ, Nicholl J, Reeves BC, Seiler CM, Strasberg SM, Meakins JL, Ashby D, Black N, Bunker J, Burton M, Campbell M, Chalkidou K, Chalmers I, de Leval M, Deeks J, Ergina PL, Grant A, Gray M, Greenhalgh R, Jenicek M, Kehoe S, Lilford R, Littlejohns P, Loke Y, Madhock R, McPherson K, Meakins J, Rothwell P, Summerskill B, Taggart D, Tekkis P,  Thompson M, Treasure T, Trohler U, Vandenbroucke J. No surgical innovation without evaluation: the IDEAL recommendations. Lancet. 2009 Sep 26;374(9695):1105-12. doi: 10.1016/S0140-6736(09)61116-8. PubMed PMID: 19782876

Ergina PL, Cook JA, Blazeby JM, Boutron I, Clavien PA, Reeves BC, Seiler CM; Balliol Collaboration, Altman DG, Aronson JK, Barkun JS, Campbell WB, Cook JA, Feldman LS, Flum DR, Glasziou P, Maddern GJ, Marshall JC, McCulloch P, Nicholl J, Strasberg SM, Meakins JL, Ashby D, Black N, Bunker J, Burton M, Campbell M, Chalkidou K, Chalmers I, de Leval M, Deeks J, Grant A, Gray M, Greenhalgh R, Jenicek M, Kehoe S, Lilford R, Littlejohns P, Loke Y, Madhock R, McPherson K, Rothwell P, Summerskill B, Taggart D, Tekkis P, Thompson M, Treasure T, Trohler U, Vandenbroucke J. Challenges in evaluating surgical innovation. Lancet. 2009 Sep 26;374(9695):1097-104. doi: 10.1016/S0140-6736(09)61086-2. PubMed PMID: 19782875

Barkun JS, Aronson JK, Feldman LS, Maddern GJ, Strasberg SM; Balliol Collaboration, Altman DG, Barkun JS, Blazeby JM, Boutron IC, Campbell WB, Clavien PA, Cook JA, Ergina PL, Flum DR, Glasziou P, Marshall JC, McCulloch P, Nicholl J, Reeves BC, Seiler CM, Meakins JL, Ashby D, Black N, Bunker J, Burton M, Campbell M, Chalkidou K, Chalmers I, de Leval M, Deeks J, Grant A, Gray M, Greenhalgh R, Jenicek M, Kehoe S, Lilford R, Littlejohns P, Loke Y, Madhock R, McPherson K, Rothwell P, Summerskill B, Taggart D, Tekkis P, Thompson M, Treasure T, Trohler U, Vandenbroucke J. Evaluation and stages of surgical innovations. Lancet. 2009 Sep 26;374(9695):1089-96. doi: 10.1016/S0140-6736(09)61083-7. PubMed PMID: 19782874

Useful publications:


Agha RA, Camm CF, Edison E, Orgill DP. The methodological quality of randomized controlled trials in plastic surgery needs improvement: a systematic review. J Plast Reconstr Aesthet Surg. 2013 Apr;66(4):447-52. doi: 10.1016/j.bjps.2012.11.005. Epub 2012 Dec 14. Review. PubMed PMID: 23245758

Avery KN, Barham CP, Berrisford R, Blazeby JM, Blencowe NS, Donovan J, Elliott J, Falk SJ, Goldin R, Hanna G, Hollowood AD, Metcalfe C, Noble S, Sanders G, Streets CG, Titcomb DR, Wheatley T. Understanding surgical interventions in RCTs: the need for better methodology. Lancet. 2013 Jan 5;381(9860):27-8. doi: 10.1016/S0140-6736(13)60015-X. PubMed PMID: 23290962

Brody BA, Ashton CM, Liu D, Xiong Y, Yao X, Wray NP. Are surgical trials with negative results being interpreted correctly? J Am Coll Surg. 2013 Jan;216(1):158-66. doi: 10.1016/j.jamcollsurg.2012.09.015. Epub 2012 Nov 22. PubMed PMID: 23177270

del Junco DJ, Fox EE, Camp EA, Rahbar MH, Holcomb JB; PROMMTT Study Group.
Seven deadly sins in trauma outcomes research: an epidemiologic post mortem for  major causes of bias. J Trauma Acute Care Surg. 2013 Jul;75(1 Suppl 1):S97-103.
doi: 10.1097/TA.0b013e318298b0a4. PubMed PMID: 23778519

Hannink G, Gooszen HG, van Laarhoven CJ, Rovers MM. A systematic review of
individual patient data meta-analyses on surgical interventions. Syst Rev. 2013
Jul 5;2:52. doi: 10.1186/2046-4053-2-52. PubMed PMID: 23826895


Cook JA, Bruckner T, MacLennan GS, Seiler CM. Clustering in surgical trials–database of intracluster correlations. Trials. 2012 Jan 4;13:2. doi: 10.1186/1745-6215-13-2. PubMed PMID: 22217216

D’Souza R, Srinivasa S, Cribb B, Hill L, Hill A. A comparison of the colorectal surgical research across Europe, USA and Australasia. Colorectal Dis. 2012 Sep;14(9):1145-51. doi: 10.1111/j.1463-1318.2011.02852.x. PubMed PMID: 21985765

Diener MK, Simon T, Büchler MW, Seiler CM. Surgical evaluation and knowledge transfer–methods of clinical research in surgery. Langenbecks Arch Surg. 2012 Dec;397(8):1193-9. doi: 10.1007/s00423-011-0775-x. Epub 2011 Mar 22. Review. PubMed PMID: 21424797

Garas G, Ibrahim A, Ashrafian H, Ahmed K, Patel V, Okabayashi K, Skapinakis P, Darzi A, Athanasiou T. Evidence-based surgery: barriers, solutions, and the role of evidence synthesis. World J Surg. 2012 Aug;36(8):1723-31. doi: 10.1007/s00268-012-1597-x. Review. PubMed PMID: 22535211

Jacobs WC, Kruyt MC, Verbout AJ, Oner FC. Effect of methodological quality measures in spinal surgery research: a metaepidemiological study. Spine J. 2012  Apr;12(4):339-48. doi: 10.1016/j.spinee.2012.01.015. Epub 2012 Mar 3. Review. PubMed PMID: 22381574

Losina E, Wright J, Katz JN. Clinical trials in orthopaedics research. Part III. Overcoming operational challenges in the design and conduct of randomized clinical trials in orthopaedic surgery. J Bone Joint Surg Am. 2012 Mar 21;94(6):e35. doi: 10.2106/JBJS.K.00009. PubMed PMID: 22438008

Sauaia A, Moore EE, Crebs JL, Maier RV, Hoyt DB, Shackford SR. Evidence level
of individual studies: a proposed framework for surgical research. J Trauma Acute
Care Surg. 2012 Jun;72(6):1484-90. doi: 10.1097/TA.0b013e318256dc4d. Review.
PubMed PMID: 22695411

Wenner DM, Brody BA, Jarman AF, Kolman JM, Wray NP, Ashton CM. Do surgical
trials meet the scientific standards for clinical trials? J Am Coll Surg. 2012 Nov;215(5):722-30. doi: 10.1016/j.jamcollsurg.2012.06.018. Epub 2012 Jul 21.  Review. PubMed PMID: 22819638

White SM, Griffiths R, Moppett I. Type of anaesthesia for hip fracture surgery – the problems of trial design. Anaesthesia. 2012 Jun;67(6):574-8. doi: 10.1111/j.1365-2044.2012.07120.x. Erratum in: Anaesthesia. 2012 Aug;67(8):936. Moppett, K [corrected to Moppett, I K]. PubMed PMID: 22563954


Audigé L, Ayeni OR, Bhandari M, Boyle BW, Briggs KK, Chan K, Chaney-Barclay K, Do HT, Ferretti M, Fu FH, Goldhahn J, Goldhahn S, Hidaka C, Hoang-Kim A, Karlsson J, Krych AJ, LaPrade RF, Levy BA, Lubowitz JH, Lyman S, Ma Y, Marx RG, Mohtadi N, Marcheggiani Muccioli GM, Nakamura N, Nguyen J, Poehling GG, Roberts LE, Rosenberg N, Shea KP, Sohani ZN, Soudry M, Voineskos S, Zaffagnini S; International Society of Arthroscopy, Knee Surgery and Orthopaedic Sports Medicine. A practical guide to research: design, execution, and publication. Arthroscopy. 2011 Apr;27(4 Suppl):S1-112. doi: 10.1016/j.arthro.2011.02.001. Erratum in: Arthroscopy. 2011 Jun;27(6):878. PubMed
PMID: 21749939

Cook JA, Ramsay CR, Vale LD; DELTA Project Group. Guidance on minimally  important clinical difference and trial size is needed. BMJ. 2011 Jul 25;343:d4375. doi: 10.1136/bmj.d4375. PubMed PMID: 21788268

Chang DC, Rhee DS, Papandria D, Aspelund G, Cowles RA, Huang EY, Chen C,
Middlesworth W, Arca MJ, Abdullah F. Outcomes research in pediatric surgery. Part
2: how to structure a research question. J Pediatr Surg. 2011 Jan;46(1):226-31.
doi: 10.101/j.jpedsurg.2010.09.095. PubMed PMID: 21238673

Gillinov AM, Argenziano M, Blackstone EH, Iribarne A, DeRose JJ Jr, Ailawadi G, Russo MJ, Ascheim DD, Parides MK, Rodriguez E, Bouchard D, Taddei-Peters WC,
Geller NL, Acker MA, Gelijns AC. Designing comparative effectiveness trials of
surgical ablation for atrial fibrillation: experience of the Cardiothoracic Surgical Trials Network. J Thorac Cardiovasc Surg. 2011 Aug;142(2):257-64.e2. doi: 10.1016/j.jtcvs.2011.04.010. Epub 2011 May 25. PubMed PMID: 216165076

Gurusamy KS, Gluud C, Nikolova D, Davidson BR. Design of surgical randomized
controlled trials involving multiple interventions. J Surg Res. 2011 Jan;165(1):118-27. doi: 10.1016/j.jss.2009.09.054. Epub 2009 Oct 30. Review. PubMed PMID: 20097373

Iribarne A, Gelijns AC, Acker MA, Ascheim DD. Innovative clinical trial design in cardiac surgery. Semin Thorac Cardiovasc Surg. 2011 Winter;23(4):271-3. doi:  10.1053/j.semtcvs.2011.12.003. PubMed PMID: 22443645

Katz JN, Wright JG, Losina E. Clinical trials in orthopaedics research. Part II. Prioritization for randomized controlled clinical trials. J Bone Joint Surg Am. 2011 Apr 6;93(7):e30. doi: 10.2106/JBJS.J.01039. Review. PubMed PMID: 21471410

Murad MH, Montori VM, Sidawy AN, Ascher E, Meissner MH, Chaikof EL, Gloviczki
P. Guideline methodology of the Society for Vascular Surgery including the experience with the GRADE framework. J Vasc Surg. 2011 May;53(5):1375-80. doi:  10.1016/j.jvs.2011.01.036. Epub 2011 May 5. PubMed PMID: 21575755

Pitak-Arnnop P, Hemprich A, Pausch NC. Evidence-based oral and maxillofacial  surgery: some pitfalls and limitations. J Oral Maxillofac Surg. 2011 Jan;69(1):252-7. doi: 10.1016/j.joms.2010.07.082. Epub 2010 Nov 2. PubMed PMID: 21050635

Redline S, Amin R, Beebe D, Chervin RD, Garetz SL, Giordani B, Marcus CL, Moore RH, Rosen CL, Arens R, Gozal D, Katz ES, Mitchell RB, Muzumdar H, Taylor HG, Thomas N, Ellenberg S. The Childhood Adenotonsillectomy Trial (CHAT): rationale, design, and challenges of a randomized controlled trial evaluating a standard surgical procedure in a pediatric population. Sleep. 2011 Nov 1;34(11):1509-17. doi: 10.5665/sleep.1388. PubMed PMID: 22043122

Study to Prospectively Evaluate Reamed Intramedullary Nails in Tibial Fractures (SPRINT) Investigators, Sun X, Heels-Ansdell D, Walter SD, Guyatt G, Sprague S, Bhandari M, Sanders D, Schemitsch E, Tornetta P 3rd, Swiontkowski M. Is a subgroup claim believable? A user’s guide to subgroup analyses in the surgical literature. J Bone Joint Surg Am. 2011 Feb 2;93(3):e8. doi: 10.2106/JBJS.I.01555. PubMed PMID:21266635

Thoma A, Cornacchi SD, Farrokhyar F, Bhandari M, Goldsmith CH; Evidence-Based
Surgery Working Group. How to assess a survey in surgery. Can J Surg. 2011 Dec;54(6):394-402. doi: 10.1503/cjs.025910. PubMed PMID: 21939608


Pocard M, Soria JC, Aldaz-Carroll L, Bellet D. Phase 0 clinical trials in oncology: an exploratory methodology for constructing a study with patients  undergoing surgery for metastatic disease. J Clin Oncol. 2010 Oct 20;28(30):4551-3. doi: 10.1200/JCO.2010.29.2870. Epub 2010 Sep 13. PubMed PMID: 20837954


Bains S. Getting educated: e-learning resources in the design and execution of surgical trials. J Long Term Eff Med Implants. 2009;19(3):235-9. PubMed PMID: 20939784

Brubaker L, Moalli P, Richter HE, Albo M, Sirls L, Chai T, Kraus SR, Norton P,  Chang D, Tennstedt SL. Challenges in designing a pragmatic clinical trial: the mixed incontinence — medical or surgical approach (MIMOSA) trial experience. Clin Trials. 2009 Aug;6(4):355-64. doi: 10.1177/1740774509339239. Epub 2009 Jul  22. PubMed PMID: 19625327

Cook JA. The challenges faced in the design, conduct and analysis of surgical randomised controlled trials. Trials. 2009 Feb 6;10:9. doi: 10.1186/1745-6215-10-9. PubMed PMID: 19200379

McCulloch P. Developing appropriate methodology for the study of surgical techniques. J R Soc Med. 2009 Feb;102(2):51-5. doi: 10.1258/jrsm.2008.080308. PubMed PMID: 19208868

Singh JA, Murphy S, Bhandari M. Assessment of the methodologic quality of medical and surgical clinical trials in patients with arthroplasty. J Rheumatol. 2009 Dec;36(12):2642-54. doi: 10.3899/jrheum.090333. Epub 2009 Nov 2. Review. PubMed PMID: 19884268


Bessa-Nogueira RV, Vasconcelos BC, Niederman R. The methodological quality of
systematic reviews comparing temporomandibular joint disorder surgical and non-surgical treatment. BMC Oral Health. 2008 Sep 26;8:27. doi: 10.1186/1472-6831-8-27. PubMed PMID: 18822118

Cook JA, Ramsay CR, Norrie J. Recruitment to publicly funded trials–are surgical trials really different? Contemp Clin Trials. 2008 Sep;29(5):631-4. doi:  10.1016/j.cct.2008.02.005. Epub 2008 Mar 7. PubMed PMID: 18397843

Neugebauer EA, Morino M, Habermalz B. Surgical research or comic opera? Let’s give answers! Surg Endosc. 2008 Jun;22(6):1411-2. doi: 10.1007/s00464-008-9906-z.
Epub 2008 Apr 9. PubMed PMID: 18398640

Simunovic N, Devereaux PJ, Bhandari M. Design considerations for randomised trials in orthopaedic fracture surgery. Injury. 2008 Jun;39(6):696-704. doi:  10.1016/j.injury.2008.02.012. Epub 2008 May 27. Review. PubMed PMID: 18502423

Sprague S, McKay P, Thoma A. Study design and hierarchy of evidence for surgical decision making. Clin Plast Surg. 2008 Apr;35(2):195-205. doi:  10.1016/j.cps.2007.10.007. Review. PubMed PMID: 18298992

Thoma A, McKnight L, McKay P, Haines T. Forming the research question. Clin Plast Surg. 2008 Apr;35(2):189-93. doi: 10.1016/j.cps.2007.10.009. Review. PubMed PMID: 18298991


Boutron I, Ravaud P, Nizard R. The design and assessment of prospective randomised, controlled trials in orthopaedic surgery. J Bone Joint Surg Br. 2007 Jul;89(7):858-63. PubMed PMID: 17673575



Leave a Reply

Your email address will not be published. Required fields are marked *