Transparency and Access to Data

These resources deal with the essential topic of open and comprehensive disclosure in research.  Without knowing exactly what was planned, carried out and observed in a study it is impossible to determine its importance.  Failures to disclose this information has led to substantial harm to patients and wasted research effort. Continue reading

Outcome Measures and External Validity

These resources deal with how we identify, define and measure the outcomes of trials in surgery and other complex interventions.  A key aspect is the external validity of these measures, that is, whether they accurately and consistently represent the health outcomes they are supposed to measure. Continue reading

The need for core outcome sets in surgery

By Natalie Blencowe and Jane Blazeby

Compared with pharmaceutical trials, the quality of surgical randomised trials is poor and the evidence base for many surgical procedures remains weak. Reasons for this are multi-factorial including problems with recruitment, blinding, and the fact that surgical procedures are constantly evolving.

Another major difficulty relates to outcome assessment because there are currently no recognised definitions or standards for measuring surgical outcomes, including complications.

Lack of consistent outcomes

A recent systematic review of oesophageal cancer surgery has highlighted the extent of this problem, as not a single outcome was reported across all 122 included papers.

Anastomotic leak was reported in 80 studies but only defined in 28, using 22 different definitions  [1]. Similar problems have been reported in reviews of colorectal cancer, obesity and reconstructive breast surgery [2].

If studies do not all report the same outcomes, or provide definitions, it becomes impossible to accurately synthesise data so that outcomes can be compared between hospitals.

In addition, most studies measure and report surgeon-selected outcomes rather than patient-reported outcomes. This means that patients’ perspectives are often not considered and also that the intervention in question cannot be fully evaluated.

Core outcome sets

One solution is to provide a core outcome set for each surgical condition or procedure.

Core outcome sets contain a minimum agreed set of outcomes to be reported in all studies of a particular condition or procedure, and agreed definitions should also be provided as part of this.

If definitions and outcomes are standardised, meaningful cross-study comparisons can be made which minimises outcome reporting bias.

Developing core outcome sets via COMET

One way of developing core outcome sets is to use Delphi methodology to reach consensus by surveying key stakeholders, including patients.

The Core Outcome Measurement in Effectiveness Trials initiative (COMET) facilitates development of such measures in all areas of healthcare, including surgery. We are developing a core outcome sets for oesophageal and colorectal cancer surgery, for obesity surgery and for breast reconstructive surgery 1, 2.

To achieve this, we are working with the respective sub speciality organisations and with patient support groups. Whilst it is anticipated that clinical and patient-reported outcomes will be included, the final items (and their definitions) are yet to be decided.

If you would like to find out more, please visit our website, or post a question below.

  1. Blencowe NS, Strong S, McNair AG, Brookes ST, Crosby T, Griffin SM, Blazeby JM. Reporting of short-term clinical outcomes after esophagectomy: a systematic review. Ann Surg. 2012; 255(4):658-66.
  2. Potter S, Brigic A, Whiting PF, Cawthorn SJ, Avery KN, Donovan JL, Blazeby JM. Reporting clinical outcomes of breast reconstruction: a systematic review. J Natl Cancer Inst. 2011; 103(1):31-46.


Open Science and Data Sharing in Clinical Research

Basing Informed Decisions on the Totality of the Evidence

In a March 2012 editorial for Circulation: Cardiovascular Quality and Outcomes, Harlan Krumholz, Director of the YODA project, identified the steps required to achieve the fullest use of clinical research data to benefit patient care:

  1. Post, in the public domain, the study protocol for each published trial. The protocol should be comprehensive and include policies and procedures relevant to actions taken in the trial.
  2. Develop mechanisms for those who own trial data to share their raw data and individual patient data.
  3. Encourage industry to commit to place all its clinical research data relevant to approved products in the public domain. This action would acknowledge that the privilege of selling products is accompanied by a responsibility to share all the clinical research data relevant to the products’ benefits and harms.
  4. Develop a culture within academics that values data sharing and open science. After a period in which the original investigators can complete their funded studies, the data should be de-identified and made available for investigators globally.
  5. Identify, within all systematic reviews, trials that are not published, using sources such as and regulatory postings to determine what is missing.
  6. Share data.

Krumholz, H.M. 2012. Open Science and Data Sharing in Clinical Research: Basing Informed Decisions on the Totality of the Evidence. Circ Cardiovasc Qual Outcomes. 5:141-142.

Other recent relevant articles include:

Ross, J.S., Lehman, R. and C.P. Gross. 2012. The Importance of Clinical Trial Data Sharing: Toward More Open Science. Circ Cardiovasc Qual Outcomes. 5:238-240.

Spertus, J.A. 2012. The Double-Edged Sword of Open Access to Research Data. Circ Cardiovasc Qual Outcomes. 5:143-144.

Gotzsche, P.C. 2012. Strengthening and Opening Up Health Research by Sharing Our Raw Data. Circ Cardiovasc Qual Outcomes. 5:236-237.

Yale University Open Data Access (YODA) Project

We are very grateful to Harlan Krumholz and Richard Lehmann for providing this short overview of the YODA project.  YODA provides a clear model of how we can address the fifth component of the IDEAL framework:  long-term follow-up.

The aim of the YODA project is to promote and facilitate the sharing of clinical research data so that research can be reproduced and extended in the service of advancing the public’s interest.

The project seeks to develop, test and implement methods to disseminate research data as widely, comprehensively, responsibly and productively as possible.

Why we need YODA

There is clear evidence that many clinical research studies are never published and that much of the clinical research cannot be reproduced – or risks being duplicated unnecessarily – because data are not shared.

This issue resides across private and publicly funded efforts – and involves academics, industry scientists and leaders, funders, policymakers, and journal editors.

How YODA can help

Our aspiration is to find common ground between the interests of academia, industry, government and the public in promoting more open science and data exchange. Through this course we believe that all parties can benefit from greater confidence by the public in the scientific process and the principal actors.

We are specifically focusing on the sharing of data that may result in better information about the risk and benefits of products and strategies that are in use, rather than on pre-clinical or pre-approval research, which might have issues of intellectual property. Our ultimate goal is to ensure that support better informed decisions by ensuring that data are not hidden from view. We are seeking partners with data and making efforts to forge together mechanisms by which the data can be shared. Our approach is to accomplish this sharing through mutual collaboration rather than external regulation.

Progress to date

In its initial project, YODA has developed one such dissemination model which provides a means for rigorous and objective evaluation of clinical trial data to ensure that patients and practitioners possess all necessary information about a drug or device when making treatment decisions.

This model is designed to provide industry with confidence that the analyses will be conducted in a scientifically rigorous, objective and fair manner. Several features of the model are specifically focused on promoting transparency and protecting against industry influence:

  • The company engaging in the model must provide all relevant product data
  • Two independent research groups, selected after a competitive application process, systematically review and analyze all relevant product data
  • An independent Steering Committee, including leaders in the field of clinical research and biomedical ethics, advise the YODA project team
  • A Clinical Advisory Committee, including leaders in the clinical practice that uses the product under evaluation, advise the project
  • Project leadership are committed to transparency, publication, and making the data publicly available

Applying the model to recombinant bone growth factors

In August 2011, Medtronic Inc. reached agreement with Yale University to commit to this model for analyzing all data relating to its products containing recombinant bone morphogenetic protein-2 (rhBMP-2). Yale University has obtained full individual patient data from 17 trials conducted by Medtronic together with all necessary supporting materials (meta-data).

Yale selected two highly regarded independent research groups, University of York and Oregon Health and Science University, to conduct analyses of these data independent of each other and without any direct involvement or influence from Medtronic or Yale University.

Dissemination plans

The research groups are on target to complete their analyses in August 2012. Their results will be disseminated simultaneously soon after completion, and the full Medtronic data sets will then be made publicly available for further evaluation by external investigators. The completion of this project will yield important methodological lessons for further development and deployment of this model of retrospective data disclosure.
The YODA project looks forward to learning from the results of this unique collaboration, as part of its continuing effort to explore the best means for sharing of data from all interventional human trials. More importantly, we hope that this project is just the first of many that will follow.

Find out more

For further details, visit the YODA Web site:

See also: Krumholz, H.M. and J.S. Ross. 2011. “A Model for Dissemination and Independent Analysis of Industry Data.” JAMA. 306(14):1593-1594.

Implementing IDEAL at the IJS

Cover fo the IJSThere is good evidence that poorly reported research is more likely to be biased. Just as the CONSORT Statement improved the quality of research reporting, IDEAL has the potential to improve the quality of surgical case reports and other publication types that report research in surgery and complex interventions.

But is there still a problem? Riaz Agha at the International Journal of Surgery (IJS) thinks so:

“Recent work I have done on Plastic Surgery RCTs has shown how people don’t report conflicts of interest, funding sources, ethical approval, etc in addition to how they randomised their patients.

Journal Editors could have easily corrected all that through having strict policies in place. We publish all this information each and every time at the IJS as we have a good Manuscript Administrator who ensures compliance before things go to Production. It’s incorporated into our submission process so we capture it at source.”

If you want to find out more about how the IJS is implementing IDEAL Framework in journal publication, please visit their website.

Of course, IDEAL is a whole framework for how to bring new interventions through and not just about reporting the right information. Whilst journal editors have a very influential role in improving the quality of research reporting, change needs to happen in parallel with other developments.

“By the time the manuscript gets to a journal Editor – the work has already been done and it’s too late to influence the research itself. It would be useful for instance if new technologies/interventions were mapped onto the IDEAL framework in Research Reports from the RCS – giving it credence and making it the roadmap we use and expect new interventions to proceed along.

I think we can raise awareness with editorials, debate, presentations at major conferences, etc. but it will take time and a cultural change.“

Research inside or outside your comfort zone: Time for a paradigm shift.

By Michael Baum

Almost by definition scientific research is conducted within a conceptual framework (paradigm) where its limits are invisible or wilfully ignored by the vast majority of the  researchers in that field of endeavour.

If you are in a field, then like a cow you keep your head down eating the grass seldom lifting your head to chew the cud and rarely noticing the hedgerows.

What I am trying to define are the two fundamental processes of scientific philosophy, normal and revolutionary science. [1,2]

Normal science

Normal science starts with a hypothesis (the generation of that hypothesis is another story) and sets up experiments to “hazard its refutation”. Well-conducted normal science proceeds with incremental, modest steps as the hypothesis is slowly refined building upon the observation of each prior set of experiments.

In the long history of science crises occur when the explanatory power of the paradigm begins to break down following the accumulation of new observations. This should in the fullness of time, allow for a conceptual revolution (paradigm shift) where a new framework will be constructed upon which to build a new direction of research.

This new paradigm will have sufficient explanatory power to account for the successes of the past whilst comfortably accommodating all the accumulated outlying observations and should launch a new epoch of revolutionary science.

Sadly at this point human nature gets in the way, as those who adhere to the ancien regime have already translated the prevailing paradigm into a quasi-religious dogma, invested too much personal prestige into the status quo, or are so short sighted they cannot see the wood for the trees.

As these adherents to the established belief system are by far the majority, then the revolutionaries face a tough time as they subjected to ridicule, character assassination or even threatened with murder at the hands of the inquisition, as in the well documented case of Galileo and the moons of Jupiter.

Scientific revolutions in breast cancer care

I would like to illustrate these abstract constructs with real examples taken from a lifetime’s experience with research into the natural history and treatment of breast cancer. I claim to have survived one scientific revolution and am currently fighting to defend my own reputation in the face of what I see as a second scientific revolution.

The first revolution occurred in the mid 1970s although its seeds were planted and first showed their green shoots in the late 1960s.  That means that most readers of this commentary, those in the prime of their scientific careers at the moment, say aged 30-50, would either not have been born or just have been leaving high school at the time. In other words not experiencing the “blood shed” at first hand.

When I first entered the arena in about 1965, the prevailing conceptual model to explain the nature of breast cancer was an anatomical or mechanistic view of the centrifugal spread of cancer cells along the lymphatic system, the therapeutic consequence of which was the Halsted radical mastectomy. [3]

Anyone reading this piece will have learnt of the history of its fall from grace spearheaded by Dr. Bernard Fisher of Pittsburgh PA. [4] It is now a given that breast cancer has to be considered as a systemic disease at the time of clinical presentation, the therapeutic consequences of which include breast conserving surgery and adjuvant systemic therapy.

Furthermore, detecting breast cancer before it is clinically apparent might indeed find  disease that has not yet had the opportunity to spread, hence the popularization of screening.

The successes of the therapeutic regimens are beyond dispute, [5,6] and modern readers must think how quaint it was to have believed the nonsense of the mechanistic model of disease.

Unlike you I was there. The proponents of the Halstedian belief system were outraged by the challenge from us cheeky young upstarts and our leader, Bernie Fisher, was often accused of being responsible for the deaths of thousands of women.  These people were not converted: they retired and died unshaken in their loyalty to the faith.

Without wishing to deny the amazing progress in the treatment of breast cancer over the last 40 years, I have to alert you to the fact that outlying observations, which cannot be explained according to conventional wisdom, are rapidly accumulating and I believe it’s time for a paradigm shift!

The national breast screening programmes around the world have provided us with a natural experiment of the greatest historical importance, not because of their success in reducing breast cancer mortality, but because of the observations concerning the over-diagnosis of the disease. [7,8]

Cancer was defined by its microscopic appearance about two hundred years ago. The 19th century saw the birth of scientific oncology with the discovery and use of the modern microscope. Rudolf Virchow, often called the founder of cellular pathology, provided the scientific basis for the modern pathologic study of cancer. [9]

However the material he was studying came from the autopsy of patients dying from cancer. In the mid 19thC pathological correlations were performed either on cadavers or on living subjects presenting with locally advanced or metastatic disease that almost always were pre-determined to die in the absence of effective therapy.  Since then without pause for thought, the microscopic identification of cancer according to these classic criteria has been associated with the assumed prognosis of a fatal disease if left untreated. These pathological entities might have microscopic similarity to true cancers but these appearances alone are insufficient to predict a life threatening disease. [10]

Conventional mathematical models of cancer growth are linear or logarithmic, in other words completely predictable at the outset; predicting transition from in-situ phases to early invasive and from early invasive to late invasive over time.  These mathematical formulae may be appropriate for designing theme park rides but cannot begin to explain the exquisite organization of cell proliferation and the complex inter-relationships of cells of different progeny.

Most natural biological mechanisms are non-linear or better described according to chaos theory. [11] Prolonged latency followed by catastrophe should not be all that surprising. We accept the case for prostate cancer, as we know that most elderly men will die with prostate cancer in situ and not of prostate cancer. In fact the UK national PSA screening trial (ProtecT) is predicated on that fact with two a priori outcome measures defined, deaths from prostate cancer versus the number of cancers over-detected and treated unnecessarily. [12]

Next it is worth noting, that contrary to all common sense predictions, the increased rate of detection of duct carcinoma in situ (DCIS), has lead to an increase in the mastectomy rate for the screened population. [13] Up to 45% of screen detected cases of DCIS end up having mastectomy because of the multi-centricity of the disease. [14] Yet the paradox is that clinically detected multi-centric invasive breast cancer is relatively uncommon. [15] Surely that is proof enough that at least half of these foci of DCIS regress if left alone; which half of course, remains the problem.

In conclusion therefore we can state with a great deal of conviction, that a large proportion (in the order of 50%), of screen detected (pre-clinical) foci of breast cancer, are not programmed to progress if left unperturbed. This observation is of seismic importance and could set the agenda for breast cancer research for the next decade.

For a start we should consider a trial of active surveillance versus conventional therapy for screen-detected cases of DCIS. Using this platform we might then learn what the clinical or biological characteristics of the disease are that allow it to leave its dormant phase and enter the transition to early invasion.

If we choose to ignore these observations, either because they do not support our prejudices, or for some sleazy political agenda, then we will have missed an opportunity of a lifetime and that would be unforgivable.

I’ve already been publically accused on Newsnight with Jeremy Paxman of being responsible for the deaths of thousands of women, so I must be going in the right direction.

There’s non so blind as them who will not see.