P. McCulloch: The IDEAL Collaboration

The work of the EQUATOR Network has been very important in gradually transforming the quality of reporting within clinical research. Reporting Guidelines such as CONSORT, PRISMA and SQUIRE have raised the quality of reporting of different types of research by providing a rational framework and examples which researchers can follow. Further helpful guidelines are in development such as the TIDIER checklist. In addition initiatives such as COMET (Core Outcome Measures in Effectiveness Trials) have contributed by emphasising the importance of the development and application of agreed standardised sets of outcomes in clinical research. Although much remains to be done, the scene has undoubtedly improved enormously.

The case of surgical research:

The problems of clinical research, however, are not all related to reporting standards. Many specialities where interventions are complex, customised and operator dependant also have to deal with methodological problems which often frustrate the desire to do good valid clinical studies. The IDEAL Collaboration was born of a desire to improve both reporting standards and methodology in surgery, as we see these two as inextricably linked.

The IDEAL framework (http://www.ideal-collaboration.net/framework/) outlines a realistic view of how surgical interventions, medical devices and other complex interventions actually develop in practice. Evaluation needs to recognise these stages of development, and both reporting standards and methodology need to be tailored so they can answer the appropriate questions at different stages. How this could be done is outlined in the IDEAL recommendations (http://www.ideal-collaboration.net/recommendations/). Here the common goals of EQUATOR and IDEAL become very obvious, because the recommendations contain both methodological and reporting requirements. In the very early stages of development, we emphasise the need for a searchable registry of “first in man” surgical procedures. This is essential to ensure that learning from new experimental surgery should begin as soon as possible, and that where procedures turn out to be ineffective patients will not be harmed by others attempting the same procedure. In order to avoid creating a target for litigation, it’s essential that this type of registry has protections for those that wish to make a confidential report, and we’re currently discussing with NICE interventional procedures panels and others how this could be brought about.

Once a new technique has been tried out, it needs development, and there is inevitably a stage of rapid modification in the early part of this process. The IDEAL recommendations emphasise the need for an explicit, clear and comprehensive account of the development process, indicating exactly when and why the technique was changed, when the indications were changed, and reporting each patient considered for the procedure sequentially without exceptions. We are currently working with several surgical groups to produce reports in this format, rather than in the much less informative and potentially biased traditional surgical case series format.

IDEAL advocates the use of preliminary collaborative prospective cohort studies by surgeons who are attempting to develop randomised trials together. This is principally to reduce uncertainty about the trial population and question, as achieving agreement between surgeons on these points has been a difficultly which has caused failure in many attempts to set up randomised trials in surgery in the past. However an additional benefit of this kind of study is it allows the study of surgeons’ learning curves as they begin to do a new procedure, and we advocate reporting of these curves using statistical process control techniques such as CUSUM. The IDEAL collaboration has not yet examined in detail the reporting requirements for surgical randomised trials, but this might be something on which IDEAL could work together with the CONSORT NPT group to include surgery-specific guidance. The IDEAL recommendations emphasise the need to clearly define the intervention, and the usefulness of including surgical quality control measures in any trial evaluation. Mention or discussion of the learning curve issue for participants in the trial is also an important reporting point from the IDEAL point of view.

Once techniques have become accepted and are subjected to long term study to look for rare and late effects, we emphasise the need for registries which are as comprehensive as possible in terms of the population included, and as complete as possible in terms of data entry. It is important that ambitions of registries in terms of these objectives are consistent with their resources, and that the data fields are selected by the clinical community rather than manufacturers or political bodies.

If anyone is interested in exploring the advocacy work of IDEAL in trying to improve reporting and methodology in surgery please visit our main website at http://www.ideal-collaboration.net or contact our Project Manager allison.hirst@nds.ox.ac.uk. We’d be very glad to share ideas and discuss possible collaboration.

For more details of the IDEAL framework, recommendations and proposals at all stages see our publications below.

Peter McCulloch, Chair, on behalf of the IDEAL Collaboration.

IDEAL key publications
Hirst A, Agha RA, Rosin D, McCulloch P. How can we improve surgical research and innovation?: The IDEAL framework for action. Int J Surg. 2013 Oct 21. doi:10.1016/j.ijsu.2013.09.016

McCulloch P, Cook JA, Altman DG, Heneghan C, Diener MK; IDEAL Group. IDEAL framework for surgical innovation 1: the idea and development stages. BMJ. 2013 Jun 18;346:f3012.

Ergina PL, Barkun JS, McCulloch P, Cook JA, Altman DG; IDEAL Group. IDEAL framework for surgical innovation 2: observational studies in the exploration and assessment stages. BMJ. 2013 Jun 18;346:f3011.

Cook JA, McCulloch P, Blazeby JM, Beard DJ, Marinac-Dabic D, Sedrakyan A; IDEAL Group. IDEAL framework for surgical innovation 3: randomised controlled trials in the assessment stage and evaluations in the long term study stage. BMJ. 2013 Jun 18;346:f2820.

McCulloch P. The IDEAL recommendations and urological innovation. World J Urol 2011 Jun;29(3):331-6. Epub 2011 Feb 17.

Blazeby JM, Blencowe NS, Titcomb DR, Metcalfe C, Hollowood AD, Barham CP. Demonstration of the IDEAL recommendations for evaluating and reporting surgical innovation in minimally invasive oesophagectomy. Br J Surg. 2011 Apr;98(4):544-51. doi: 10.1002/bjs.7387. Epub 2011 Jan 18.

McCulloch P, Altman DG, Campbell WB, Flum DR, Glasziou P, Marshall JC, Nicholl J, for the Balliol Collaboration. No surgical innovation without evaluation: the IDEAL recommendations. Lancet 2009; Sep 26; 374 (9695): 1105-12.

Ergina PL, Cook JA, Blazeby JM, Boutron I, Clavien PA, Reeves BC, Seiler CM; Balliol Collaboration, Altman DG, Aronson JK, Barkun JS, Campbell WB, Cook JA, Feldman LS, Flum DR, Glasziou P, Maddern GJ, Marshall JC, McCulloch P, Nicholl J, Strasberg SM, Meakins JL, Ashby D, Black N, Bunker J, Burton M, Campbell M, Chalkidou K, Chalmers I, de Leval M, Deeks J, Grant A, Gray M, Greenhalgh R, Jenicek M, Kehoe S, Lilford R, Littlejohns P, Loke Y, Madhock R, McPherson K, Rothwell P, Summerskill B, Taggart D, Tekkis P, Thompson M, Treasure T, Trohler U, Vandenbroucke J. Challenges in evaluating surgical innovation. Lancet. 2009 Sep 26;374(9695):1097-104.

Barkun JS, Aronson JK, Feldman LS, Maddern GJ, Strasberg SM; Balliol Collaboration, Altman DG, Barkun JS, Blazeby JM, Boutron IC, Campbell WB, Clavien PA, Cook JA, Ergina PL, Flum DR, Glasziou P, Marshall JC, McCulloch P, Nicholl J, Reeves BC, Seiler CM, Meakins JL, Ashby D, Black N, Bunker J, Burton M, Campbell M, Chalkidou K, Chalmers I, de Leval M, Deeks J, Grant A, Gray M, Greenhalgh R, Jenicek M, Kehoe S, Lilford R, Littlejohns P, Loke Y, Madhock R, McPherson K, Rothwell P, Summerskill B, Taggart D, Tekkis P, Thompson M, Treasure T, Trohler U, Vandenbroucke J. Evaluation and stages of surgical innovations. Lancet. 2009 Sep 26;374(9695):1089-96.