Gary Collins, Associate Professor and Deputy Director of the Centre for Statistics in Medicine, University of Oxford introduces TRIPOD.

Decisions based on clinical predictions are routinely made throughout medicine and at all stages in pathways of health care.  For example, in the diagnostic setting, predictions are made as to whether a particular disease is present informing the referral for further testing, initiate treatment or reassure patients that a serious cause for their symptoms is unlikely.  In the prognostic setting, predictions can be used for planning lifestyle or therapeutic decisions based on the risk of developing a particular outcome over a time period.  Yet, making a diagnostic or prognostic prediction is challenging and rarely based on a single risk factor, test result or symptom.

The multifactorial nature of making a clinical prediction makes it difficult for doctors to simultaneously and subjectively weigh multiple risk factors to produce a reliable and accurate estimate of risk. Furthermore, seeing relatively few cases and cognitive biases, it is unsurprising that numerous studies have shown that doctors are generally poor prognosticators.

However, increasingly doctors, often based on recommendations in national clinical guidelines, are using multivariable prediction models to support and guide the clinical decision-making process.  A clinical prediction model is a mathematical equation that relates multiple predictors for an individual to the probability (or risk) that a particular disease or condition is present or will occur in the future. Well-known prediction models include the Framingham Risk score, Apgar Score, Ottawa Ankle Rules, EuroSCORE, Nottingham Prognostic Index and the Simplified Acute Physiology Score (SAPS).

Introducing The TRIPOD statement

The last 10-15 years have seen an explosion in the number of published articles describing the development (and occasionally the validation) of clinical prediction models. Particular clinical areas have seen considerable numbers of models being developed for the same outcome (e.g. diabetes, TBI, prostate cancer). Developing a prediction model can be very easy; using existing data (collected for a different purpose) and a statistical package. Clearly, this is an oversimplification, but cynically, many prediction models are developed, with no compelling clinical need and thus with little intention of ever being used, and are merely an easy publication to add to one’s curriculum vitae.  In areas where there have been many models being developed, deciding which one to use is difficult. Which is made particularly more difficult, as many are developed under the guise of being a new discovery with existing models often ignored and rarely compared against.

To evaluate the methodological conduct and reporting we conducted a number of systematic reviews describing the development or validation of multivariable prediction models across different medical areas and found that reporting to be particularly poor.  What was surprising when we analysed the results from these and other systematic reviews was the clear lack of crucial information being presented. Without full and transparent reporting, evaluating, even implementing and synthesizing the results (see CHARMS Checklist) from such studies is problematic.

At the most critical level, authors were developing models yet failing to provide information on the actual model so that other researchers were unable to use the model to test the model or use on their patients – which is clearly ludicrous, to go to the extent to develop a model yet fail to tell readers what the model is.  Published articles with incomplete reporting are unusable – as noted in the recent Lancet series on reducing waste in research.

It was clear to us that authors, reviewers, editors and readers needed clear guidance on what issues should be included when describing the development and validation of a prediction model, which led us to develop the TRIPOD Statement.

The TRIPOD Statement is an annotated checklist (PDF) of items that arose from a systematic review of the literature, which was further reduced and refined following discussions during a 3-day consensus meeting in 2011 with international experts in prediction modelling (statisticians, epidemiologists, clinicians and journal editors). The resulting checklist comprises 22 items that were regarded essential for good reporting of studies developing or validating multivariable prediction models. Authors of published reports of studies describing the development, validation or updating of a prediction model should ensure that all items in the checklist are mentioned somewhere in the article.

The recommendations within TRIPOD are guidelines only for reporting research and do not prescribe how to develop or validate a prediction model. Furthermore, the checklist is not a quality assessment tool to gauge the quality of a multivariable prediction model, for which the upcoming PROBAST risk of bias tool will be available.

Many prediction models are (unfortunately) developed without involving either a statistician or epidemiologist, and therefore providing guidance that can be readily understood by study investigators of varying levels of technical (e.g. methodological) experience was of paramount importance.  Furthermore, to motivate authors (also peer reviewers and editors) to use the checklist, we also aimed to keep the checklist as brief as possible, whilst ensuring all the key details are clearly reported.

Whilst the TRIPOD Statement is foremost guidance for reporting, we produced an accompanying and extensive 22,000-word Explanation & Elaboration article discussing not only rationale and example of good reporting but also methodological aspects for investigators to consider when developing, validating or updating a prediction model.  Yet, one of the challenges we faced in providing such information on suitable methodology was to provide a balanced account of competing approaches, particularly as there is no clear consensus on many aspects in developing or validating a prediction model, and where the methodology in this field is continually evolving. Conscious efforts were made to be neutral by describing both advantages and disadvantages of various methods, cautioning against methodologically weak approaches without dictating how the study investigators should conduct their study.

To increase the visibility of the TRIPOD Statement we are co-publishing the article simultaneously in 11 leading general medical and speciality journals including the Annals of Internal Medicine, BJOG, BMC Medicine, British Journal of Cancer, British Journal of Surgery, British Medical Journal, Circulation, Diabetic Medicine, European Journal of Clinical Investigation, European Urology and the Journal of Clinical Epidemiology. We welcome other journals in endorsing TRIPOD, by including prediction model studies as a distinct study type, include TRIPOD in their instructions for authors and require authors to complete and submit a TRIPOD Checklist in their submission.

We have also developed a website (www.tripod-statement.org) where additional information, references, and PDF and Word versions of the checklist are available for download.  Journals and organisations endorsing TRIPOD will be listed on the TRIPOD website.  For announcements on TRIPOD related information, follow us on Twitter @TRIPODStatement.

The Centre for Statistics in Medicine (University of Oxford) is also home of the EQUATOR Network and will list TRIPOD among their list of key reporting guidelines, and it will be announced via the EQUATOR Newsletter, Twitter (@EQUATORNetwork) and LinkedIn.

We believe that if authors adhere to the TRIPOD Statement, that readers and potential users (of the model) will have a transparent and full account of all aspects of the prediction model study enabling them to critique and fully judge the potential merits of the model.

Gary Collins on behalf of the TRIPOD steering group (Gary Collins and Doug Altman, University of Oxford, UK; Karel Moons and Hans Reitsma, UMC Utrecht, The Netherlands)

Image credit: Oliver Tacke, Flickr.

PLOS Medicine Editorial Director, Virginia Barbour, reflects on the publication of the CONSORT and PRISMA guidelines and reminds us of the importance of checklists to medical publishing. This blog was originally published as one of eight in PLOS Medicine’s 10th Anniversary blog series on the most interesting and influential articles of the last ten years (read the original blog.)

Gilbert and Sullivan’s Lord High Executioner has, sadly, given lists a bad name. Rather than tools of revenge, lists in healthcare, however, have the power to do much good. Atul Gawande’s book on lists has explained why they should be core to medical practice. I’d argue that in medical publishing too they are critical. When Robert Boyle, one of the founders of the UK’s Royal Society, wrote the Spring of the Air he was probably the first to write in such a way that allowed other men (it was only men then of course) to repeat and test his findings.  In this way, he was, in turn, one of the first to legitimize research by making it reproducible.

More than 300 years later we have fully reproducible literature with everything fully reported, right? Wrong. There is a current crisis of confidence in research, with increasing and appropriate concern that many results, especially the most dramatic, often cannot be trusted. Contributing fundamentally (but not exclusively, obviously) to this problem is that whole swathes of the medical and scientific literature are not described in sufficient detail that anyone else can even test. In medicine this crisis literally is life-threatening; patients given treatments as a result of inadequately described studies may at best be treated sub-optimally, at worst harmed or killed.

However, in one important corner of the research endeavour, a group of individuals have, for many years now, been making a determined effort to change this poor reporting and PLOS Medicine is proud to have played its part in it. The CONSORT reporting guidelines for clinical trials and the PRISMA guidelines for Systematic Reviews and Meta-Analyses are, I’d argue two of the most important papers the journal has ever published. The premise behind both of these documents, and the checklists in them and the many other guidelines that we and other PLOS journals have published is very simple: tell us what you did so others can test it.

Is there a downside to this reporting? Yes, perhaps as a weary researcher, having another guideline to abide by and a checklist to fill can seem like one too many hurdles to jump over. And what if guidelines were used in such a way as to judge significance—would they be used to discriminate against them? Are such worries legitimate? I’d say not. And furthermore, in several ways, beyond the simple message of clarity of reporting they illustrate values that are core to PLOS.

Reporting guidelines above all allow us to judge what a paper reports, and what it does not (hint, the clue is in the name). Was this trial randomized or not; how was the search strategy done; these are the most basic of questions. The secondary questions of whether these results are ones that will change your medical practice are not for guidelines to answer. My favourite analogy of a guideline is that using one is akin to turning on a light in a room. It tells you what the room looks like; it doesn’t (unless you live in some Hogwartian dream word) clean the room for you. More prosaically, reporting guidelines are a tool, not a panacea; much like Open Access.

The two guidelines here are themselves revisions on earlier guidelines and this also illustrates an important point i.e. that such guidelines must evolve, that every iteration with more accumulated input to it will improve it.  Arguably, publishing these papers in an Open Access journal such as PLOS Medicine will accelerate this process by enabling visibility, dissemination, reuse and thus ultimately improvement—again something very close to PLOS’ heart.

And the final way in which these guidelines indirectly reflect PLOS’ values are in the way that they are the product of many people’s time, intellect and energy; none of them would have come about unless, as one of the leaders of the EQUATOR initiative once said, when drawing these guidelines up everyone was able to ”leave their egos at the door”.

So, dear researcher, when you feel wearied by a checklist, remember the philosophy, energy and lack of egos that went into making them.  If Robert Boyle and colleagues could understand the importance of good reporting 300 years ago, surely it’s time now to make it a cornerstone of the scientific and medical literature.

You can read the full guidelines here:

Schulz KF, Altman DG, Moher D,  for the CONSORT Group (2010) CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials. PLOS Medicine 2010

Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLOS Medicine 2009