Deputy Editor Claire Barnard talks to Isabelle Boutron about her research article published recently in BMC Medicine. Isabelle and her colleagues demonstrate that the COBWEB writing aid tool can improve the completeness of clinical trial reporting. Here, Prof Boutron answers our questions about the study, and discusses the implications of their findings for improving the reporting of research.

Isabelle Boutron is Professor of Epidemiology at Paris Descartes University, a researcher at the INSERM – Sorbonne Paris Cité Epidemiology and Statistics Research Centre (UMR 1153) in the METHODS team, co-convenor of the Cochrane Bias Methods group and deputy director of the French EQUATOR centre.

Isabelle has published more than 100 peer reviewed articles, and her research activities mainly focus on randomized and non-randomized studies evaluating interventions, the methodological issues when evaluating nonpharmacologic treatments, research transparency and the dissemination and interpretation of research results.

What are the problems associated with incomplete reporting of clinical research?

In 2009, Iain Chalmers and Paul Glasziou showed that at least 50% of research reports were not usable because of incomplete reporting. This waste in research has severe consequences for researchers, clinicians, decision makers and patients.

Incomplete reporting precludes advances in scientific knowledge, improvements in the care provided to patients and in public health. In particular, incomplete reporting makes it challenging or sometimes impossible to perform systematic reviews.

A recent study showed that of all trials included in Cochrane systematic reviews, 41% did not provide sufficient reporting to assess their risk of bias. Additionally, Chan et al found incomplete and selective reporting of outcomes for a median of 50% of efficacy and 65% of harm outcomes per trial.

This selective reporting is responsible for biased treatment effect estimates. Also, physicians cannot implement the results of positive trials in clinical practice because interventions are sufficiently described in less than half of published reports.

How do reporting guidelines help to address these problems?

Reporting guidelines such as the CONSORT statement have been developed to improve the completeness of reporting and reduce waste from incomplete reporting. These reporting guidelines aim to guide authors, reviewers and editors to make sure that the minimum set of information is reported.

Nevertheless, the adherence of authors to these guidelines remains low and the quality of reporting insufficient

These guidelines have been widely disseminated and are mainly implemented by editors when manuscripts are submitted or reviewed. Nevertheless, the adherence of authors to these guidelines remains low and the quality of reporting insufficient.

These implementation strategies intervene very late in the life span of a manuscript, after the main author has spent hours refining their manuscript and obtaining agreement for publication from all co-authors. Consequently, authors may overlook making important changes to their manuscript, including verifying that all key information is properly reported. Further, the checklist may not be sufficiently explicit.

For example, the CONSORT item for reporting a rehabilitation programme states to describe: “The interventions for each group with sufficient details to allow replication, including how and when they were actually administered”.

Authors may not understand that they should report all the following information: a) the type of the intervention (name of the program), b) the content of each session, c) if the intervention was delivered to an individual or a group, d) whether the treatment was supervised, e) any instruments used to provide information (computers, tablets, smartphones, other), f) the number and timing of sessions, g) the duration of each session, h) each main component of each session, i) the overall duration of the intervention, j) any procedures for tailoring the interventions to individual participants and k) any co-interventions permitted or restricted.

The CONSORT “Explanation and Elaboration” document provides this information, but authors may be overwhelmed by reading this 28 page document mixing guidance on why each item is important and should be reported, statistics of inadequate reporting, and guidance on how to report it with examples of adequate reporting.

We believe that we need to rethink the implementation of these guidelines and develop new strategies. Particularly, we propose to: 1) intervene earlier, at the stage of the first draft of the manuscript, 2) provide more explicit, succinct guidance along with an example of adequate reporting, and 3) tailor the example according to the context of the trial.

What is the COBWEB tool and how was it developed?

The COBWEB tool is an online writing aid tool dedicated to authors when writing the first draft of their article.

The COBWEB tool is an online writing aid tool dedicated to authors when writing the first draft of their article. The principle is to provide a template shell with each CONSORT item reported with the key elements that need to be presented in the form of several bullet points with an example of adequate reporting.

This template is followed by a large text box where the participants would write their text. After completing all text boxes addressing the CONSORT items, the tool provides a formatted word document.

What did your study do and find?

We evaluated the impact of this tool on the completeness of reporting of two-arm parallel group randomized controlled trials evaluating pharmacologic and non-pharmacologic interventions. We performed a split-manuscript randomized controlled trial (RCT) involving 41 students.

Each participant was randomly allocated to a different real RCT protocol. They had to write six domains of the methods section (‘trial design’, ‘randomization’, ‘blinding’, ‘participants’, ‘interventions’, and ‘outcomes’) of the manuscript for the protocol they received over a four-hour period.

They had access to the tool for three domains randomly selected out of the 6. Our results revealed a large effect of the writing tool on the completeness of reporting.

Could you describe the implications of your study for authors, reviewers and editors?

This study has two main implications. First, it shows that we need to rethink the implementation of reporting guidelines and develop new models. Second, it demonstrates that it is feasible and necessary to evaluate these strategies using high level of evidence with experimental design. In fact, most interventions used to improve the quality of reporting are assessed using before and after studies.

How will you promote use of the tool among researchers writing up clinical trials?

The tool will be freely available with a link from the EQUATOR website.

The tool will be freely available with a link from the EQUATOR website. We will involve different stakeholders such as editors, funders and researchers to promote the use of the tool. Particularly, we will contact academic funders and invite them to send the tool to all investigators that have recently completed their trial. The tool will also be used during the training sessions organized by EQUATOR.

Do you think there will be any barriers to its uptake?

We are not aware of important barriers to its uptake. However, we will conduct qualitative studies to identify possible barriers and refine the tool accordingly.

What further work should be done to build upon the COBWEB tool?

This study is a proof of concept study providing very promising results for authors, reviewers and editors. COBWEB has currently been developed for six different domains: ‘trial design’, ‘randomization’, ‘blinding’, ‘participants’, ‘interventions’, and ‘outcomes’ for the main CONSORT and the extension for Non-Pharmacological treatments.

We are extending the tool for all items of the main CONSORT statement, along with those of other CONSORT statement extensions. Additionally, the principle of the COBWEB tool could apply for reporting guidelines for other types of studies, such as epidemiological studies, diagnostic studies, systematic reviews etc.

This blog was written by Claire Barnard, Deputy Editor, BMC Medicine at BioMed Central. It has been republished with permission. The original blog is available at the BioMed Central website.

Access the COBWEB Tool
The COBWEB tool can be accessed at: http://cochrane.fr/cobweb/
(Please note that when promoted to ‘choose an ID’ you need to create your own ID to log in)

Daniël A. Korevaar & Jérémie F. Cohen, Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Academic Medical Center, University of Amsterdam, The Netherlands

Over 250 reporting guidelines are currently available in the EQUATOR library. One of the most established of these is the STARD (Standards for the Reporting of Diagnostic Accuracy Studies) statement (1, 2).

STARD, first launched in 2003, was developed by an international group of methodologists, statisticians, reviewers and editors. It aims to improve the reporting of studies that evaluate the diagnostic accuracy of medical tests. Incomplete reporting is problematic as it impedes the identification and reproducibility of the study, as well as a proper appraisal of the internal and external validity. The statement contains a checklist of 25 items that should be reported to make the study report fully informative.

The statement was initially simultaneously published in eight major medical journals. Together, these publications have now been cited more than 2,000 times (Web of Knowledge, March 2015). Over the years, multiple other journals published the STARD statement or the accompanying “Explanation & Elaboration” document, translations are available in seven languages, and at least 30 journals published one or more editorials about STARD, usually to highlight the importance of its use. More than 200 journals explicitly endorse STARD, which means that they require or recommend the use of the checklist in their instructions to authors.

Based on these data, the impact of STARD seems to be impressive. Unfortunately, the success of a reporting guideline can only be evaluated by whether or not it has achieved its main goal: improving the quality of reporting. Based on two of our recent evaluations we can conclude that this goal has been achieved, but only to a moderate extent.

We have performed a systematic review in which we included all evaluations that had assessed adherence of published diagnostic accuracy studies to the STARD checklist (3). We found 16 of these evaluations, together analyzing the reporting of 1,496 diagnostic accuracy study reports in various fields of research. Across these evaluations, the mean number of items reported varied from 9.1 to 14.3 out of 25 STARD items. Not surprisingly, this led all the included studies to conclude that reporting was generally poor, medium, suboptimal, or needed improvement. Six of these evaluations quantitatively compared the reporting of diagnostic accuracy studies that were published post-STARD with those that were published pre-STARD. When we combined them in a meta-analysis, we found a modest but significant increase of 1.4 reported items after STARD’s launch in 2003. However, because most of these evaluations assessed diagnostic accuracy studies published in the first few years after STARD’s launch, it may have been too early to expect large improvements.

We also performed an analysis of more recent studies (4). We assessed the reporting of 112 diagnostic accuracy studies published in twelve high-impact journals in 2012. Expectations were high, as all but one of these journals were STARD endorsers. Unfortunately, on average, the studies reported only 15.3 out of 25 STARD items. Yet this is a significant improvement compared to studies published in the same journals in 2000 and 2004, when the mean number of items reported was 11.9 and 13.6, respectively (5).

We conclude from these two studies that the completeness of reporting of diagnostic accuracy studies has improved in the 10 years after the launch of STARD, but that it remains suboptimal for many articles.

Over the past year, the STARD group, currently consisting of over 85 people, has been working on an update of the checklist. The three main goals of this update are

(1) to facilitate the use of the checklist by rearranging and rephrasing items

(2) to include new information, based on improved understanding of sources of bias and variability and other issues in diagnostic accuracy studies, and

(3) to improve consistency with other reporting guidelines such as CONSORT.

After two web-based surveys and a live two-day meeting in Amsterdam, a pre-final version of STARD 2015 has now been put together and is undergoing piloting. The final checklist is planned to be launched late 2015.

What do we learn from the first ten years of STARD, and how can we make sure that STARD 2015 will further improve reporting quality?

Because of the widespread attention that STARD has received by medical journals, and because of the large amount of STARD adopters among these journals, we expected that major improvements in the reporting of diagnostic accuracy studies would automatically follow. Our evaluations have shown that this is not the case. Other well-known reporting guidelines, such as CONSORT and PRISMA, have faced similar problems (6). Apparently, dissemination of a reporting guideline cannot solely focus on journals. Developers of reporting guidelines should continue to seek innovative ways to reach authors, reviewers and editors, and to convince them of the necessity of complete reporting. EQUATOR plays an important role in this.

For STARD 2015, we aim to publish online training material and workshops, build templates to facilitate writing and peer-reviewing of study reports, and encourage the development of extensions specifically designed for different fields of research. In addition, a close collaboration with the EQUATOR network should make sure that a wide audience will be reached. This collaboration will initially comprise the gradual provision of much more information related to the STARD guideline at the EQUATOR website.

With STARD 2015, we further hope to convince the scientific community of the necessity and simplicity of complete reporting of diagnostic accuracy studies.

Reference List

(1)    Bossuyt PM, Reitsma JB, Bruns DE et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Clin Chem 2003;49:7-18.

(2)    Bossuyt PM, Reitsma JB, Bruns DE et al. Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative. Radiology 2003;226:24-28.

(3)    Korevaar DA, van Enst WA, Spijker R, Bossuyt PM, Hooft L. Reporting quality of diagnostic accuracy studies: a systematic review and meta-analysis of investigations on adherence to STARD. Evid Based Med 2014;19:47-54.

(4)    Korevaar DA, Wang J, van Enst WA et al. Reporting Diagnostic Accuracy Studies: Some Improvements after 10 Years of STARD. Radiology 2015;274:781-789.

(5)    Smidt N, Rutjes AW, van der Windt DA et al. The quality of diagnostic accuracy studies since the STARD statement: has it improved? Neurology 2006;67:792-797.

(6)    Turner L, Shamseer L, Altman DG et al. Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev 2012;11:MR000030.

Elizabeth’s article originally appeared on the BioMed Central blog network on 27 February 2015.