Enhancing the QUAlity and Transparency Of health Research
Deputy Editor Claire Barnard talks to Isabelle Boutron about her research article published recently in BMC Medicine. Isabelle and her colleagues demonstrate that the COBWEB writing aid tool can improve the completeness of clinical trial reporting. Here, Prof Boutron answers our questions about the study, and discusses the implications of their findings for improving the reporting of research.
Isabelle Boutron is Professor of Epidemiology at Paris Descartes University, a researcher at the INSERM – Sorbonne Paris Cité Epidemiology and Statistics Research Centre (UMR 1153) in the METHODS team, co-convenor of the Cochrane Bias Methods group and deputy director of the French EQUATOR centre.
Isabelle has published more than 100 peer reviewed articles, and her research activities mainly focus on randomized and non-randomized studies evaluating interventions, the methodological issues when evaluating nonpharmacologic treatments, research transparency and the dissemination and interpretation of research results.
What are the problems associated with incomplete reporting of clinical research?
In 2009, Iain Chalmers and Paul Glasziou showed that at least 50% of research reports were not usable because of incomplete reporting. This waste in research has severe consequences for researchers, clinicians, decision makers and patients.
Incomplete reporting precludes advances in scientific knowledge, improvements in the care provided to patients and in public health. In particular, incomplete reporting makes it challenging or sometimes impossible to perform systematic reviews.
A recent study showed that of all trials included in Cochrane systematic reviews, 41% did not provide sufficient reporting to assess their risk of bias. Additionally, Chan et al found incomplete and selective reporting of outcomes for a median of 50% of efficacy and 65% of harm outcomes per trial.
This selective reporting is responsible for biased treatment effect estimates. Also, physicians cannot implement the results of positive trials in clinical practice because interventions are sufficiently described in less than half of published reports.
How do reporting guidelines help to address these problems?
Reporting guidelines such as the CONSORT statement have been developed to improve the completeness of reporting and reduce waste from incomplete reporting. These reporting guidelines aim to guide authors, reviewers and editors to make sure that the minimum set of information is reported.
Nevertheless, the adherence of authors to these guidelines remains low and the quality of reporting insufficient
These guidelines have been widely disseminated and are mainly implemented by editors when manuscripts are submitted or reviewed. Nevertheless, the adherence of authors to these guidelines remains low and the quality of reporting insufficient.
These implementation strategies intervene very late in the life span of a manuscript, after the main author has spent hours refining their manuscript and obtaining agreement for publication from all co-authors. Consequently, authors may overlook making important changes to their manuscript, including verifying that all key information is properly reported. Further, the checklist may not be sufficiently explicit.
For example, the CONSORT item for reporting a rehabilitation programme states to describe: “The interventions for each group with sufficient details to allow replication, including how and when they were actually administered”.
Authors may not understand that they should report all the following information: a) the type of the intervention (name of the program), b) the content of each session, c) if the intervention was delivered to an individual or a group, d) whether the treatment was supervised, e) any instruments used to provide information (computers, tablets, smartphones, other), f) the number and timing of sessions, g) the duration of each session, h) each main component of each session, i) the overall duration of the intervention, j) any procedures for tailoring the interventions to individual participants and k) any co-interventions permitted or restricted.
The CONSORT “Explanation and Elaboration” document provides this information, but authors may be overwhelmed by reading this 28 page document mixing guidance on why each item is important and should be reported, statistics of inadequate reporting, and guidance on how to report it with examples of adequate reporting.
We believe that we need to rethink the implementation of these guidelines and develop new strategies. Particularly, we propose to: 1) intervene earlier, at the stage of the first draft of the manuscript, 2) provide more explicit, succinct guidance along with an example of adequate reporting, and 3) tailor the example according to the context of the trial.
What is the COBWEB tool and how was it developed?
The COBWEB tool is an online writing aid tool dedicated to authors when writing the first draft of their article.
The COBWEB tool is an online writing aid tool dedicated to authors when writing the first draft of their article. The principle is to provide a template shell with each CONSORT item reported with the key elements that need to be presented in the form of several bullet points with an example of adequate reporting.
This template is followed by a large text box where the participants would write their text. After completing all text boxes addressing the CONSORT items, the tool provides a formatted word document.
What did your study do and find?
We evaluated the impact of this tool on the completeness of reporting of two-arm parallel group randomized controlled trials evaluating pharmacologic and non-pharmacologic interventions. We performed a split-manuscript randomized controlled trial (RCT) involving 41 students.
Each participant was randomly allocated to a different real RCT protocol. They had to write six domains of the methods section (‘trial design’, ‘randomization’, ‘blinding’, ‘participants’, ‘interventions’, and ‘outcomes’) of the manuscript for the protocol they received over a four-hour period.
They had access to the tool for three domains randomly selected out of the 6. Our results revealed a large effect of the writing tool on the completeness of reporting.
Could you describe the implications of your study for authors, reviewers and editors?
This study has two main implications. First, it shows that we need to rethink the implementation of reporting guidelines and develop new models. Second, it demonstrates that it is feasible and necessary to evaluate these strategies using high level of evidence with experimental design. In fact, most interventions used to improve the quality of reporting are assessed using before and after studies.
How will you promote use of the tool among researchers writing up clinical trials?
The tool will be freely available with a link from the EQUATOR website.
The tool will be freely available with a link from the EQUATOR website. We will involve different stakeholders such as editors, funders and researchers to promote the use of the tool. Particularly, we will contact academic funders and invite them to send the tool to all investigators that have recently completed their trial. The tool will also be used during the training sessions organized by EQUATOR.
Do you think there will be any barriers to its uptake?
We are not aware of important barriers to its uptake. However, we will conduct qualitative studies to identify possible barriers and refine the tool accordingly.
What further work should be done to build upon the COBWEB tool?
This study is a proof of concept study providing very promising results for authors, reviewers and editors. COBWEB has currently been developed for six different domains: ‘trial design’, ‘randomization’, ‘blinding’, ‘participants’, ‘interventions’, and ‘outcomes’ for the main CONSORT and the extension for Non-Pharmacological treatments.
We are extending the tool for all items of the main CONSORT statement, along with those of other CONSORT statement extensions. Additionally, the principle of the COBWEB tool could apply for reporting guidelines for other types of studies, such as epidemiological studies, diagnostic studies, systematic reviews etc.
Access the COBWEB Tool
The COBWEB tool can be accessed at: http://cochrane.fr/cobweb/
(Please note that when promoted to ‘choose an ID’ you need to create your own ID to log in)
I recently attended the 8th Congress on Peer Review in Chicago (10-12 September 2017). Unusually this event takes place only every 4 years, so this was the 8th such Congress in a series starting in 1989. I’m one of very...