EQUATOR 'pick'

This page highlights interesting articles, thought-provoking quotes, observations and suggestions relating to research reporting and relevant broader issues.

New paper assessing the impact of the CONSORT extension for cluster randomised trials

Published in September 2011 in the BMJ, this paper evaluates the impact of the CONSORT extension for cluster randomised trials regarding the quality of reporting and study methodology in a random sample of 300 trials.

Ivers NM, Taljaard M, Dixon S, Bennett C, McRae A, Taleban J, Skea Z, Brehaut JC, Boruch RF, Eccles MP, Grimshaw JM, Weijer C, Zwarenstein M, Donner A. Impact of CONSORT extension for cluster randomised trials on quality of reporting and study methodology: review of random sample of 300 trials, 2000-8. BMJ. 2011;343:d5886. doi: 10.1136/bmj.d5886. PMID: 21948873

*******

Evaluations of quality of reporting

Over the past year a steady stream of research papers assessing the quality of reporting have been published. The papers cover a wide range of medical specialties and generally all reach similar conclusions: that reporting of what was done and what was found in a research study is inadequate.

Some examples of recent evaluations of quality of reporting are listed below:

Breau RH, Gaboury I, Scales CD Jr, Fesperman SF, Watterson JD, Dahm P. Reporting of harm in randomized controlled trials published in the urological literature. J Urol. 2010;183(5):1693-7. PMID: 20299044

Brundage M, Bass B, Davidson J, Queenan J, Bezjak A, Ringash J, Wilkinson A, Feldman-Stewart D. Patterns of reporting health-related quality of life outcomes in randomized clinical trials: implications for clinicians and quality of life researchers. Qual Life Res. 2011;20(5):653-64. PMID: 21110123

Carlsen B, Glenton C.What about N? A methodological study of sample-size reporting in focus group studies. BMC Med Res Methodol. 2011;11:26. PMID: 21396104

Demauro SB, Giaccone A, Kirpalani H, Schmidt B. Quality of Reporting of Neonatal and Infant Trials in High-Impact Journals. Pediatrics. 2011 Aug 22. [Epub ahead of print] PMID: 21859916

Hurst D. Quality of reporting randomised controlled trials in major dental journals suboptimal. Evid Based Dent. 2011;12(2):52-3. doi: 10.1038/sj.ebd.6400796. PMID: 21701551

Kiehna EN, Starke RM, Pouratian N, Dumont AS. Standards for reporting randomized controlled trials in neurosurgery. J Neurosurg. 2011;114(2):280-5. PMID: 21054137

Langan S, Schmitt J, Coenraads PJ, Svensson A, von Elm E, Williams H; European Dermato-Epidemiology Network (EDEN). The reporting of observational research studies in dermatology journals: a literature-based study. Arch Dermatol. 2010;146(5):534-41. PMID: 20479302

Parsons NR, Hiskens R, Price CL, Achten J, Costa ML. A systematic survey of the quality of research reporting in general orthopaedic journals. J Bone Joint Surg Br. 2011;93(9):1154-9. PMID: 21911523

Toulmonde M, Bellera C, Mathoulin-Pelissier S, Debled M, Bui B, Italiano A. Quality of randomized controlled trials reporting in the treatment of sarcomas. J Clin Oncol. 2011;29(9):1204-9. PMID: 21321290

Thakur B, Shalhoub J, Hill AM, Gohel MS, Davies AH. Heterogeneity of reporting standards in randomised clinical trials of endovenous interventions for varicose veins. Eur J Vasc Endovasc Surg. 2010;40(4):528-33. PMID: 20655771

van der Meersch A, Dechartres A, Ravaud P. Quality of reporting of bioequivalence trials comparing generic to brand name drugs: a methodological systematic review. PLoS One. 2011;6(8):e23611. PMID: 21858184

*******

Reporting guidelines currently in development

Guidelines are currently being developed for the reporting of prognostic (risk prediction) studies, STROBE checklist for conference abstracts, guidelines for reporting manuals to describe complex interventions, with particular reference to mental health services research and the SPIRIT Initiative essential items to be addressed when writing a trial protocol.
Visit our reporting guidelines in development web page for further details.

*******

Implementation of journal’s guidelines improves the quality of reporting of observational studies

In March 2006, the Journal of Pediatric Surgery developed and implemented guidelines for reporting observational studies in pediatric surgery. The journal editors published an editorial (1) introducing these guidelines and detailing the steps required for their implementation. The Journal asked authors to submit a completed guidelines form and asked all their peer reviewers to use the guidelines in their reviews.
A recently published before-and- after study shows a major improvement in the reporting quality of papers published in the journal after the guideline implementation (2).

“The considerable benefit realized from the implementation of the Guidelines, especially when available online, suggests that they are a simple, effective, and feasible way of improving quality of reporting in nonrandomized studies.”


References:
1. Moss RL & Grosfeld JL. A new standard for reporting clinical research in the Journal of Pediatric Surgery. J Ped Surg 2006; 41:4-6.

2. Wynne KE., Simpson BJ., Berman L., Rangel SJ., Grosfeld JL., Moss RL. Results of a longitudinal study of rigorous manuscript submission guidelines designed to improve the quality of clinical research reporting in a peer-reviewed surgical journal. J Ped Surg 2011; 46:131-137. PMID:21238654

*******

Reporting of subgroup analyses

Subgroup analyses are commonly reported in randomised controlled trials and their results can substantially influence clinical decision making. However, evidence suggests that subgroup findings can be misleading. Clear reporting (e.g. whether the analyses were prespecified or not) and an understanding of other factors underlying reporting of subgroup analyses (e.g. trial funding) may aid in the interpretation and appropriate use of subgroup findings.

Two research studies were recently published in BMJ:

Myles et al. (1) report trial results that investigated the effect of epidural analgesia for major abdominal surgery for cancer on recurrence-free survival of patients. In the methods section they very clearly report subgroup analyses:

“We planned to include patients’ sex, age, and type of surgery in our adjusted analyses and tests of subgroup interaction. Also, in view of recent publications suggesting a possible effect modification of age groups under 65 years and 65 years or over, local spread at the time of surgery, and survival pattern after surgery, we did additional post hoc exploratory analyses to investigate such effects.”

In the same journal issue, Xin Sun and colleagues investigated the impact of industry funding on reporting of trial subgroup analyses and showed a clear association between industry funding and the misreporting of subgroup analyses (2).


References:
1. Myles PS, Peyton P, Silbert B, Hunt J, Rigg JR, Sessler DI; ANZCA Trials Group Investigators. Perioperative epidural analgesia for major abdominal surgery for cancer and recurrence-free survival: randomised trial. BMJ. 2011 Mar 29;342:d1491. doi: 10.1136/bmj.d1491.
http://www.bmj.com/content/342/bmj.d1491.long

2. Sun X, Briel M, Busse JW, You JJ, Akl EA, Mejza F, Bala MM, Bassler D, Mertz D, Diaz-Granados N, Vandvik PO, Malaga G, Srinathan SK, Dahm P, Johnston BC, Alonso-Coello P, Hassouneh B, Truong J, Dattani ND, Walter SD, Heels-Ansdell D, Bhatnagar N, Altman DG, Guyatt GH. The influence of study characteristics on reporting of subgroup analyses in randomised controlled trials: systematic review. BMJ. 2011 Mar 28;342:d1569. doi: 10.1136/bmj.d1569.
http://www.bmj.com/content/342/bmj.d1569.full

*******

What makes a high quality clinical research paper

And finally we pick not a new article but definitively a very useful paper by Trish Groves “What makes a high quality clinical research paper” – it is short, written in a pragmatic style and it is full of practical advice.

Reference:
1. Groves T. What makes a high quality clinical research paper? Oral Dis. 2010;16(4):313-5. PMID:20233312

*******

Clear instructions for authors of the UK NIHR Health Technology Assessment programme (HTA) reports

We applaud the UK HTA programme for its guidance and resources for authors - they provide very specific requirements on reporting and refer very clearly to the use of appropriate guidelines.

*******

International register of ongoing systematic reviews

In response to a growing concern about reporting biases, and calls for the prospective registration of systematic reviews, the Centre for Reviews and Dissemination (CRD) in York, UK has taken the initiative in establishing an international register of ongoing systematic reviews. To read more information about this register development please visit their website.

*******

Misleading communication of risk

An article by Gerd Gigerenzer and his colleagues calls for more transparent reporting of risk in the abstracts of published papers.

"Conveying relative risks without baseline risk is the first "sin" against transparent reporting. The second is mismatched framing—reporting benefits, such as relative risk reductions, in big numbers and harms, such as absolute risk increases, in small numbers"

"How can those who are responsible for accurate communication of risk do better? And who should be monitoring them to ensure that they do? Steps can be taken to improve the transparency of risk communication.Firstly, editors should enforce transparent reporting in journal abstracts: no mismatched framing, no relative risks without baseline risks, and always give absolute numbers such as absolute risks and NNT.
Secondly, institutions that subscribe to medical journals could give journal publishers two years to implement the first measure and, if publishers do not comply, cancel their subscriptions.
Thirdly, writers of guidelines, such as the CONSORT statement, should stipulate transparent reporting of benefit and harms in abstracts."

Reference:
Gigerenzer G, Wegwarth O, Feufel M. Misleading communication of risk. BMJ 2010; 341: doi:10.1136/bmj.c4830

*******

White hat bias: a threat to the integrity of scientific publishing

Cope and Allison, the authors of this article, define white hat bias as "bias leading to distortion of information in the service of what may be perceived to be righteous ends", They note that when discussing conflicts of interests attention mostly focuses on financial aspects and far less attention is being devoted to the other factors that may potentially bias research and research reporting. This commentary draws on their previous research related to breastfeeding and obesity and encourages "faithful reporting, acknowledging study limitations and evaluating bodies of evidence without selectively excluding information on the basis of its desirability".

Reference:
Cope MB, Allison DB. White hat bias: a threat to the integrity of scientific reporting. Acta Paediatr. 2010; 99(11):1615-7. doi: 10.1111/j.1651-2227.2010.02006.x.
PMID:21039822

*******

CONSORT 2010 commentary

A number of commentaries introducing new CONSORT 2010 have been published; we would like to highlight the article by Prof Hywel Williams:
Williams HC. Cars, CONSORT 2010, and Clinical Practice. Trials 2010, 11:33. PMID:20334635

*******

Editorial: Who is afraid of reviewers' comments?

Another interesting editorial was published by the editors of the European Journal of Clinical Investigation. It discusses the problems of papers resubmitted for publication that have already been peer reviewed but ignore the provided comments. Using examples from their own journal, the editors alert to the questionable practices of some authors and discuss these in much broader context of the general peer review issues.
Ioannidis JP, Tatsioni A, Karassa FB: Who is afraid of reviewers' comments? Or, why anything can be published and anything can be cited. Eur J Clin Invest 2010, 40:285-287. PMID:20486989

*******

Quality of prognosis research

Editorial by Sorensen and Rothman (1) discusses the quality of prognosis research and a registration of observational studies. The suggestion to register observational studies was made by Hemingway and colleagues (2) and recently discussed in the BMJ editorial (3).

1. Sorensen HT, Rothman KJ. The prognosis for research. BMJ 2010;340:c703. PMID:20164129
2. Hemingway H, Riley RD, Altman DG. Ten steps towards improving prognosis research. BMJ 2009;339:b4184. PMID:20042483
3. Loder E, Groves T, MacAuley D. Registration of observational studies. BMJ 2010;340:c950. PMID:20167643

*******

Surrogate end points

The article by David Grimes and colleagues highlights problems of using surrogate end points that lack validation and asks journals to consider warning their readers about the hidden dangers.
Grimes DA, Schulz KF, Raymond EG. Surrogate end points in women's health research: science, protoscience, and pseudoscience. Fertil Steril 2010 Apr;93(6):1731-4. PMID:20153470

*******

Outcome reporting bias

A research study carried out by Kirkham and colleagues examined the prevalence of outcome reporting bias and its impact on Cochrane reviews. The outcome reporting bias, which happens when only a subset of the originally recorded outcome variables is selected for publication on the basis of obtained results, is an under-recognised problem that affects the conclusions in a substantial proportion of Cochrane reviews.
The article alerts systematic reviewers to this problem and offers a nine point classification system for missing outcome data in randomised trials (ORBIT).
Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 2010;340:c365. PMID:20156912

*******

A vision for the European Journal of Clinical Investigation: note from the new editors

Excellent editorial setting up a clear vision for implementing steps leading to more transparent and accurate reporting; read at Eur J Clin Invest. 2010 Jan;40(1):1-3.

*******

Revealing the wood and the trees: reporting qualitative research

The article by Blignault and Ritchie provides a very useful general guide to presenting qualitative research for publication. The article is very clearly written and includes many examples of good reporting. Although there is no CONSORT-type checklist to follow anyone writing up the qualitative study would  benefit from its reading.

“As the researcher, you know what it is you have done, and why, but what is obvious to you may not be obvious to the reader. Therefore you need to take the reader through the research process with you, giving reasons for your choices and decisions and guiding them to the study conclusions.”

The article is now included in the EQUATOR Library – Guidance for reporting qualitative research.
Blignault I, Ritchie J. Revealing the wood and the trees: reporting qualitative research. Health Promot J Austr 2009 Aug;20(2):140-5. PMID:19642963

*******

Reporting guidelines in animal research

The need for clear, transparent and accurate reporting is not confined to the area of clinical research; it is a clearly universal requirement of any good research publication.

Recently, two initiatives were established to improve the reporting of animal research.

The REFLECT Statement (Reporting Guidelines For Randomized Controlled Trials for livestock and food safety) provides 22-item checklist guiding the authors to address unique aspects of reporting of livestock trials. The REFLECT Statement is a result of a robust development process that is clearly described in the paper (see reference below). It is based on the CONSORT Statement (guideline for reporting randomised trials with human participants); the checklist clearly indicates which parts were left out of the original CONSORT and what are the additions. This is a very good practice that helps users to put the new guideline in the context of existing guidance. Unfortunately we do not always see this in new guidelines using  previously published work (e.g. the REHBaR guideline for reporting experiments in homeopathic basic research mentioned above lifted some of the CONSORT items verbatim but without making this clear).

References:
O'Connor AM, Sargeant JM, Gardner IA, Dickson JS, Torrence ME, Dewey CE, et al. The REFLECT Statement: Methods and Processes of Creating Reporting Guidelines for Randomized Controlled Trials for Livestock and Food Safety by Modifying the CONSORT Statement. Zoonoses Public Health 2010 Jan 12.
PMID:20070653
Sargeant JM, O'Connor AM, Gardner IA, Dickson JS, Torrence ME, Dohoo IR, et al. The REFLECT Statement: Reporting Guidelines for Randomized Controlled Trials in Livestock and Food Safety: Explanation and Elaboration. Zoonoses Public Health 2010 Jan 12.
PMID:20070652

Website: http://www.reflect-statement.org/statement/

Another initiative to improve reporting of research using animals is driven by the UK National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC3Rs). Carol Kilkenny and her colleagues carried out a large comprehensive survey of the quality of experimental design, statistical analysis and reporting (see reference below).The survey identified a number of issues – particularly reporting omissions. Building on these results, the authors are developing a set of reporting guidelines to assist transparent and complete reporting of animal research.
Kilkenny C, Parsons N, Kadyszewski E, Festing MF, Cuthill IC, Fry D, et al. Survey of the quality of experimental design, statistical analysis and reporting of research using animals. PLoS One 2009;4(11):e7824. PMID:19956596

*******

Peer reviewing and reporting search strategies

High quality comprehensive searching is a crucial part of systematic review methodology. The following articles provide information on how to assess and report search strategies.

  • Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. Journal of Clinical Epidemiology 2009;62(9):944-52. PMID: 19230612
  • Sampson M, McGowan J, Lefebvre C, Moher D, Grimshaw J. PRESS: Peer Review of Electronic Search Strategies [Technology report number 477]. Ottawa: Canadian Agency for Drugs and Technologies in Health, 2008:1-41. Available from: http://www.cadth.ca/index.php/en/publication/781
  • Lefebvre C, Manheimer E, Glanville J. Chapter 6: Searching for studies. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.2 (updated September 2009). The Cochrane Collaboration, 2009. Available from: www.cochrane-handbook.org

*******

Comparative effectiveness research - a new US government initiative

A recent PLoS editorial discusses comparative effectiveness research stimulated by the US government funding; the editorial quotes from the Institute of Medicine report:
“Objectivity will be central to the public's trust and confidence in the integrity of the CER Program. Conflict of interest and bias in clinical research—published in even the most respected medical journals—is well-documented…. Selective reporting or publication bias is common. Positive findings are more likely to be published than negative results…. In addition, there have been significant instances in which leading journals have not sufficiently enforced disclosure requirements for authors and reviewers…. CER is as vulnerable to bias and conflict of interest as any other area of medical research. The ultimate value of the CER enterprise will rest, in part, on vigilant attention to these issues”

The comparative effectiveness research movement highlights the need for openness in the design and reporting of all types of research.

The editorial concludes:
"Medical practice stands to benefit from a clearer understanding of what works effectively and what doesn't. Such knowledge can only emerge from unbiased reporting of the research that supports it, including data that contest prevailing practices. The integrity of the medical literature requires that negative results assume their rightful place alongside those that support blockbuster sales and breakthrough headlines."

  • The PLoS Medicine Editors (2009) Ensuring Integrity in Comparative Effectiveness Research: Accentuate the Negative. PLoS Med 6(9): e1000152. doi:10.1371/journal.pmed.1000152

*******

Responsible conduct of research

For those interested in discussing the meaning of responsible conduct of research, there is an opportunity at the US Office of Research Integrity blog

  • Meaning of ‘Responsible Conduct of Research (RCR)’ blog
*******

An increasing problem in research reporting - image manipulation

Nature News highlights an increase in the number of ‘improved’ images in submitted manuscripts presenting journals with yet another problem that compromises reliability of published research.
Read the full news on the Nature News page.

*******

International Council for Science.Freedom, Responsibility and Universality of Science. 2008:
Responsibilities of scientists, p.10
On the conduct of science
The starting point for considering the conduct of science is that all work should be of the highest quality and, in so far as is possible, reported in a way that makes it reproducible and/or verifiable. Poor quality science—no matter how important the field—should not be tolerated and all efforts should be made to ensure the robustness of any information or data that are produced. In the 1940s the American sociologist Robert Merton formulated four principles, which he believed constituted a ‘moral consensus’ in science. The, so called, CUDOS norms (Communalism, Universalism, Disinterestedness and Organized Scepticism ) may no longer be sufficient to fully accommodate the changes that have taken place in science and society since that time. Nevertheless, they remain an important foundation on which to build. …

… All scientists have a responsibility to ensure that they conduct their work with honesty and integrity; to ensure that methods and results are reported in an accurate, orderly, timely and open fashion. …

Suggested roles for key institutions in dealing with issues relating to scientific freedoms and responsibilities, p.19
Research providers and funders
Promote and support high-quality science, with no discrimination on the basis of ethnic origin, religion, citizenship, language, political stance, gender, sex or age.
Develop good research practice and ethics guidelines to be followed as a condition of funding. Monitor the implementation of these guidelines and, where appropriate, develop mechanisms for dealing with individual cases.

*******

Roger J. Vallance. Research ethics: Reforming postgraduate formation. Issues In Educational Research, Vol 15, 2005:

'Research quality always rests on the researcher - oversight boards and even research supervisors can only be aware of a small part of the interactions that comprise research between and among human subjects. Good research is more likely to be done by well informed and well intentioned researchers, and research training needs to address both the technical side of research preparation and the human, ethical side of research formation.'

'Research ethics must inform the whole of the research process from shaping a research question to the publication of findings.'

*******


Page last edited: 18 November 2011