New reporting guidelines for trials in artificial intelligence published

The EQUATOR Network welcomes the publication of two long-awaited reporting guidelines: CONSORT-AI and SPIRIT-AI, the extensions for artificial intelligence (AI) for clinical trials and their protocols. This is very timely since there has been much hype on the use of various forms of AI in medicine. In recent years the number of studies in this area has multiplied, however, the quality of the reporting of these studies has been inadequate, which prompted the development of guidelines for their reporting

AI systems have been researched and used for a long time in medicine. However, it seems that recent advances in some of its techniques, such as deep learning and neural networks, have fuelled the development of diagnostic and prognostic tools and models in health that promise to make it easier to detect disease and make treatment decisions. Several recent critical reviews, however, have evidenced that research studies underpinning model development and validation are poorly reported, and therefore impossible to reproduce, as one can’t be sure that good methodological choices were made if they are not described in detail.

To tackle inadequate reporting, meta-researchers have developed the extensions for CONSORT (the reporting guideline for clinical trials) and SPIRIT (for protocols) for interventions using AI or machine learning systems. Both extensions for CONSORT and SPIRIT are available through the EQUATOR database. They were developed mainly by researchers from the University of Birmingham, with representatives from the Steering Groups of the main CONSORT and SPIRIT guidelines and the EQUATOR Network

What is AI?

Artificial intelligence is any type of computerised systems that can perform tasks that would typically require human intelligence or the capacity of making decisions. By delegating to a computer some of these tasks, it is possible to lessen the burden to humans like doctors, radiologists, nurses and other health professionals when they need to detect disease, triage patients and decide a clinical approach.

What is machine learning?

Machine learning is a branch of AI that allows computer programs to learn decisions and tasks. AI systems train machines through millions of repetitions, using models or algorithms that solve problems using specific rules. For example, suppose a computer can learn that a particular feature of an image in an exam equals a suspicion of a malignant lesion. In that case, it can help detect cancer in a large cohort of patients more quickly.

How can reporting guidelines help?

The CONSORT-AI reporting guideline, for clinical trials testing interventions in AI, will help authors to report their papers more fully by setting out important content items that should be described in all articles reporting on a trial of an AI intervention. For clinical trials (studies that test an intervention in health), it is recommended that researchers address in their report the 25 items from the CONSORT checklist and also the 14 items from the new checklist for AI. For example, CONSORT-AI (PDF) asks authors to state the intended use of the AI tool and also how the intervention was integrated into the clinical setting, how the data were acquired, how the AI intervention contributes to clinical decision-making and other items. The checklist of the AI extension is presented together with the main CONSORT checklist so that authors can easily access both checklists when writing up their study for publication.

Authors publishing protocols (or the project methodology description) of clinical trials or depositing them in registry databases or submitting them to ethics committees should describe the methodology in enough detail to allow the evaluation and reproducibility of procedures. They can do this easily using the SPIRIT reporting guideline, with 33 items. Now, for protocols describing studies using AI systems, authors can use the SPIRIT-AI (PDF) extension, which sets out additional AI-specific details that should be included in all protocols describing a trial involving AI. For example, SPIRIT-AI states that authors should specify any plans to identify and analyse performance errors from the AI system — an important methodological step that should be clearly reported. Including information about the availability of code or restrictions to its use is also recommended in the protocol phase of the study.