Research shows that the findings from clinical trials on treatments for mental disorders have a high risk of bias.
Are research results that show how various treatments can help people with mental disorders reliable? Not necessarily.
New research recently published in the International Journal of Eating Disorders finds a severe risk of bias in the published research results on treatments for mental disorders.
Other trials have replicated the results of only 7 of 201 trials of potential treatments for anorexia.
“Our study shows that we need to be much more cautious about interpreting the published results of clinical trials related to mental disorders. People think that if a research result is published based on a clinical trial, it is credible, but behind the façade, things might not look as they should,” explains author Daniel Quintana, Senior Researcher, Institute of Clinical Medicine, University of Oslo, Norway.
Registration should prevent bias but does not
For clinical trials involving people, registering the trial at, for example, ClinicalTrials.gov is good scientific practice.
Researchers should do this before the trial begins, because this can ensure that the researchers follow consistent prospective methods in their research and do not represent a selected part of their research as the main results.
For example, researchers could use 20 measures to determine whether a treatment for anorexia has been effective, and perhaps only one of these measures demonstrates that the treatment is effective.
If the researchers only publish the results using the successful measure, this treatment will appear to be effective, although effectiveness is unlikely.
Registration at ClinicalTrials.gov ensures that other people can examine the researchers’ methods and findings and confirm that the researchers followed their initial clinical protocol. This system was established to counter bias.
“The researchers are supposed to register the trial before they start collecting data, but unfortunately researchers can still get permission to register a clinical trial after they have started to collect data. This contributes to undermining the system,” says Daniel Quintana.
Only 41 of the 201 registered trials have been published
Daniel Quintana and colleagues investigated the outcomes of 201 clinical trials of treatments for anorexia based on the initial clinical protocols registered at ClinicalTrials.gov.
Only 101 of these trials were ever completed. The other 100 were not.
A clinical trial may not be completed for many reasons. The researchers might not have enough money, but they could also have discovered during the clinical trial that their treatment was not as effective as they had hoped.
In addition, Daniel Quintana and his team found that researchers published the results from only 41 of the 100 completed trials. The reasons may be the same as those mentioned previously, but another possibility is that no journals were interested in publishing negative results.
“There is a strong bias towards publishing positive results. If the published research shows that many trials of treatments for anorexia have been carried out with many positive results, this biases the reality of what the overall research might actually show, because only one quarter of the results have ever been published,” says Daniel Quintana.
Both doctors and laypeople follow clinical trials
Daniel Quintana says that the data registered at ClinicalTrials.gov are often not followed up, and this is a major problem.
Both doctors and laypeople have access to ClinicalTrials.gov, and many follow the trials based on personal or professional interest. They want to know why a clinical trial has been stopped or not published.
Knowing whether a treatment is ineffective is relevant, but this information is not available today.
Only eight clinical trials were registered before data were collected
Daniel Quintana went a step further in his study and found that only eight trials actually published results based on the initial clinical protocol registered at ClinicalTrials.gov.
Most of the trials had a high risk of bias since they were not registered until after data collection began, and prospective registration is required to determine whether there is a risk that a secondary result has been turned into a main result, for instance.
The gold standard in clinical research is replication of the results of trials, but the findings were replicated in only 7 of the 201 registered trials.
Daniel Quintana suspects that the disappointing results of his research apply not only to clinical trials of treatment for anorexia but presumably to trials involving other mental disorders.
“Unfortunately, this happens all too often, and this again contributes to bias in clinical research results,” he explains.
An emerging publication format can help avoid bias
Daniel Quintana thinks that new standardized guidelines for clinical research should be introduced to ensure that researchers comply with the procedures intended to prevent bias.
He hopes that more journals will use the Registered Report format for accepting manuscripts for publication. This ensures that researchers get their trial approved for publication before they carry it out.
“This is a great way to report clinical trial outcomes because it ensures that researchers publish the results of what they set out to investigate. Then whether the results are positive or negative does not matter,” says Daniel Quintana.
Conversely, the journals provide an in-principle agreement to publish the results of the trials for which they have approved the choice of methods and objectives.
According to Daniel Quintana, this can significantly avoid bias in clinical research because all results prospectively approved by the journals will be published.
“Several journals have already adopted the registered report format. The highest-profile journals are Nature Human Behaviour, BMC Medicine and eLife, which all provide the option to approve trials for publication based on the choice of methods, even before the researchers start collecting data or have any results at all,” explains Daniel Quintana.
“Registration, reporting, and replication in clinical trials: the case of anorexia nervosa” has been published in the International Journal of Eating Disorders. In 2016, The Novo Nordisk Foundation awarded a grant to Daniel Quintana for the project Investigating Oxytocin System Dysfunction as a Common Link between Metabolic and Psychiatric Symptoms in Severe Mental Illness.