1 INTRODUCTION

In 2000, the National Institutes of Health (NIH) created the registry and results database Clinicaltrials.gov. The FDA Amendments Act of 2007 mandates registration and submission of randomized controlled trials (RCTs) to Clinicaltrials.gov (Zarin et al., 2015). One aim of RCT registration is to minimize selective reporting of clinical trial outcomes (Chan et al., 2017; Roest et al., 2015; Turner, 2013). Selective reporting impairs the valid interpretation of trials and leads to bias with regards to the clinical evidence (Ioannidis et al., 2017). In the field of psychiatry, selective reporting was demonstrated by our group for antipsychotics in schizophrenia and schizoaffective disorders (Lancee et al., 2017) and by others for antidepressants in anxiety disorders (Roest et al., 2015). However, a systematic analysis of selective reporting in trials of schizophrenia spectrum, bipolar and depressive disorders is currently lacking. These specific disorders were chosen as they represent a large group of well-defined psychiatric disorders with specific pharmacotherapy treatments consisting of antipsychotics, antidepressants and mood stabilizers. Our primary objective was to investigate factors associated with selective outcome reporting in the selected disorders. Our secondary objective was to examine whether selective outcome reporting is associated with the directionality of primary outcomes. We performed systematic analyses targeting primary and secondary outcomes to ultimately provide a starting point for possible solutions.

2 METHODS

We performed a search on 31 August 2019 on Clinicaltrials.gov using the key words “schizophrenia OR schizoaffective disorder AND antipsychotics”, “antidepressant or mood stabilizer and depressive disorder or bipolar disorder”. Studies registered from 1 January 2006 to 31 December 2015 were included. This registration date was chosen in light of the mandated WHO registration of clinical trials since 2006. Inclusion criteria for registry records were: study phase II–IV; completed randomized controlled trials; trials including either major depressive disorder, bipolar disorder, schizophrenia or schizoaffective disorder; and FDA-approved medication given as investigational compound for that diagnosis. Both prospectively and retrospectively registered records were included. Trials were excluded when registry records had no matching publication on Pubmed, Embase or CIHNAL before 31 August 2019 (retrieved through NCT number), or when they did not meet inclusion criteria (i.e., other study designs, interventions or diagnoses).

All publications matching the NCT number were retrieved. If more than one publication for a certain registry record was found, we included the publication that in our opinion most accurately represented the purpose of the study mentioned on Clinicaltrials.gov. Data extraction procedures were according to our previous publication (Lancee et al., 2017).

Primary and secondary outcomes were compared between registry records and publications. Outcomes were considered primary outcome measures when they were stated as such in the publication, or when they were the main focus of the article corresponding with the clearly defined primary outcome measures on Clinicaltrials.gov. We extracted the variables listed in Table 1 for all retrieved publications from clearly defined sections on Clinicaltrials.gov. Selective outcome reporting was defined by two separate authors as adding, changing or deleting of an outcome measure. Discrepancies were resolved by discussions with the other co-authors. To allow for conservative estimations of outcome reporting bias, slightly different phrasing of similar outcome variables between records and publications was allowed. Published results of primary outcomes were rated negative (supporting the null hypothesis) or positive (rejecting the null hypothesis). Prior to statistical analyses, we published our study protocol at the Open Science Framework (https://osf.io/g9mpd/?view_only=525197d231104725a15fc74253bea50b).

TABLE 1. Study characteristics associated with selective outcome reporting in primary outcomes Variables Number of trials (N = 151), percentage of total Selective outcome reporting inprimary outcomes (N = 21 trials) Adjusted odds ratio (AOR), (95% CI) p-value Disorder Bipolar 38 (25%) 1 1 (Reference) MDD 51 (34%) 9 12.68 (1.40–115.2) 0.024 Schizophrenia 62 (41%) 11 14.53 (1.65–127.8) 0.016 Funding Industry 115 (76%) 12 1 (Reference) NIH 4 (3%) 0 0.00 (NA) 0.993 Other 32 (21%) 9 5.26 (1.39–19.77) 0.014 Centers Mono 29 (19%) 6 1 (Reference) Multi 122 (81%) 15 0.99 (0.22–4.72) 0.975 Study start yeara,b NA NA 0.99 (0.84–1.18) 0.978 Anticipated enrollmenta,c NA NA 0.99 (0.98–1.00) 0.657 Note: Significant results are shown in bold. Abbreviations: MDD, major depressive disorder; NA, not applicable; NIH, National Institutes of Health. a Coded as a numerical variable; the odds ratio shows how much the likelihood of selective outcome reporting changes with a one-unit increase in the numerical variable. b Study start year: Mean 2008, median 2008, range 1997–2014. c Anticipated Enrollment: Mean 602, Median 401, Range 20–18,239.

R, version 3.4.4, was used for the statistical analyses. According to previously established methods (Chan et al., 2017), multivariate logistic regression was performed with selective reporting as outcome, and funding source (categorized on ClinicalTrials.gov as industry, NIH or ‘other’, e.g., academia or unfunded), psychiatric disorder, year of study start date, participating centers, and anticipated sample size as explanatory variables, after testing for multicollinearity. Adjusted odds ratios (AOR) were calculated. Study start year and anticipated enrollment were included as numerical values, while all other variables were included as categorical values. Due to the small sample size, two-sided Fisher exact test was used to compare the proportions of newly added positive primary outcomes with the proportions of positive results in the overall group of primary outcomes.

A post-hoc multivariate logistic regression was performed excluding studies with an anticipated enrollment above three standard deviations from the median, to identify whether trials with large sample sizes influenced the results. This was done post-hoc as we had not anticipated large variation in study population sizes. Following our unanticipated finding of less selective outcome reporting in industry-funded RCTs, we used two-sided Fisher exact test for a second and final post-hoc test to compare changing of primary outcome measures in registry records between industry-funded trials and non-industry funded trials.

3 RESULTS

A total of 151 trials (with N = 94,303 participants in total) were included (Figure 1).

Flowchart of trial-selection. *Terminated is defined on Clinicaltrials.gov as: “Study halted prematurely and will not resume; participants are no longer being examined or receiving intervention”

Of these, 21 (14%) showed irregularities between registered and published primary outcomes. Of the 22 added primary outcomes, 14 outcomes (64%) were ‘positive’, which was not different from the percentage of positive results in the prespecified group of primary outcomes (75%, p = 0.30). For secondary outcomes, irregularities between registration and reporting were found in 126 (83%) of included trials. 19 trials (13%) provided published outcomes entirely in line with primary and secondary outcomes as specified in Clinical Trials.gov.

Higher odds of irregularities in primary outcome measures were associated with non-industry-funded RCTs (AOR 5.3; p = 0.014) and trials investigating major depressive disorder (AOR 12.7; p = 0.024) or schizophrenia (AOR 14.5 and p = 0.016; Table 1). Study start year, number of participating centers and anticipated enrollment were not associated with selective outcome reporting (Table 1).

A post-hoc analysis showed outcomes in registry records from industry-funded trials were not changed more often than non-industry funded trials (OR 0.59, p = 0.35), making it unlikely that updating explains high registry adherence by industry. Another post-hoc analysis excluding studies with a large enrollment (anticipated enrollment >3 SD above the median, N = 3 trials) did not change the results.

4 DISCUSSION

We demonstrate discrepancies between trial registrations and publications in primary (14% of the RCTs) and secondary outcomes (83%) across RCTs investigating debilitating psychiatric disorders, especially in non-industry funded RCTs. An explanation for the high rates in secondary outcomes is that they are less well conceived than the primary outcome before trial registration and therefore more prone to editing. Results for selective outcome reporting in primary and secondary outcomes are comparable to previous studies (Chan et al., 2017; Ioannidis et al., 2017; Lancee et al., 2017; Roest et al., 2015). Trials in bipolar disorder had a very low degree of selective outcome reporting relative to trials in psychotic and depressive disorders, for which we have no explanation.

Strengths of this study are the systematic assessment of both primary and secondary outcomes and adding different explanatory variables to our multivariate logistic regression. All data were separately reviewed by two investigators and discrepancies were resolved by discussions with the other co-authors, minimizing the risk of inter-observer variation impacting study findings.

Limitations of this study are its relatively small number of included trials, potentially limiting the statistical power needed to compute the multivariate logistic regression. Furthermore, three large psychiatric disorders were included, which may have resulted in selection bias and thereby making these results potentially not applicable to all RCTs investigating psychopharmacotherapy.

Although we included only studies on three disorders, our results point to widespread selective reporting across prevalent and debilitating psychiatric disorders. Our findings warrant further studies examining how selective reporting can be minimized, such as quality control procedures for peer reviewers and journal editors (Ioannidis et al., 2017). Increased insight into the nature of selective reporting in psychiatry and ways to tackle it will enhance the reliability of clinical trial results and optimize patient care.

CONFLICT OF INTERESTS

None.

Data are publicly available from the references listed.

REFERENCES

Chan, A. W., Pello, A., Kitchen, J., Axentiev, A., Virtanen, J. I., Liu, A., & Hemminki, E. (2017). Association of trial registration with reporting of primary outcomes in protocols and publications. Journal of the American Medical Association, 318(17), 1709– 1711. Ioannidis, J. P., Caplan, A. L., & Dal-Ré, R. (2017). Outcome reporting bias in clinical trials: Why monitoring matters. British Medical Journal, 356, j408. Lancee, M., Lemmens, C. M. C., Kahn, R. S., Vinkers, C. H., & Luykx, J. J. (2017). Outcome reporting bias in randomized-controlled trials investigating antipsychotic drugs. Translational Psychiatry, 7(9), e1232. Roest, A. M., de Jonge, P., Williams, C. D., de Vries, Y. A., Schoevers, R. A., & Turner, E. H. (2015). Reporting bias in clinical trials investigating the efficacy of second-generation antidepressants in the treatment of anxiety disorders: A report of 2 meta-analyses. Journal of the American Medical Association Psychiatry, 72(5), 500– 510. Turner, E. H. (2013). Publication bias, with a focus on psychiatry: Causes and solutions. CNS Drugs, 27(6), 457– 468. Zarin, D. A., Tse, T., & Sheehan, J. (2015). The proposed rule for U.S. clinical trial registration and results submission. New England Journal of Medicine, 372(2), 174– 180.