Background and Hypothesis Machine Learning (ML) models have been argued to reliably predict diagnosis and symptoms of schizophrenia based on voice data only. However, it is unclear to what extent such ML markers would generalize to different clinical samples and different languages, a crucial assessment to move towards clinical applicability. In this study, we systematically assessed the generalizability of ML models of vocal markers of schizophrenia across contexts and languages.

Study Design We trained models relying on a large cross-linguistic dataset (Danish, German, Chinese) of 217 patients with schizophrenia and 221 controls, and used a conservative pipeline to minimize overfitting. We tested the models’ generalizability on: (i) new participants, speaking the same language; (ii) new participants, speaking a different language; (iii) further, we assessed whether training on data with multiple languages would improve generalizability using Mixture of Expert (MoE) and multilingual models.

Results Model performance was comparable to state-of-the-art findings (F1-score ∼ 0.75) within the same language; however, models did not generalize well - showing a substantial decrease - when tested on new languages. The performance of MoE and multilingual models was also generally low (F1-score ∼ 0.50).

Conclusions Overall, the cross-linguistic generalizability of vocal markers of schizophrenia is limited. We argue that more emphasis should be placed on collecting large open cross- linguistic datasets to systematically test the generalizability of voice-based ML models, and on identifying more precise mechanisms of how the clinical features of schizophrenia are expressed in language and voice, and how different languages vary in that expression.

Riccardo Fusaroli has been a paid consultant on related but not overlapping topics for Roche. The other authors have no real or potential conflicts of interest that could have influenced the research.

A.P. was supported by a Marie Skłodowska-Curie Actions - H2020-MSCA-IF-2018 grant (ID: 832518, Project: MOVES). K.K. has been supported by Japan Society for the Promotion of Science (JSPS) (PE 07550). The study was supported by seed funding from the Interacting Minds Center (Aarhus University).

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

The participants received written and oral information about the project, and written informed consent was obtained before inclusion. The study was approved by The Central Denmark Region Committees on Biomedical Research Ethics (Ref: M-2009-0035; Ref: 2007-58-0010) and the Danish Data Protection Agency. The project complied with the Helsinki Declaration of 1975, as revised in 2008.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Yes