The study utilized the Central Malignant Melanoma Registry in Germany (CMMR) to retrospectively identify patients diagnosed with stage IV melanoma between 2015-01-01 and 2018-12-31 who were initially treated at the Department of Dermatology, University Hospital Tübingen which is a tertiary referral center for melanoma patients. The study protocol was approved by the institutional ethics board, and informed consent was waived due to the retrospective study design. 272 patients who met the inclusion criteria (stage IV melanoma, baseline (pre-treatment) contrast-enhanced CT, lymph node and/or soft tissue metastasis at baseline CT, available first follow- up CT after therapy initiation), were selected. CTs of these patients, with various types of soft tissue and lymph node metastases in baseline and follow-up CTs, were divided into a training and validation set for developing the pipeline [12], and a testing set for the present study. The metastases were radiologically identified by morphological criteria or their behavior under therapy in the follow-up CTs. The training, validation, and reference segmentations for the testing set were manually conducted by an experienced resident radiologist (FP 4 years) in consensus reading with two senior radiologists with extensive experience in oncologic imaging (AO 8 years and SG 9 years) using a custom-made reading software (SATORI; Fraunhofer MEVIS, Bremen). Figure 1 shows exemplary soft tissue metastases used in our study. In the appendix, Figure A1 shows the detailed study workflow, Table B1 shows the patient characteristics and Table B2 lists the scanner types and number of scans acquired with the respective scanner.

The training and validation set included 4308 lesions (2603 soft tissue and 1705 lymph node lesions) from 214 patients split into 3445 (2081 soft tissue and 1364 lymph node lesions) and 863 (522 soft tissue and 341 lymph node lesions) lesions for training and validation, respectively. The datasets included cases from different institutions and were therefore obtained with different CT scanners with various protocols. Typical CT imaging parameters used in our center for staging of melanoma patients are reported in the appendix Table C1.

The testing dataset included 126 soft tissue and 135 lymph node lesions from 58 patients referred for the first follow-up CT after therapy started that were not included in the training and validation dataset. We selected patients with lower lesion counts for the testing set to obtain a diverse set of lesions. For details refer to Table B1 and B2. The lesions were stratified by diameter size in the follow-up scan as smaller than 10 mm (n = 58), 10–20 mm (n = 94), and larger than 20 mm in diameter (n = 55), with a mean size of 17.9 mm ± 15.2 mm (range: 5.0–140.5 mm). 54 lesions showed complete response.

Readers viewed baseline and follow-up CTs simultaneously in a single window using the custom-made reading software. Manual segmentation involved outlining lesions on the follow-up CT images using a cursor with optional interpolation for neighboring slices. The AI-assisted workflow used automatically generated volumetric segmentations that were displayed in SATORI (see Figure G1 in appendix). The readers had the option to accept the automated segmentation as perfect, accept it as passable and make manual corrections, or dismiss it and perform manual segmentation. If the AI produced a segmentation for metastases showing a complete response in follow-up CTs, radiologists could reject the proposed structure and save an empty structure. A schematic representation of the proposed pipeline is presented in Fig. 2. Extensive technical details have been published in a previous publication [12] and are summarized in the appendix D. Three radiologists from two institutions independently segmented the testing set to assess inter-reader agreement variability. The radiologists were reader 1 (MM, specialist, Tübingen), reader 2 (HA, resident, Tübingen) and reader 3 (BG, physician, Bremen) with 7, 2, and 5 of experience in oncologic radiology, respectively.

Schema of the proposed pipeline for AI-assisted segmentation of lymph node and soft tissue metastases in follow-up CT scans. The AI-assisted segmentation pipeline includes four major components: 1.) Extraction of the region of interest (ROI) around the lesion in the baseline scan; 2.) Registration of the baseline to the follow-up scan; 3.) Propagation of the ROI to the follow-up scan to constrain the search region. Inference of the trained U-Net to segment all the lesions in the defined region; 4) Selection of the corresponding lesion from the output of the U-Net. This reader study focuses on the user-interaction on the follow-up image

The readers manually segmented the first 50% of the testing cohort, followed by AI-assisted segmentation of the second 50%. After two weeks, they performed AI-assisted segmentation of the first 50% and manual segmentation of the second 50% of the cohort to avoid recall bias. To prevent an artificial habituation effect, patients were sorted by ID and not by the number of metastases. The readers were blinded to their previous segmentation results, those of other readers, and the reference standard of the follow-up examinations. In the following, Mi and Ai denote the manual session and the assisted session of the reader i with i ∈ .