Patient selection

The Institutional Review Board of our institution approved this retrospective study and waived the requirement for obtaining informed consent from the patients. The case collection for this study was based on consecutive cases referred to the Department of Respiratory Surgery or Medicine as suspected lung cancer cases with indications for surgery between June 2020 and May 2022 and for which CT scans were performed under the preoperative lung tumor screening protocol at our hospital. The collection process was initiated by searching for diagnostic imaging reports using the name of the relevant protocol as the search term. The CXR data were collected from the nearest dates before and after the CT examination. The cohort of 399 participants in this study had been previously reported in a study that evaluated the performance of other deep learning-based automatic detection software [12].

Image acquisition

Chest radiographs were acquired using CALNEO HC (DR-ID900, Fujifilm Corporation, Tokyo, Japan), and the imaging parameters were unified (120 kVp, 160 mA, automatic exposure control, grid ratio of 12:1). CT images were acquired using one of the three types of multi-slice CT scanners available at our institution (Light Speed VCT64/Revolution CT, GE Healthcare, Milwaukee, WI, USA; Somatom Definition Flash; Siemens Healthineers, Erlangen, Germany). The scanning and reconstruction parameters of the CT scanners were as follows: voltage, 120 kVp; quality reference, 280 mAs or Noise Index, 9/11; rotation period, 0.4 or 0.5 s; detector collimation, 128 × 0.6 or 64 × 0.6; pitch, 0.508–1.0; and section thickness, 1.25 or 1.5 mm.

AI software information

The commercially available AI-based software CXR-AID (Fujifilm, Tokyo, Japan), which was approved by the PMDA in 2021, was used in this study. This software automatically detects abnormal lesions as colored overlays and generates a continuous probability index between 0 and 100 corresponding to the probability of nodules, consolidation, and pneumothorax on the chest radiograph. The results are displayed as a color-coded map corresponding to the generated probability index on the chest radiograph. As CXR-AID does not classify abnormal lesions, all detected lesions were included as targets in this study. The maximum probability index of the identified lung-lesion candidates was defined as the probability index of the target.

Image evaluationReference standard and performance of AI software

The reference data for the lesions created by a radiologist were developed using the results analyzed in the previous study. The detailed process is as follows [12]. Two radiologists (18 and 9 years of experience) retrospectively reviewed the chest radiographs and corresponding CT images and annotated pulmonary nodules on the chest radiographs with bounding boxes without referring to the results of CXR-AID. The lesion was not annotated if the presence of an abnormal lesion identified on CT could not be confirmed on the chest radiograph. The bounding boxes were annotated after reaching a consensus. We performed annotation and visibility score assessment on the entire lesion for lung cancer showing pneumonia-like findings or lung cancer accompanied by secondary changes in the surrounding areas. In cases with more than four nodules, the top three nodules were selected based on size and visibility scores, as described below. The boundaries of lesion recognition by CXR-AID were identifiable through the extraction of color pixels. In our study, there were no cases in which two nodules were close to each other, as assessed visually by a radiologist. If the center of the final bounding box annotated by the radiologists was within the area segmented by CXR-AID, the area was considered as successfully detecting the nodule, and the lesions identified by CXR-AID were designated as target areas (“true positives” in this study). All other areas identified by CXR-AID were defined as non-target areas (false positives). The probability indices of the target and non-target areas were compared. The Dice similarity coefficient (DSC) and intersection over union (IoU) were calculated to evaluate the extent of interobserver variability in manual segmentation among the radiologists.

Nodule evaluation on chest radiographs

Information on nodule characteristics and the background lung condition was obtained from data from a previous study [12]. The following nodule characteristics were evaluated by the two radiologists: nodule type (solid and subsolid), nodule location (craniocaudal and transaxial), and the presence of overlapping/masking structures (clavicle/first rib, hilar vessels, heart, and diaphragm). Nodule visibility (visibility score) was rated on a 4-point scale with reference to the report by Jang et al. [10], with each score indicating the following: 1, very subtle; 2, subtle; 3, moderately visible; and 4, distinctly visible. The background lung status (background lung score) was graded on a 4-point scale with reference to the modified anatomical noise described by De Boo et al. [13], with each score indicating the following: 1, none; 2, mild; 3, moderate; and 4, severe. The visibility and background lung scores for the initial 30 patients were reviewed concurrently by both radiologists, whereas the remaining patients were reviewed independently. In cases of disagreement, the scores were determined by reaching a consensus. One radiologist measured the size of the solid region of the nodule on the CT image. Lung abnormalities like atelectasis, scarring, bronchiolitis, fibrosis, or emphysema were confirmed on the CT image through consensus between the two radiologists. The definitions of the nodule locations, scores, and findings are described in Appendix S1. In cases where surgical intervention was performed, pathology results of the nodules were obtained from the hospital information system.

Analysis of the non-target areas

For images with non-target areas, one radiologist (6 years of experience) referred to the CT image to determine the probable cause of detection by CXR-AID. The non-target areas were further classified into two categories: non-target normal areas, where normal structures (pulmonary vessels, bone/cartilage, and hilar structures) were misidentified by CXR-AID; and non-target abnormal areas, where non-neoplastic abnormal findings (scarring, pleural thickening/plaque, fibrosis, and emphysema/bra) were identified by CXR-AID.

Statistical analysis

Sensitivity analyses were performed on a per-lesion basis. The number of non-target areas and non-target normal areas per chest radiograph was calculated as the total non-target areas or non-target normal areas divided by the number of chest radiographs, respectively. Statistical analyses were performed using R software (version 4.2.1; R Project for Statistical Computing). Nominal variables were compared using the chi-squared test or Fisher’s exact test. Continuous variables, pertaining to patient characteristics and pathological data, were compared using Welch’s two-sample t-test. The Cochran–Armitage trend test was utilized to compare the sensitivity and visibility scores, the sensitivity and background lung scores, as well as the number of non-target areas per image and the background lung score. Univariate logistic regression analysis was performed to identify the factors predictive of detected or undetected nodules and background lung disease. A p-value < 0.05 was considered statistically significant for all tests. The agreement between the two readers was calculated using non-weighted kappa statistics. The Κ-values were interpreted as follows: poor (κ < 0.20), fair (κ = 0.21–0.40), moderate (κ = 0.41–0.60), good (κ = 0.61–0.81), or excellent (κ = 0.81–1.00).