Remember me
Lung cancer is a well-known malignancy that causes severe respiratory symptoms. There were 1.8 million lung cancer–related fatalities and 2.2 million newly diagnosed patients worldwide in 2020 []. Patients with lung cancer frequently encounter complex health care challenges, such as unanticipated visits to the emergency department (ED) due to disease progression, treatment-related problems, and comorbidities []. Despite advances in medical technology, which have increased the survival rates of patients with lung cancer, ongoing management is still needed after initial oncology treatment due to the diverse characteristics and causes of the disease and the fact that each patient’s disease stage and conditions vary [].
Patients with lung cancer often experience acute complications or disease progression that may require urgent medical attention []. The frequency of ED visits among patients with lung cancer increases with the length of their survival []. Prior research has shown that approximately 10% of all cancer-related ED visits are attributable to lung cancer []. In addition, compared to patients with other types of cancer, those with lung cancer who visit the ED tend to have a worse prognosis []. Shin et al [] examined the 28-day mortality rate among intubated patients with cancer in the ED. Their findings revealed that patients with lung cancer faced a higher mortality risk compared to those with other types of cancer. Another previous study found that patients with lung cancer had the highest mortality rate (48.1%) within 28 days in contrast to those with other cancers who presented to the ED with septic shock []. Further, a comprehensive study conducted on hospitalizations related to sepsis incidence and mortality rates among patients with cancer in the United States revealed that patients with lung cancer had the highest mortality rate []. These findings from previous studies demonstrate the poor prognosis for patients with severe lung cancer–related conditions in the ED.
To lower the risk of a poor prognosis, it is of the utmost importance to predict visits to the ED among patients with lung cancer in advance []. Nevertheless, the existing literature on this subject is limited, with only a handful of studies identifying the risk factors associated with these visits. Consequently, the lack of comprehensive research in this field hinders clinicians’ ability to intervene in a timely manner. Hong et al [] developed a machine learning–based model for predicting ED visits in patients who were undergoing radiotherapy or chemoradiotherapy. Sutradhar et al [] proposed a risk prediction approach for ED visits among patients with cancer using the Edmonton symptom assessment system and conventional statistical analysis and logistic regression methods []. Sutradhar and Barbera [] also developed a machine learning model to predict 7-day ED visits in patients with cancer using the Edmonton symptom assessment system and other clinical information. Previous studies were predominantly conducted among patients with all types of cancer, so they did not identify risk factors specific to those with lung cancer for ED visits.
Therefore, the primary objective of this study was to identify the influential factors for patients with lung cancer that may impact ED visits. This study developed a machine learning–based prediction model to forecast ED visits among patients with lung cancer using data from the Observational Medical Outcomes Partnership (OMOP) common data model (CDM) []. Additionally, the significance of various types of clinical information in influencing the decision-making process of the machine learning model was evaluated. The ability to predict ED visits enables health care service providers to identify high-risk patients in advance, allowing for prompt intervention and appropriate management. By intervening expeditiously, clinicians may be able to prevent or mitigate emergency situations, leading to improved patient outcomes. This study contributes to minimizing preventable health deterioration by facilitating early intervention during lung cancer treatment by clinicians.
This was a retrospective observational study using electronic health records at Seoul National University Bundang Hospital (SNUBH), a tertiary general hospital in South Korea. The electronic health record data were converted to the OMOP CDM, which is a standardized data format in the observational health data sciences and informatics community. The data set comprised information from visit histories encompassing a broad range of categories, such as diagnoses, medication prescriptions, laboratory test results, performed procedures, and clinical observations.
Ethical ConsiderationsThis study was conducted with approval and waivers of informed consent or exemptions from the SNUBH institutional review board (no. X-2308-844-903). The CDM data used in this study were deidentified and are securely maintained within an internal network.
Target PopulationPatients who were 18 years of age or older and diagnosed with lung cancer (International Classification of Diseases, Tenth Revision code C34: malignant neoplasm of bronchus and lung []) at least once between 2010 and 2017 were eligible. presents the inclusion and exclusion criteria for the study participants. This study included all outpatient visits within 5 years of the patient’s initial diagnosis of lung cancer. Moreover, since the focus of this study was on health information routinely collected in hospitals based on the CDM, target populations should visit the hospital regularly for disease management. Considering the follow-up period in the treatment guidelines for patients with lung cancer, patients who were unable to follow up for more than 6 months during the study period were excluded.
Figure 1. Flowchart of the study participant selection process. SNUBH: Seoul National University Bundang Hospital. Primary Outcomepresents an overview of the observation and prediction windows for the risk prediction task. The outpatient visits were defined as the index date. Multiple outpatient visits for the same patient were considered independent index dates. During the 30 days prior to the index date, predictors were extracted and aggregated to be used as input variables for the machine learning models. The primary outcome was the occurrence of ED visits within the 30-day period following the index dates. Since the objective of this study was to predict preventable ED visits that occurred during disease progression or treatment among patients with lung cancer, the primary outcome was defined as ED visits that satisfied specific criteria. Valid outcomes were restricted to ED visits that did not result in a transfer to another hospital.
Figure 2. An overview of the observation and prediction windows for the risk prediction task. ED: emergency department; ML: machine learning. PredictorsData extracted from the CDM over successive time windows, referred to as the observation window, for 30 days prior to the index dates were used for candidate predictors. This included patient demographics, visit histories, clinical information, laboratory results, and vital signs. If patients had multiple records for the same data item during the observation window, the median value was calculated. Detailed information on the selected features and their data sources is presented in .
The selected features used to predict the primary outcome were analyzed using statistical methods. This study provided descriptive statistics for continuous variables in the form of median and IQR values, whereas categorical variables were presented as frequencies and respective percentages. The P value of each variable was also calculated to explore the probability of a relationship between the selected features for prediction tasks. In the significance test, the Mann-Whitney U test was used when analyzing continuous variables, while the chi-square test was used for categorical data.
Model Training and EvaluationThis study defined the prediction task for each event date as a binary classification problem. We used 4 different machine learning models—logistic regression, random forest, extreme gradient boosting, and light gradient boosting machine (LGBM)—to discover the model with the highest performance. These models were selected to provide a comparison of models by evaluating the performances of various alternatives, ranging from conventional linear-based approaches to more complex methods, such as ensemble-based models.
The selected features were preprocessed for use as input variables in the machine learning models. Missing values were replaced with their median value, and aberrant observations that were not acceptable from a theoretical perspective were removed based on prior research and the expertise of clinical domain experts to prevent extreme outliers from leading to failure in the prediction task. The entire data set was split into training and testing sets using repeated 7-fold cross-validation, with stratified sampling accounting for the incidence of the primary outcome. Demographic information of the patients, including age, gender, and comorbidities, was also taken into account given that some patients had made multiple outpatient visits, which corresponded to the index dates in this study. By using repeated k-fold cross-validation, the estimated performance of a machine learning model can be enhanced. The cross-validation procedure was iterated 1000 times. Finally, categorical variables were encoded, and continuous variables were normalized.
The following 4 performance metrics were used for model evaluation: area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (PRAUC), sensitivity, and specificity. The evaluation results were reported as the average value encompassing all folds from all iterations, and the 95% CI was estimated. This approach ensures a more accurate estimate of the true unknown underlying mean performance of the model on the data set than using the standard error. Using the Shapley additive explanations (SHAP) value from the best performing model, the significance of the features was also analyzed to identify risk factors for ED visits by patients with lung cancer []. All the experiments were performed using the Python 3.7.6 environment (Python Software Foundation).
A total of 222,127 outpatient visits occurred for 5,000 patients. presents the baseline characteristics of the target population. The median age was 67.23 (IQR 60.54-75.57), and 3812 (76.24%) patients were older than 60 years. Men accounted for 64.14% (n=3322) of the participants, which was slightly greater than the number of women (n=1678). According to the Charlson comorbidity index (CCI) [], more than 66.4% (n=3320) of participants did not have any comorbidities before being diagnosed with lung cancer. There were 1212 (24.24%) patients who had a history of smoking, while the others had no smoking history or their history was unknown. About half (n=2827, 56.54%) of the patients lived in the metropolitan area near the hospital.
Table 1. Baseline characteristics of the selected participants (n=5000).CharacteristicValueAge (years), median (IQR)67.23 (60.54-75.57)Age group (years), n (%)aOther areas of residence included Gangwon-do, Chungcheongbuk-do, Chungcheongnam-do, Gyeongsangbuk-do, Gyeongsangnam-do, Jeollabuk-do, Jeollanam-do, Jeju-do, Daejeon, Sejong-si, Daegu, Ulsan, Busan, and Gwangju.
There were 8192 visits to the ED, and 2790 patients (55.80% of total patients) had ED visits during the course of their disease. The results of an explanatory data analysis of ED visits by the target population are displayed in . Of all the visits, 81.33% (n=6663) were from home, while the rest were from outpatient visits or other institutions, including transfers from hospitals, independent clinics, and inpatient care units. Most ED visits occurred between 7 AM and 10 PM. The median length of time spent in the ED was approximately 13.64 (IQR 2.98-15.84) hours. More than half (n=4956) of the patients were discharged home, while the remaining patients were hospitalized or died in hospital. Of all ED visits, 38.32% (n=3139) resulted in hospitalization, and 1.18% (n=97) resulted in death. The most frequent causes of visits to the ED were neoplasms, including malignant neoplasms of the bronchus and lung and secondary malignant neoplasms of other sites. The other primary causes were diseases of the respiratory system. Patients with symptoms, signs, and abnormal clinical and laboratory findings not elsewhere classified primarily visited the ED for the following reasons: fever of other and unknown origin, hemorrhage from respiratory passages, abnormalities of breathing, pain in the throat and chest, and abdominal and pelvic pain.
Table 2. Explanatory data analysis results of emergency department (ED) visits (n=8192) by selected patients.FeatureValueAge (years), median (IQR)67.10 (60.26-75.50)Gender, n (%)aICD-10: International Classification of Diseases, Tenth Revision.
The complete list of descriptive statistics (N=222,127) of the selected features used as input variables for the predictive model are displayed in . The median value of elapsed days since the first diagnosis of lung cancer was about 333 (IQR 103.60-810.42) days. Chemotherapy was administered at 39.27% (n=87,221) of visits, compared to 16.47% (n=36,575) for radiation therapy and 2.32% (n=5159) for lung cancer–related surgery. Analgesics were administered at 27.35% (n=60,744) of visits, and the use of antibacterials for systemic use accounted for 18.83% (n=41,825) of visits. The results of blood tests revealed a median value of 6.27 (IQR 4.87-8.06) for leukocytes, 237.00 (IQR 188.00-296.00) for platelets, 61.90 (IQR 53.50-70.05) for neutrophils, and 12.10 (IQR 10.80-13.30) for hemoglobin. The shock index, which refers to the ratio of the heart rate to systolic blood pressure, was calculated from the collected vital signs and showed a median value of 0.67 (IQR 0.59-0.78).
Performance Evaluation Results of the Machine Learning ModelsThe performance evaluation results from the machine learning models are presented in . The overall AUROC score was ≥0.70 in all models, ranging from 0.70 to 0.73. The optimal prediction threshold was defined using the precision-recall curve since the data set had an imbalanced class distribution. The highest PRAUC score was 0.24 in the LGBM model.
Table 3. Performance comparison of the machine learning models.ModelSensitivity (95% CI)Specificity (95% CI)AUROCa (95% CI)PRAUCb (95% CI)LRc0.76 (0.7554-0.7581)0.54 (0.5369-0.5398)0.71 (0.7054-0.7065)0.22 (0.2191-0.2205)RFd0.74 (0.7360-0.7395)0.57 (0.5634-0.5667)0.71 (0.7086-0.7097)0.22 (0.2205-0.2221)XGBe0.77 (0.7729-0.7759)0.51 (0.5080-0.5112)0.70 (0.7022-0.7033)0.21 (0.2134-0.2145)LGBMf0.76 (0.7575-0.7605)0.58 (0.5752-0.5780)0.73 (0.7312-0.7323)0.24 (0.2360-0.2374)aAUROC: area under the receiver operating characteristic curve.
bPRAUC: area under the precision-recall curve.
cLR: logistic regression.
dRF: random forest.
eXGB: extreme gradient boosting.
fLGBM: light gradient boosting machine.
The SHAP values for the highest-performing model, the LGBM, are depicted in . The key influencing features for predicting the risk of ED visits in the LGBM model were identified as recent ED visits, elapsed days since the initial diagnosis for lung cancer, the use of analgesics, and lymphocyte and albumin levels. The SHAP value quantifies the influence of a specific feature on the predictions; consequently, a comprehensive interpretation can be obtained by calculating the SHAP values for the model’s output. As shown in , the number of ED visits, CCI, administration of analgesics, and radiotherapy, as well as high values of leukocytes, alkaline phosphatase, monocytes, and neutrophils, increased the possibility of visits to the ED. In contrast, the combined presence of low values of lymphocytes, albumin, hemoglobin, and hematocrit indicated an increased likelihood of visiting the ED.
Figure 3. Summary plots for SHAP values in the light gradient boosting machine model. ALP: alkaline phosphatase; CCI: Charlson comorbidity index; ED: emergency department; Hb: hemoglobin; Hct: hematocrit; HR: heart rate; PLT: platelets; SHAP: Shapley additive explanations. This study developed a predictive model for ED visits among patients with lung cancer using machine learning and CDM data. Predicting ED visits among patients with lung cancer has numerous important implications. First, the ED visit prediction model is able to facilitate care coordination and patient management. As shown in , over 38% of all visits resulted in hospitalization. These results were marginally lower than the findings in prior literature, which indicated that 54.5% of ED visits resulted in hospitalization or death []. This may be due to the SNUBH-specific aspect that patients with cancer are treated via outpatient visits, and patients who were normally discharged might return to other inpatient care units or independent clinics after emergency treatments. If patients at risk could be identified during a previous scheduled session, many of these visits might be avoided. Moreover, it would be possible to perform elective procedures to manage patients at imminent risk of requiring ED visits. There have been several prior studies on predicting ED visits [-]; however, to our knowledge, there have been no studies predicting ED visits applicable to outpatient visits by patients with lung cancer. In addition, previous studies mainly focused on patient-reported outcomes, whereas this study mainly used clinical information, such as laboratory test results, as input variables for the prediction model. The clinical information–based machine learning model enhances the clinical use of the predictive model since patients with lung cancer continuously visit the hospital during their disease or after treatment for follow-up [].
The identification of risk factors associated with ED visits among patients with lung cancer is another significant finding of this study. The risk factors were identified by analyzing the importance of features that influenced the decision-making of machine learning models, as shown in , and the identified variables had a tendency to match theoretical knowledge found in the medical domain. Prior ED use was known to be a significant predictor of future ED visits for all types of patients with cancer []. Patients with cancer who had recently visited the ED more frequently may experience more severe symptoms, complications, or comorbidities requiring immediate medical care, thereby increasing their likelihood of future ED visits. The number of elapsed days since the initial lung cancer diagnosis was inversely proportional to the likelihood of visiting the ED. Given that cancer-related treatments, such as surgery, chemotherapy, and radiotherapy, are typically administered within a year of diagnosis, acute diseases and transient side effects of the cancer-related treatment can result in emergency situations []. Similarly, the use of analgesics indicates that patients were suffering from severe pain and distress, which may have prompted visits to the ED for pain management. Several clinical pieces of information, such as laboratory test results routinely collected during cancer treatment and follow-up monitoring, were demonstrated to be useful predictors of ED visits in patients with lung cancer. Lower lymphocyte counts may indicate decreased immune function or increased vulnerability to infections, and higher leukocyte, monocyte, and neutrophil counts may be associated with inflammation; both of these abnormal values may lead to visits to the ED. Lower levels of hemoglobin, hematocrit, and erythrocyte counts may be signs of anemia, which may cause fatigue and result in visits to the ED. Significant weight loss and lower albumin levels may be associated with disease progression or malnutrition, increasing the risk of complications that may require ED visits. Similarly, a high CCI indicated the presence of multiple comorbid conditions. Elevated alkaline phosphatase levels may indicate liver or bone involvement, and decreased platelet counts may indicate thrombocytopenia, necessitating evaluation and treatment in the ED. While radiotherapy is a treatment for lung cancer, it can cause side effects and complications, such as radiation pneumonitis, that may necessitate visits to the ED. Thus, close monitoring of laboratory test results in patients with lung cancer receiving treatments or routine follow-up is necessary to prevent ED visits.
This study has several limitations. First, this study was conducted at a single institution in South Korea, a tertiary-level general hospital. Thus, regional bias may have resulted in reduced generalizability. Nonetheless, this study demonstrated the possibility of conducting observational cancer research with CDM data. Developing a predictive model using data from a single institution may lead to biased results based on regional or institution-specific characteristics, making it difficult to generalize; therefore, external validation is essential for clinical applications. Due to the heterogeneity of the format for managing medical records across institutions as well as privacy concerns, it is difficult for researchers to share data; even a collaborative study with other institutions requires immense effort, time, and resources. Ahmadi et al [] reported that the OMOP CDM has the potential to facilitate international collaborative analyses, which is a crucial element of cancer precision medicine. Since our prediction model was developed using OMOP CDM data, it is expected that external validation will be possible for other institutions that have oncology CDM data through the observational health data sciences and informatics data network in the future. The data set used in this study is administered in accordance with the OMOP CDM, allowing different institutions to conduct collaborative research using a distributed research network.
Second, there were the inherent performance limitations imposed on the machine learning models. The primary outcome defined in this study was derived from all ED visits, only excluding the visits that resulted in patients being transferred to other institutions; therefore, there may have been non–cancer-related reasons for ED visits, such as fractures, injuries from traffic accidents, or wounds from animal attacks. In addition, a paucity of information may be one of the reasons for the results of unfavorable effects on the prediction model outcome. Staging or subtypes of lung cancer represent primary information for predicting the prognosis of patients; however, this information was not collected in our data source thus far. The acquisition of additional information could contribute to the enhancement of the predictive model’s performance; therefore, the establishment of extended lung cancer–specific data sets in the CDM is needed for more precise prediction. Nevertheless, the identified risk factors for ED visits could be used in the future as an avenue for the proactive management of patients with lung cancer during treatment.
Lastly, research bias should be taken into account when interpreting the results. This study imputed missing values with the median value during the data processing phase, which could potentially have introduced bias into the analysis. However, the characteristics of clinical data are such that the lack of even a solitary test result frequently results in a substantial amount of missing data among other relevant data, indicating the presence of a clear-cut pattern. In such situations, it was noted that median imputation demonstrated performance levels that were comparable to those of more complex algorithms. Furthermore, it offered the advantage of being readily executable, thereby enhancing its efficiency in terms of time and resources. Additionally, it is critical to acknowledge that while removing outliers is a standard procedure for enhancing data quality, extreme values may have clinical ramifications due to the unique attributes of ED visits among patients diagnosed with lung cancer.
In summary, this study developed a machine learning model to predict ED visits, with a specific focus on patients with lung cancer, and identified influential factors that exerted a significant impact on the model output. Continuous monitoring of the identified influencing clinical features will allow health care providers to efficiently allocate resources, ensuring that patients with a high likelihood of requiring preventive care receive prompt attention. By identifying patients at risk, health care service providers can initiate targeted interventions, such as closer monitoring, treatment plan adjustments, and timely referrals to the most appropriate specialists. The predictive model for ED visits by patients with lung cancer developed in this study not only enhances patient outcomes but also optimizes resource utilization, reducing strain on the ED and minimizing the burden on the medical staff. Ultimately, this proactive approach contributes to preventing or mitigating emergency situations, resulting in an improved quality of life for patients with lung cancer and possibly reducing the need for hospitalization or more invasive interventions.
This research was supported by a grant from the Korea Health Technology R&D Project through the Korea Health Industry Development Institute funded by the Ministry of Health & Welfare, Republic of Korea (grant HI22C0471).
AL and HP conceptualized the design of the study. SK and LS contributed to the data extraction. AL wrote the original draft. AL and HP performed the experiments and visualizations. AY and LS contributed to the data analysis and interpretation. SY revised the manuscript and supervised the research. All authors reviewed and approved the final version of the manuscript.
None declared.
Edited by G Eysenbach, C Lovis; submitted 24.09.23; peer-reviewed by Z Su, D Hu, S Matsuda; comments to author 11.10.23; revised version received 31.10.23; accepted 24.11.23; published 06.12.23
©Ah Ra Lee, Hojoon Park, Aram Yoo, Seok Kim, Leonard Sunwoo, Sooyoung Yoo. Originally published in JMIR Medical Informatics (https://medinform.jmir.org), 06.12.2023.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included.
Comments (0)