Introduction

Carbon monoxide (CO) poisoning is one of the most common causes of poisoning death and is responsible for approximately 50 000 emergency department (ED) visits per year in the USA [1]. In 2014 in the USA, 1319 deaths from accidental or intentional CO poisoning were reported, and a decline in annual cases has been seen since 1999 [2]. Worldwide, the estimated incidence of CO poisoning is 137 cases per million and 4.6 deaths per million [3]. However, the numbers of intoxication cases are most likely underestimated [4,5]. Carbon monoxide is an odorless and colorless gas, and the symptoms of CO poisoning are nonspecific: headache, dizziness, nausea, vomiting, fatigue, and loss of consciousness. The main sources of intoxication are fires, heating defects, and cars [6]. The diagnosis of CO poisoning requires a history of recent CO exposure, the presence of symptoms, and an elevated carboxyhemoglobin (COHb) level [7]. Either arterial or venous blood may be used to measure the COHb level [7–9]. Since 2005, noninvasive pulse CO oximeters have been available and approved by the Food and Drug Administration (FDA). They can quickly estimate COHb using readings at eight wavelengths of light at the fingertip (SpCO, CO saturation by pulse oximetry) [10,11]. As early as 2006, protocols were published on the use of noninvasive pulse CO oximeters for the triage of the victims of suspected CO poisoning directly on the scene by first responders [12]. Actually, CO oximeters are used daily by thousands of fire and emergency medical services around the world [13]. Nevertheless, the results of studies evaluating the accuracy of SpCO in assessing COHb were heterogeneous and brought into question its use in clinical practice [7,11]. Some experts do not recommend its use for the diagnosis of patients suspected of having CO poisoning in the ED [10].

The aim of this systematic review and meta-analysis was to evaluate the accuracy of SpCO (index test) to estimate COHb in any population.

Methods

This systematic review was conducted according to the Preferred Reporting Items for a Systematic Review and Meta-analysis-DTA 2018 guidelines [14] and has been registered on International Prospective Register of Systematic Reviews (PROSPERO, CRD42020177940).

Population

We included studies evaluating the accuracy and reliability of multiple wavelength pulse CO-oximeter (SpCO) measurements (index test) in human volunteers or ill patients, including children, compared to blood COHb levels (reference test). Studies of all designs published after 2000 were eligible, without language restrictions. Case reports were not included.

Search strategy, information sources, and study selection

Two authors (F.J. and C.L.) independently screened titles, abstracts, and full texts, following the inclusion criteria. The discrepancies (n = 6) were resolved after a discussion between the two authors (and if necessary, the intervention of a third author, M.P.; n = 0). We performed searches of the following databases on 2 August 2022: Medline (via Pubmed), Embase, and Cochrane Central Register of Controlled Trials (CENTRAL) on the Cochrane Library. We also searched the grey literature via OpenGrey. Our search algorithms for different databases included the following terms: (‘oximetry’, ‘pulse oximetry’, ‘CO oximeter’, ‘CO oximetry’, ‘pulse CO oximetry’, ‘noninvasive’) and (‘carboxyhemoglobin’, ‘COHb’). Quantitative data from the studies were independently extracted by two authors (M.P. and F.J.).

Quality assessment

The risk of bias was independently evaluated by two authors (M.P. and C.L.) using the Quality Assessment of Diagnostic Accuracy Studies-2 tool [15]. The assessed risks of bias were selection bias, studied test use, reference test use, and timing between the tests. Each of these items allowed the study to be classified into different risk categories (low, high, or unclear). In cases of discordance, the classification was discussed between the authors. All discrepancies were resolved by consensus (n = 2).

Study outcomes

The primary outcome was to estimate the diagnostic accuracy by modeling receiver operating characteristic (ROC) curves and measuring sensitivity (Se) and specificity (Sp) of SpCO for estimating COHb by blood sampling (primary measures). The secondary measures were to report the limits of agreement (LOA), the mean bias, and to perform subgroup analyses: according to the SpCO device used, and among subjects with suspected CO poisoning.

Data analysis

The analyses were performed by a biostatistician (B.L.) using R software version 4.0.3 (The R Foundation, Boston, Massachusetts, USA) (2020-10-10). The meta-analysis was performed using the mada package version 0.5.10 (mada: Meta-Analysis of Diagnostic Accuracy, https://cran.r-project.org/web/packages/mada), focusing on Se and Sp measures. The mean Se and Sp, and their 95% confidence regions were estimated using a bivariate linear mixed model that was proposed in 2005 by Reitsma et al [16]. In the absence of a definition of the threshold of positivity, it was set at ≥10% (SpCO and COHb) when individual data were available. Summary ROC curves (SROC) were estimated using three methods: Rutter and Gatsonis [17], Moses et al. [18], and Rücker and Schumacher [19]. LOA calculations were based on the method described by Tipton and Shuster [20]. The analysis of publication bias was performed using a Deeks’ funnel plot and its significance was tested using the associated test. This method is recommended for the evaluation of diagnostic studies [21].

Results Study selection

Of the 293 reports identified from databases and 6 from citation searching and websites, 21 studies were included in the systematic review and 11 in the meta-analysis for the primary measures (absence of exploitable quantitative data, n = 10), and 18 for the secondary measures (absence of exploitable quantitative data, n = 3) (Fig. 1).

Fig. 1:

Flow diagram of study selection. COHb, carboxyhemoglobin; SpCO, pulse carbon monoxide oximetry.

Study characteristics

Of the 21 articles included [22–42], 5 studies were experimental and 16 observational (Table 1). The experimental studies consisted of breathing CO in healthy volunteers and performing several measurements of SpCO and COHb. Among the observational studies, 10 were conducted in the ED and 2 in a hyperbaric center. Three different devices were used to measure SpCO: Rad-57 (Masimo, Irvine, California, USA) in 14 studies, Radical-7 (Masimo, Irvine) in 6 studies, and V-Spec Monitoring System (Senspec, Rostock, Germany) used once, in an experimental study.

Table 1 - Main characteristics of the studies included in the systematic review (n = 21) Study Design Clinical setting Number of subjects Number of observations Participant characteristics SpCO device (index test) COP SpCO threshold COHb measurement (reference test), sample size, and devices COP COHb threshold Funding and COI Inclusion in the meta-analysis for primary (SROC, Se, Sp) or secondary (LOA, mean bias) measures Barker et al. (2006) [22] Experimental NA 10 530 Healthy adults RAD-57 (Masimo) NA Arterial blood
ABL730 (Radiometer, Københav, Denmark) and OSM-3s (Radiometer) NA Masimo and Department of Anesthesiology, University of Arizona College of Medicine Secondary Bidstrup et al. (2021) [23] Observational Hyperbaric center 10 50 COP adults treated with HBO2 Radical-7 (Masimo) NA Arterial or venous blood
ABL800 FLEX (Radiometer) NA Loan of equipment (device, sensors, computer) by Masimo Secondary Caboot et al. (2012) [24] Observational Comprehensive sickle cell center 50 50 Children with sickle cell disease Radical-7 (Masimo) NA Arterial or venous blood
RapidLab 1265 (Siemens, Huntersville, North Carolina, USA) NA Supported by the National Heart, Lung, and Blood Institute, and the National Center for Research Resources Secondary Cardwell et al. (2012) [25] Observational Operating room 203 203 Preanesthetic children Radical-7 (Masimo) NA Venous blood
RapidLab C1265 (Siemens) NA Supported by NIH/NCRR Colorado CTSI
The pulse CO-oximeters were purchased by Children’s Hospital Colorado
The reusable finger sensors and SpCO software were loaned by Masimo Secondary Coulange et al. (2008) [26] Observational ED 12 12 Suspected COP nonsmoking patients RAD-57 (Masimo) ≥10% (individual data) Venous blood
IL 682 CO-oximeter (Instrumentation Laboratory, Barcelona, Spain) ≥10% (individual data) None Primary
Secondary Feiner et al. (2013) [27] Experimental NA 12 283 Healthy nonsmoking adults breathing CO gas to produce a target %COHb level of ≈10% to 12% Radical-7 (Masimo) ≥6.6% Arterial blood
ABL800 FLEX (Radiometer) ≥10% Masimo Primary
Secondary Kaya et al. (2018) [28] Observational ED 213 213 Suspected COP adults RAD-57 (Masimo) >23% Arterial or venous blood
ABL700 (Radiometer) >23% None Secondary Kot et al. (2008) [29] Observational Hyperbaric center 49 49 Suspected COP patients RAD-57 (Masimo) ≥20% Capillary blood
CO-xylite AVL 912 (AVL Biomedical Instrument, Basle, Switzerland) ≥20% None Primary
Secondary Koyuncu et al. (2020) [30] Observational ED 106 106 Suspected COP adults RAD-57 (Masimo) NA Arterial blood >5% for nonsmokers
>10% for smokers None Secondary Kulcke et al. (2016) [31] Experimental NA 6 888 Healthy nonsmoking adults V-Spec (Senspec) ≥8.9% Arterial blood
ABL90 FLEX (Radiometer) or OSM3, (Radiometer) ≥10% University of California, San Francisco, Department of Anesthesia and Perioperative Care. Oximeter and probes were supplied by Senspec Primary
Secondary Nagano et al. (2008) [32] Experimental NA 10 10 Healthy smoking adults RAD-57 (Masimo) NA Sample blood (not specified) NA None Absence of exploitable data for the meta-analysis Piatkowski et al. (2009) [33] Observational Burn center 20 20 Suspected COP patients RAD-57 (Masimo) ≥10% (individual data) Venous blood
ABL700 (Radiometer) ≥10% (individual data) Pulse CO-oximeter device was supplied by Masimo Primary
Secondary Roth et al. (2011) [34] Observational ED 1578 1578 All ED patients Radical-7 (Masimo) >6.6% Arterial or venous blood
ABL700 (Radiometer) >6.6% Masimo Primary
Secondary Ruppel et al. (2011) [35] Observational Pulmonary function laboratory 139 139 Patients referred for DLCO and arterial blood gas measurement. RAD-57 (Masimo) NA Arterial blood
RapidPoint 405 (Siemens) NA None Secondary Sebbane et al. (2013) [36] Observational ED 93 93 Suspected COP adults RAD-57 (Masimo) ≥9% Venous blood
IL 682 CO-oximeter (Instrumentation Laboratory, Milan, Italy) >5% for nonsmokers
>10% for smokers None Primary
Secondary Suner et al. (2008) [37] Observational ED 64 64 All ED patients ≥17 years old RAD-57 (Masimo) ≥9% for nonsmokers
≥13% for smokers Venous blood ≥9% for nonsmokers
≥13% for smokers Masimo loaned the devices and supported the data collection by providing funds for research assistants Primary Touger et al. (2010) [38] Observational ED 120 120 Suspected COP patients RAD-57 (Masimo) ≥15% Arterial or venous blood
RapidLab 1200 (Siemens) ≥15% Devices and training were supplied by Masimo Primary
Secondary Villalba et al. (2019) [39] Observational ED 126 126 Suspected COP patients with SpCO ≥10% Radical-7 (Masimo) ≥10% Venous blood
XN9000 hematology analyzer (Sysmex America, Lincolnshire, Illinois, USA) ≥10% Masimo funding for the technicians and the administrative costs; loaning of the SpCO sensors and equipment; and data safety monitoring
COI: the authors have disclosed a relationship with Masimo Primary
Secondary Weaver et al. (2013) [40] Observational ED 1363 1363 All ED patients RAD-57 (Masimo) >6% Arterial or venous blood
ABL825 (Radiometer) >6% Centers for Disease Control and SciMetrika. Masimo provided the oximeters and educated the research staff
COI: the authors have disclosed relationships with SciMetrika and Masimo Absence of exploitable data for the meta-analysis Zaouter et al. (2012) [41] Experimental NA 9 148 Healthy nonsmoking adults RAD-57 (Masimo) ≥10% Venous blood
ABL80 FLEX (Radiometer) ≥10% None Primary
Secondary Zorbalar et al. (2014) [42] Observational ED 38 38 Headache patients with SpCO ≥10% and ≥14 years old RAD-57 (Masimo) >10% Venous blood
GEM Premier 3000-GEM OPL CO-Oximeter (Instrumentation Laboratory) >10% Masimo loaned the devices and sensors Secondary

COHb, carboxyhemoglobin; COI, conflicts of interest; COP, carbon monoxide poisoning; DLCO, diffusing capacity of the lungs for carbon monoxide; ED, emergency department; HBO2, Hyperbaric hyperbaric oxygen; LOA, limits of agreement; NA, not applicable; ROC, receiver operating characteristic; Se, sensitivity; Sp, specificity; SpCO, pulse carbon monoxide oximetry; SROC, summary ROC curves.

Risk of bias and concerns

The patient selection was the principal source of bias and concerns regarding applicability (Table 2). Indeed, the index test was mostly well documented such as the reference one. Besides, the concerns regarding applicability were low because the threshold was predefined, and the biological result was not subject to interpretation. However, the patient selection was less clear in some of the studies. We found that in 33% of the studies, patient selection could have introduced a high risk of bias, and in 33%, the risk of bias induced by patient selection is doubtful. Obviously, this is leading to an increase in concerns regarding the applicability of the patient selection. To a lesser extent, the flow and timing were also unclear, with 28% of studies with a high risk of bias induced by the flow and timing design.

Table 2:

Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2)

Masimo was involved in the funding or loan of materials in 11 of 20 of the studies evaluating their devices (RAD-57 and Radical-7), and Senspec also provided materials for the study evaluating the V-Spec Monitoring System.

Outcomes of individual studies

The observed sensitivity values were very different between studies, ranging from 48% for Touger et al. [38] to 100% for Piatkowski et al. [33], Roth et al. in nonsmokers subgroup [34], and Sebbanne et al. in smokers subgroup [36]. The same finding was observed for specificity ranging from 51% for Feiner et al. when the COHb threshold was ≥5% [27] to 99% for Touger et al. (threshold ≥ 15%) [38]. The highest area under the ROC curve was 99% (Feiner et al.) when a COHb threshold ≥5% was used to define positive cases [27]. The LOA varied from ±3% for Kulcke et al. [31] to ±15% (calculated on individual data) for Zorbalar et al [42].

Primary measures

A total of 11 studies were included to estimate SROC and mean Se and Sp (10 studies without exploitable data for the primary measures). Overall, 2089 subjects and 3381 observations were analyzed (i.e. coupled measurement of SpCO and COHb). The mean Se and Sp were 0.77, 95% CI (0.66–0.85) and 0.83, 95% CI (0.74–0.89), respectively (Fig. 2). The area under the SROC curve, estimated using the Rutter and Gatsonis method, was equal to 86%. To achieve a Se of 95% the Sp was between 45 and 50%, depending on the method used. Conversely, for a Sp of 95%, the Se was between 42 and 55% (Fig. 3).

Fig. 2:

Sensitivities and specificities of the 11 included studies.

Fig. 3:

Summary ROC curves from the 11 included studies. ROC, receiver operating characteristic.

Although no publication bias was statistically found, with an estimated bias of −6.37 (not significantly different from zero, P = 0.22), there was an asymmetry on the funnel plot concerning only studies with small sample sizes (Fig. 4).

Fig. 4:

Funnel plot of the 11 included studies.

Secondary measures

A total of 18 studies were included to estimate the LOA (3 studies without exploitable data for the LOA). Overall, 2794 subjects and 4646 observations were analyzed. The mean bias was 0.75%, 95% CI (−6.26 to 7.75) and the LOA was −7.08% to 8.57%, 95% CI (−8.89 to 10.38) (Fig. 5).

Fig. 5:

Limits of agreement of the 18 included studies for the secondary measures.

When the device used was the Rad-57, the mean Se and Sp were 0.70, 95% CI (0.56–0.81) and 0.88, 95% CI (0.77–0.94) (Figure E1 in the online data supplement, Supplemental Digital Content 1, https://links.lww.com/EJEM/A377), respectively; and the LOA was −7.83% to 8.73%, 95% CI (−10.09 to 10.99) (mean bias, 0.45%; Figure E2 in the online data supplement, Supplemental Digital Content 1, https://links.lww.com/EJEM/A377). The mean Se and Sp were 0.74, 95% CI (0.59–0.85) and 0.79, 95% CI (0.73–0.85) when using the Radical-7 device (Figure E3 in the online data supplement, Supplemental Digital Content 1, https://links.lww.com/EJEM/A377), respectively; and the LOA was −5.89% to 8.45%, 95% CI (−9.74 to 12.30) (mean bias, 1.28%; Figure E4 in the online data supplement, Supplemental Digital Content 1, https://links.lww.com/EJEM/A377). Among patients with suspected CO poisoning the mean Se and Sp was 0.70, 95% CI (0.56–0.81) and 0.88, 95% CI (0.77–0.94) (Figure E5 in the online data supplement, Supplemental Digital Content 1, https://links.lww.com/EJEM/A377), respectively; and the LOA was −8.11% to 8.59%, 95% CI (−10.74 to 12.23) (mean bias, 0.74%; Figure E6 in the online data supplement, Supplemental Digital Content 1, https://links.lww.com/EJEM/A377).

Discussion

In this meta-analysis, we found that the SpCO level had a mean Se of 77% and a mean Sp of 83% to determine CO poisoning using COHb as reference. The LOA of SpCO level was ±8% (mean bias +1%). Some experts have suggested by consensus that acceptable LOA values in this context are ±5% [6,38]. Nevertheless, the FDA 510(k) accuracy specification of the RAD-57 is 3% (SD) corresponding to LOA of ±5.9% [13]. Therefore, the noninvasive measurement of COHb is clearly less accurate than advertised by the manufacturers. Indeed, here we found LOA of ±8.3% when the Rad-57 was used.

The use of CO oximeters for prehospital triage can save time for admission to the hyperbaric center and limit the number of hospital transfers [43]. In clinical practice, SpCO level has a major influence on the decision to transfer, or not, suspected CO poisoning victims to hospitals [44]. However, to achieve a safety rule-out the test used must have a high Se [45]. Here, we found a false-negative rate (1 – Se) of 23% with the SpCO measurement, considering the blood COHb level as reference. By selecting only studies with subjects with suspected CO poisoning, up to 30% of false-negatives were found. There were even negative SpCO levels (0%) while COHb was very high (>35%) [38]. The use of CO oximeters for the triage of potential CO poisoning victims is already discouraged by several experts because of their lack of accuracy, and the results from our meta-analysis also suggest this [6,10,11].

The CO oximeters are proposed as a screening tool in the ED in populations with nonspecific symptoms, to detect occult CO poisoning [10]. Among the three studies included in our systematic review evaluating this population (unselected cohorts in the ED), the false-positive rates were 9%, 23%, and 54% [34,37,40]. A study started in an urban ED on all admitted patients was prematurely stopped because all patients with positive SpCO (>8% in nonsmokers and >13% in smokers, n = 5) had negative blood COHb levels. Emergency staff became skeptical about the clinical usefulness of these devices and decided to stop this research [46]. In addition, an incidental finding of elevated COHb does not always correlate with CO poisoning [47]. Indeed, it can be due to endogenous production of CO, as found in hemolytic anemia [48]. The use of CO oximeters in an unselected population in the ED would likely be responsible for misdiagnoses as well as an increase in unnecessary COHb blood tests.

In our meta-analysis, the positive likelihood ratios (LR+) was 4.5 and the negative LR (LR−) was 0.3. Usually, LR+ >10 and LR− < 0.1 are considered to provide strong evidence to admit (rule-in) or exclude (rule-out) diagnoses, respectively [49,50]. With LR+ between 2 and 5 and LR− between 0.2 and 0.5, the contribution of SpCO in determining blood COHb level is considered low [51].

A possible alternative to CO oximeters is the measurement of exhaled CO, but the correlation with blood COHb level is uncertain and the measurement can be biased, especially in patients with severe airflow obstruction [52–55]. The correlation between capillary and venous COHb seems to be very high and could be assayed as part of triage using point-of-care analyzers [56].

Limitations

First, we included heterogeneous populations in our analysis, which may have introduced noise into our model. Nevertheless, the whole population is concerned by CO poisoning, and for this reason, we did not exclude healthy subjects, sick subjects, or children.

Second, the studies included in the meta-analysis did not use the same COHb thresholds to define CO poisoning (from >5% to >20%); sometimes without distinction between smokers and nonsmokers. The same is true for the SpCO thresholds used, which also varied from >5% to >20%. It is conventionally accepted that COHb values >3–4% in nonsmokers and >10% in smokers are considered pathological [8]. The use of these different thresholds has an influence on the calculated Se and Sp values. In addition, in most studies, SpCO measurements were performed in optimal settings, and subjects with nail pathologies or nail polish were excluded, which is a deviation from current practice and tended to overestimate the Se and Sp.

Third, we were unable to obtain the individual patient-level data from all the studies and create a global Bland-Altman plot. Furthermore, some studies could not be included in the quantitative analysis due to the lack of usable results. Indeed, diagnostic studies should clearly report the 2 × 2 diagnostic contingency matrix to be properly exploitable [57]. Therefore, there are only a limited number of studies included in this systematic review, and also a small number of subjects.

Conclusion

Noninvasive measurement of COHb (SpCO) using current pulse CO-oximeters does not seem to be highly accurate to estimate blood COHb. In clinical practice, it does not appear possible to replace them with blood tests either for safely ruling in or ruling out CO poisoning. Future studies are needed to assess whether SpCO measurement has an impact on outcomes or provides any benefit in the management of subjects with suspected CO poisoning.

Acknowledgements

The authors thank Dr Oussama Gasmi (Department of Paediatric Emergency Medicine, Nantes University Hospital, France) for his help in structuring the database.

F.J., C.L., and B.L. conceived and designed the study. F.J. and C.L. conducted the literature search and selected the studies. M.P. and C.L. analyzed the quality of the studies. M.P. and F.J. collected the data. B.L. analyzed the data, B.L. and F.J. interpreted the data. M.P. and F.J. wrote the first draft of the article, with all other authors making important critical revisions. All authors have read and approved the final version of the article.

Conflicts of interest

F.J. is a member of the editorial board of this journal. For the remaining authors, there are no conflicts of interest.

References 1. Hampson NB, Weaver LK. Carbon monoxide poisoning: a new incidence for an old disease. Undersea Hyperb Med 2007; 34:163–168. 2. Hampson NB. U.S. mortality due to carbon monoxide poisoning, 1999-2014. Accidental and intentional deaths. Ann Am Thorac Soc 2016; 13:1768–1774. 3. Mattiuzzi C, Lippi G. Worldwide epidemiology of carbon monoxide poisoning. Hum Exp Toxicol 2020; 39:387–392. 4. Barret L, Danel V, Faure J. Carbon monoxide poisoning, a diagnosis frequently overlooked. J Toxicol Clin Toxicol 1985; 23:309–313. 5. Hampson NB, Dunn SL; UHMCS/CDC CO Poisoning Surveillance Group. Symptoms of carbon monoxide poisoning do not correlate with the initial carboxyhemoglobin level. Undersea Hyperb Med 2012; 39:657–665. 6. Chiew AL, Buckley NA. Carbon monoxide poisoning in the 21st century. Crit Care 2014; 18:221. 7. Hampson NB,