Introduction: Studies of the users’ engagement with smoking cessation application (apps) can help understand how these apps are used by smokers, in order to improve their reach and efficacy. Objective: The present study aimed at identifying the best predictors of the users’ level of engagement with a smartphone app for smoking cessation and at examining the relationships between predictors and outcomes related to the users’ level of engagement with the app. Methods: A secondary analysis of data from a randomized trial testing the efficacy of the Stop-Tabac smartphone app was used. The experimental group used the “full” app and the control group used a “dressed down” app. The study included a baseline and 1-month and 6-month follow-up questionnaires. A total of 5,293 participants answered at least the baseline questionnaires; however, in the current study, only the 1,861 participants who answered at least the baseline and the 1-month follow-up questionnaire were included. Predictors were measured at baseline and after 1 month and outcomes after 6 months. Data were analyzed using machine learning algorithms. Results: The best predictors of the outcomes were, in decreasing order of importance, intention to stop smoking, dependence level, perceived helpfulness of the app, having quit smoking after 1 month, self-reported usage of the app after 1 month, belonging to the experimental group (vs. control group), age, and years of smoking. Most of these predictors were also significantly associated with the participants’ level of engagement with the app. Conclusions: This information can be used to further target the app to specific groups of users, to develop strategies to enroll more smokers, and to better adapt the app’s content to the users’ needs.

© 2023 The Author(s). Published by S. Karger AG, Basel

Introduction

Tobacco smoking is one of the most important causes of mortality globally [1]. Smartphone applications (apps) for smoking cessation are expensive to develop, but they can reach thousands of smokers at a low cost per participant [2, 3]. In two separate recent meta-analyses of randomized trials, only one smoking cessation app (the same one in both studies) increased smoking cessation rates, and the overall conclusions of both meta-analyses, based on limited evidence, were in favor of no effect [2, 4]. Possible conclusions are that either the content or quality of the included apps was insufficient, or that the apps were of good quality but smokers did not use them frequently enough or during a long enough time, or they did not follow the apps’ advice and recommendations, or that even intensive use of the best possible app and optimal adherence to recommendations are insufficient to treat tobacco dependence [2, 4–6].

The quality and content of smoking cessation apps vary widely, and few of them have been scientifically tested [3, 7]. Problems that are common to many smoking cessation apps are low enrolment rates of the target audiences, high attrition rates, and low frequency of use among those enrolled [8–12], and low completion rates of tasks and modules [13]. This seriously limits the potential impact of those apps at population level [14].

Studies of user engagement with smoking cessation apps can provide useful information to develop more attractive apps or improve existing apps. This can ensure that apps are effective and used repeatedly by a large proportion of the target audience, and that they meet users’ needs.

While this topic has seldom been explored, the available studies show that engagement with smoking cessation apps is associated with user’s characteristics such as literacy skills, younger age, being a woman, not being depressed, lower levels of tobacco dependence, greater acceptance of craving, and perceived utility of the app [8–10, 14]. Identifying the characteristics of participants who are less likely to engage with the app can help develop elements targeted at these specific subgroups and improve their experience. For instance, apps can include gender- or age-specific content, video and audio files for those with low reading skills, and content targeted at those with high levels of cigarette dependence or low acceptance of craving.

Study Purpose

Therefore, the present study aims at identifying the best predictors of the users’ engagement with a smartphone application for smoking cessation and at exploring the associations between these predictors and utilization variables that were collected automatically by the app.

Research Questions

This was a secondary analysis of data from a randomized controlled trial testing the efficacy of the Stop-Tabac smartphone app for smoking cessation [6]. We addressed six questions: what are the most important predictors (in ranking order) of the number of different days when participants accessed the app (RQ1), of the number of times the app was opened (RQ3), and of the app use duration (i.e., interval in days between the first and last day the app was opened) (RQ5)? We also examined the relationships (association probabilities) between each predictor and each of the same three outcomes (RQ2, RQ4, and RQ6). Because this was an exploratory study, no hypotheses associated with the research questions were made, but all the tested associations made sense.

MethodsThe Stop-Tabac App

The full Stop-Tabac app is based on the self-determination theory of behavior change [15], allowing high emphasis with autonomy (enhancing patient determination of behavior change, personalization of the app and rewards), competence (e.g., tools to enhance behavior change capabilities, tailored messages), and relatedness (e.g., with the app messages and with the app community, sharing progress). It is also built, in coherence with the Capability-Opportunity-Motivation-Behaviour (COMB) theory [16], trying as much as possible to reinforce capability and motivation. Like in some other digital tools [17], the app is adapted to include automated versions of several components offered by smoking cessation interventions such as assessment and feedback, motivational enhancement interventions, and relapse prevention strategies [18] as well as goal setting, action planning, and feedback in relation to goal achievement [19]. For instance, the Stop-Tabac app includes information pages, calculators (number of cigarettes not smoked, money and days of life gained since quitting), individually tailored (automatized) counseling reports, a quiz, phone numbers of quitlines, and a module on nicotine replacement therapy and e-cigarettes. The Stop-Tabac app is available at no charge in the Apple App Store and Google Play Store. It is a stand-alone intervention, and there was no human involvement or support in this intervention. It was rated among the best smoking cessation apps globally by two distinct studies independent from the authors [3, 20]. The control version of the app includes a few features that are liked by users (brief information pages, calculators of unsmoked cigarettes, money saved, and life expectancy gained since quitting). For more information, see description of the protocol trial [21].

Participants’ Characteristics

We enrolled 5,293 daily smokers in the randomized trial. However, in the present study we included only the 1,861 participants (35% of 5,293) who completed at least the baseline and 1-month follow-up questionnaire (treatment group n = 902, control group n = 959), because we used the information collected after 1 month to predict outcomes after 6 months. The 1,861 participants were included in all data analysis models built in the present study.

Recruitment and Sampling Procedures

The recruitment procedure was previously described [6, 20]. Briefly, eligible participants were adult daily cigarette smokers who lived in Switzerland or in France, and who set a quit date within 1 month of enrollment. Participants were enrolled via advertisements on the Internet. After downloading the app from the app stores, participants clicked on a link in the app leading to the online informed consent form and online screening and baseline questionnaire. Then, eligible participants received a code that enabled them to unlock either a full version of the Stop-Tabac smoking cessation app or a control version. Eligibility assessment was done automatically.

Data Collection

Participants answered follow-up questionnaires online 1 month and 6 months after their target quit date. Non-respondents received three reminders by e-mail, and then one text or WhatsApp message; then, they received the questionnaire by postal mail and the remaining non-respondents were contacted by phone. The questions were developed for the purpose of this study and were not submitted to an evaluation of their psychometric properties or to prior validation tests. The registration and consent form and the baseline and follow-up questionnaires are available here: https://archive.org/details/@stopdependance_ch.

Predictor Variables

Variables collected at baseline were country of residence, age, sex; experiment group category (full Stop-Tabac app or control version); smoking status, number of years as a smoker, number of cigarettes smoked per day, number of minutes between waking up and smoking the first cigarette of the day (which is an indicator of dependence), smoking other products, use of heated tobacco, electronic cigarettes and nicotine replacement medications (patch, gum, tablet, etc.), each on 4-point response scales, and a 2-item depression screening test (yes, no) [22]. The 1-month questionnaire covered the same variables as at baseline plus: usage of any smoking cessation app after entry in the study (yes, no), intention to stop smoking (3-point response scale), and perceived helpfulness of the app (6-point response scale).

Outcome Variables

The following information was collected automatically by the app itself in 100% of participants at the end of the study: (a) the number of different days when each participant used the app, (b) the number of times the app was opened, and the (c) the duration of app use (i.e., the interval in days between the first and last day the app was opened). Online supplementary material (for all online suppl. material, see www.karger.com/doi/10.1159/000530111) presents a table with descriptive statistics of the participants’ responses on the predictors and outcome variables.

Ethics

The study protocol was submitted to the Cantonal Ethics Committee in Geneva (number: Req-2018-00356) who answered that the app being no medical device, the study did not need their approval. The commission did therefore not review the protocol but wrote in an e-mail dated May 16, 2018, that “everything indicates that this study will take place in compliance with the general ethical principles applicable to any research involving people.”

Data Analysis

To analyze data we used machine learning algorithms instead of traditional methods, because these algorithms have hyperparameters that can be used to choose the models that best fit the data and optimization parameters to improve prediction capabilities [23, 24]. Machine learning classification and regression algorithms are prediction-designed, but they do not offer inference statistics. Therefore, we resorted to logistic regression to obtain inference information (variable association probability metrics).

To answer RQ1, RQ3, and RQ5, we built machine learning regression models using the random forest (RF) algorithm [23, 24]. Machine learning models are essentially predictive. They are constructed in two phases [23, 24]: the learning stage where the model analyzes and “learn” from the variables associations/relations; and the second stage where the model uses the “learned knowledge” to predict. RF regression models yield, among other outputs, the importance of each predictor variable determined on the basis of a measure called %IncMSE (percent increase in mean squared error). The %IncMSE reflects the increase in MSE (estimated with out-of-bag cross-validation) as a result of each variable being permuted (values randomly shuffled); in simple words, it describes how much (in terms of percentage) the model increases its MSE by excluding each variable. The more the MSE increases, the more important the variable is for the prediction. Thus, the variables can be presented in ranking order of importance. The selection of the most important predictors can be based on the ranking order and in the %IncMSE median value: predictors with %IncMSE values above the median value can be selected as the most important among all set of modeled predictors. RF models are nonparametric; i.e., they do not require a particular structure on the data, and as such they can capture nonlinear relationships, including interactions between the predictors [23, 24].

To answer RQ2, RQ4, and RQ6, we built multinomial logistic regression models because the outcome variables were not normally distributed (they were highly skewed to the left), and therefore they could not be used in linear models. We recoded the outcomes into two categories: low (minimum [min] – first quartile [Q1] values = coded 0) versus intermediate/high (Q2 to maximum values = coded 1).

Some of follow-up measures had missing values. These missing values were handled by the RF algorithm function “rfImpute()” which uses a nearest neighbor machine learning approach either to impute values or to weight their absence [18, 19]. We also used this algorithm to control if the imputation had an effect of the analysis results, which was not the case.

ResultsParticipants’ Characteristics

Participants were 38.2 years old on average (range 19–75 years, SD = 10.9); most were women (66%); most lived in France (73%); they smoked on average 15.5 cigarettes per day (SD = 7.6); they had been smoking for 19.3 years on average (SD = 11.2); and 59% tested positive on the depression screening test.

Utilization of the App

In the treatment group, the utilization of the app was as follows: different days participants open the app: median = 7, mean = 28; the number of times participants accessed the app: median = 26, mean = 152.87; the interval in days the participant used the app: median = 46, mean = 123.71. In the control group, the utilization of the app was as follows: different days participants open the app: median = 15, mean = 16.84; the number of times participants accessed the app: median = 69, mean = 65.23; the interval in days the participant used the app: median = 87, mean = 90.8.

The Best Predictors of the Number of Different Days when the App Was Accessed

Table 1 presents the machine learning results and lists the 20 variables retained by the algorithm as the best predictors of the number of different days when the app was accessed. As shown in Table 1, among the 20 variables, the %IncMSE ranged from a high of 47.21 (intention to quit smoking assessed at 1-month follow-up) to a low of 0.48 (sex), with a median value of 4.03 (use of heated tobacco product as self-reported after 1-month follow-up). As mentioned, the higher the %IncMSE value, the more important the variable is for successful prediction. In other words, the %IncMSE of a given predictor variable reflects the value of the MSE increase in the prediction model if that variable is removed.

Table 1.

Predictors of the number of different days when the app was accessed, ranked in decreasing order of importance

Predictors%IncMSEIntention to quit smoking assessed at 1-month follow-up47.21Cigarettes/day measured at 1-month follow-up38.78Perceived helpfulness of the app measured after 1 month22.88Having quit smoking after 1 month17.47Experiment group (treatment vs. control)15.45Age8.94Number of years smoking7.65Current use of any smoking cessation app, as self-reported at 1-month follow-up5.68Cigarettes/day as measured at baseline5.09Use of e-cigarettes at baseline4.56Use of heated tobacco product as self-reported after 1-month follow-up4.03Number of minutes before the first cigarette of the day3.42Use of nicotine medications after 1-month follow-up2.85Use of e-cigarettes after 1-month follow-up2.01Use of nicotine medications at baseline1.71Depression screening test positive1.22Use/smoke other tobacco products at baseline1.14Country (Switzerland vs. France)0.81Use of heated tobacco product at baseline0.58Sex0.48

The 10 most important predictors (those with %IncMSE values above the median) were, in decreasing order of importance, intention to quit smoking assessed after 1 month; number of cigarettes smoked per day measured after 1 month; perceived helpfulness of the app measured after 1 month; having quit smoking after 1 month; experiment group (treatment vs. control); age; number of years smoking; use of any smoking cessation app after 1 month; cigarettes per day measured at baseline; and use of e-cigarettes at baseline. After controlling for the group who used the “dressed down” app, the findings remained statistically the same.

Associations between Each Predictor and the Number of Different Days when the App Was Accessed

Table 2 reports the relationships between each predictor and the number of different days when the app was accessed, a summary of the multinomial logistic regression model. In Table 2, the dichotomic outcome was belonging to the group with intermediate or high number of different days when the app was accessed (“recurrent users,” 3 upper quartiles merged) versus belonging to the group with a low number of different days of use (lowest quartile). The best way to understand Table 2 is to look at the value of coefficients, the sign (negative or positive) associated with the coefficient b, at the corresponding p value, and to take into consideration that the statistical model was designed to predict the probability of a participant belonging to the “recurrent users” class (since the “lowest quartile” class was set as reference class). First, for the largest coefficients, if the p value is inferior to 0.05, that means that the relationship between the relevant predictor variable and the outcome is both substantial and statistically significant. If the sign associated with the coefficient b is positive, that means that an increase in the value of the predictor variable increases the likelihood of belonging to the “recurrent users” class; if the sign is negative, that means that an increase in the predictor values decreases the likelihood of belonging to the “recurrent users” class and increases the likelihood of belonging to the “lowest quartile” class. Thus, the predictors significantly associated with this outcome were: belonging to the control group (vs. treatment group) decreased by 43% the odds of being a recurrent user (odds ratio [OR] = 0.57); a higher number of minutes before the first cigarette (i.e., lower dependence) decreased by 13% per minute the odds of being a recurrent user (OR = 0.87 per minute); the use of any smoking cessation app after 1 month doubled the odds of being a recurrent user; a higher level of intention to quit smoking assessed after 1 month increased by 3.4 times per point on a 3-point scale the odds of being a recurrent user; a higher level of perceived helpfulness of the app measured after 1 month increased by 38% per point the odds of being a recurrent user (OR = 1.38 per point on a 6-point scale); having quit smoking after 1 month increased by 37% the odds of being a recurrent user.

Table 2.

Associations between the predictors and the number of different days when the app was accessed

Predictors’ categoryPredictorsbORSEp value*95% CIDemographicsCountry (Switzerland vs. France)−0.2660.7670.1430.0640.5791.015Age−0.0120.9880.0090.1500.9711.004Sex−0.2170.8050.1320.1000.6221.043Treatment/controlExperiment group−0.5640.5690.124<0.0010.4460.726BaselineNumber of years smoking−0.0080.9920.0080.3520.9761.009Cigarettes/day−0.0160.9840.0100.1050.9641.003Number of minutes before the first cigarette of the day−0.1420.8680.0640.0260.7660.983Use/smoke other tobacco products−0.0060.9940.0540.9180.8951.105Use of heated tobacco products0.1141.1200.1350.4010.8591.461Use of e-cigarettes−0.0400.9610.0650.5400.8461.091Use of nicotine medications−0.1010.9040.1840.5840.6311.297Depression0.1341.1440.1230.2760.8981.456Follow-up after 1 monthCigarettes/day0.0201.0210.0130.1260.9941.048Use of heated tobacco product−0.5170.5960.2750.0600.3481.022Use of e-cigarettes−0.1560.8560.0910.0870.7161.023Use of nicotine medications0.0481.0490.1610.7670.7641.440Current use of any smoking cessation app0.7852.1930.132<0.0011.6912.843Intention to quit smoking1.2363.4430.150<0.0012.5654.623Perceived helpfulness of the app0.3211.3780.063<0.0011.2181.559Having quit smoking−0.4720.6240.143<0.0010.4710.826The Best Predictors of the Number of Times the App Was Opened

Table 3 displays the machine learning results and lists the 20 best predictors of the number of times the app was opened. As shown in Table 3, among the 20 variables, the %IncMSE ranged from a high of 44.31 (intention to quit smoking assessed at 1-month follow-up) to a low of 0.03 (sex), with a median value of 3.12 (use of nicotine medications after 1 month).

Table 3.

Predictors of the number of times the app was opened, ranked in decreasing order of importance

Predictors%IncMSEIntention to quit smoking assessed after 1 month44.31Cigarettes/day measured at 1-month follow-up39.96Perceived helpfulness of the app measured after 1 month27.09Experiment group (treatment vs. control)25.89Having quit smoking after 1-month follow-up14.97Current use of any smoking cessation app, as self-reported at 1-month follow-up9.98Number of years smoking9.72Age8.31Cigarettes/day measured at baseline5.94Number of minutes before the first cigarette of the day3.63Use of nicotine medications after 1 month3.12Depression3.08Use of heated tobacco product as self-reported after 1 month2.99Use of e-cigarettes at baseline2.44Use of e-cigarettes as self-reported after 1 month2.23Country (Switzerland vs. France)0.77Use of nicotine medications at baseline0.61Use/smoke other tobacco products at baseline0.57Use of heated tobacco product as self-reported after 1-month follow-up0.49Sex0.03

The 10 best predictors (those with %IncMSE values above the median) were, in decreasing order of importance, intention to quit smoking assessed after 1 month; cigarettes per day measured after 1 month; perceived helpfulness of the app measured after 1 month; experiment group (vs. control); having quit smoking at 1 month; use of any smoking cessation app as self-reported after 1 month; number of years smoking; age; cigarettes per day; and number of minutes before the first cigarette of the day measured at baseline. After controlling for the group who used the “dressed down” app, the findings remained statistically the same.

Associations between Each Predictor and the Number of Times the App Was Opened

Table 4 displays the relationships between each predictor and the number of times the app was opened. In this table, the dichotomic outcome was belonging to the group with intermediate or high number of times when the app was opened (“frequent users”) versus belonging to the group with a low number of times when the app was opened. The predictors significantly associated with this outcome were: belonging to the control group (vs. treatment group) decreased by 60% the odds of being a frequent user (OR = 0.401); a higher number of cigarettes/day as measured at baseline decreased by 3% per cig./day the odds of being a frequent user (OR = 0.97 per cig./day); a higher number of minutes before the first cigarette decreased by 13% per minute the odds of being a frequent user (OR = 0.87 per minute); a higher number of cigarettes/day as measured after 1 month decreased by 4% per cig./day the odds of being a frequent user (OR = 1.04); the use of any smoking cessation app after 1-month follow-up doubled the odds of being a frequent user; a higher level of intention to quit smoking assessed after 1 month doubled the odds of being a frequent user; a higher perceived helpfulness of the app measured after 1 month increased by 44% per point the odds of being a frequent user (OR = 1.44 per point on a 6-point scale); having quit smoking after 1 month decreased by 44% the odds of being a frequent user (OR = 0.56).

Table 4.

Associations between the predictors and the number of times the app was opened

Predictors’ categoryPredictorsbORSEp value*95% CIDemographicsCountry (Switzerland vs. France)−0.1170.8900.1460.4240.6681.184Age−0.0070.9930.0090.4200.9761.010Sex−0.2420.7850.1350.0730.6031.023Treatment/controlExperiment group−0.9140.4010.130<0.0010.3110.518BaselineNumber of years smoking0.0001.0000.0090.9590.9831.018Cigarettes/day0.0330.9680.0110.0020.9480.988Number of minutes before the first cigarette of the day−0.1390.8700.0650.0340.7650.989Use/smoke other tobacco products−0.0280.9730.0550.6170.8731.084Use of heated tobacco product0.0991.1040.1380.4740.8421.446Use of e-cigarettes−0.0520.9490.0670.4330.8331.081Use of nicotine medications−0.2610.7700.1900.1690.5311.117Depression0.0361.0370.1260.7740.8101.328Follow-up after 1 monthCigarettes/day0.0381.0390.0130.0051.0121.067Use of heated tobacco product−0.3170.7280.2780.2540.4231.255Use of e-cigarettes−0.0890.9150.0940.3440.7621.100Use of nicotine medications0.1981.2190.1690.2400.8761.696Current use of any smoking cessation app0.7632.1450.138<0.0011.6372.811Intention to quit smoking0.8102.2470.149<0.0011.6793.008Perceived helpfulness of the app0.3631.4380.066<0.0011.2631.637Having quit smoking−0.5820.5590.150<0.0010.4160.750The Best Predictors of the Apps Use Duration

Table 5 exhibits the machine learning results and lists the best 22 predictors of the app use duration (i.e., interval in days between first and last use). The 10 most important predictors were, in decreasing order of importance, cigarettes/day measured after 1 month; intention to quit smoking assessed after 1 month; perceived helpfulness of the app measured after 1 month; having quit smoking after 1 month; use of any smoking cessation app as self-reported at 1 month; age; number of years smoking; experimental group (vs. control group); use of nicotine medications at 1 month; and minutes before the first cigarette measured at baseline.

Table 5.

The importance of each predictor of the apps use duration: decreasing order

Predictors%IncMSECigarettes/day measured at 1-month follow-up135.43Intention to quit smoking assessed after 1-month follow-up125.52Perceived helpfulness of the app measured after 1-month follow-up56.59Having quit smoking after 1-month follow-up44.66Current use of any smoking cessation app, as self-reported at 1-month follow-up28.23Age26.98Number of years smoking22.02Experiment group (treatment vs. control)20.01Use of nicotine medications after 1 month15.68Number of minutes before the first cigarette of the day14.86Use of nicotine medications at baseline12.87Use of heated tobacco product as self-reported after 1-month follow-up12.67Cigarettes/day measured at baseline9.54Sex8.51Depression screening test positive7.92Country (Switzerland vs. France)6.76Use of e-cigarettes at baseline4.21Use/smoke other tobacco products at baseline3.51Use of e-cigarettes after 1-month follow-up1.57Use of heated tobacco product at baseline0.61Associations between Each Predictor and the Apps Use Duration

Table 6 displays the relationships between each predictor variable and the app use duration. In this table, the dichotomic outcome was belonging to the group with intermediate or high duration of app use (“long-term users”) versus belonging to the group with low duration of use. The predictors significantly associated with this outcome were: living in France (vs. in Switzerland) decreased by 34% the odds of being a long-term user; older age decreased by 2% per year the likelihood of being a long-term user (OR = 0.98 per year of age); belonging to the control group (vs. treatment group) decreased by 39% the odds of being a long-term user; a higher number of cigarettes/day measured after 1 month increased the likelihood of being a long-term user (OR = 1.05 per cig/day); the use of any smoking cessation app after 1 month increased by 85% the odds of being a long-term user; intention to quit smoking assessed after 1 month increased by 2.5 times per point the odds of being a long-term user (OR = 2.5 per point on a 3-point scale); perceiving the app as helpful after 1 month increased by 40% per point the odds of being a long-term user (OR = 1.4 per point on a 6-point scale).

Table 6.

Associations between the predictors and the apps use duration

Predictors’ categoryPredictorsbORSEp value*95% CIDemographicsCountry (Switzerland vs. France)−0.4110.6630.1470.0050.4970.885Age0.