Are electronic household clusters’ data collected more efficiently?Timeliness

For each individual module, enumerators in electronic cluster households spent significantly less time administering the questionnaire, for an average time savings of 7.4 min per module per household (95% CI 6.6–8.2 min) (Fig. 3). Comparing the time required to administer the entire survey in a household, enumerators also spent significantly less time administering them in electronic cluster households (248 min total survey investment time [95% CI 239.1–258.1]) compared to paper clusters (289 min [95% CI 277.8–299.8]), for an average savings of 40 min per household (95% CI 25.4–55.1 min).

Fig. 3

Average time in minutes spent performing household data collection: Paper vs. Electronic

Paper-based questionnaires contained responses which needed to be coded into categories to allow for data analysis. Parwan DMU staff spent an average of 5 min and 54 s per household to code responses, while in Panjshir DMU staff spent an average of 8 min and 33 s to code responses per household. The first entry from paper into the electronic database of one Parwan household of data required an average of 18 min and 40 s; the verification entry required an average of 14 min and 55 s. Panjshir’s first entry household average time was 13 min and 40 s, while the verification data entry required an average of 13 min and 16 s. Additional quality control performed by two lead DMU staff required approximately 9 eight-hour working days, or an average of 18 min per household.

Programming electronic surveys for use in ODK software lasted approximately 73.5 eight-hour working days. Programmers estimated that 30% of these hours were spent revising skip logic and survey screen formatting to accommodate different questionnaire requirements for male and female respondents. 10% of these hours were spent translating the survey into Dari language. Data were transferred to the server database using WiFi hotspots installed via tablet sim cards. Coding the electronically collected data required two eight-hour working days, and no verification entry was performed or necessary. Additional data cleaning and quality control performed by two lead DMU staff required about 7 eight-hour working days each. Figure 4 presents the time used for all data management tasks (programming and quality control for ODK; coding, data entry and quality control for paper) on a household level. Average time spent for all data management tasks was approximately 74 min for paper-based households and 66 min for ODK households. Total data management for ODK data took a total of 719 h (90 person-days), only 131 h (16.4 person-days) of which were spent for processing (coding and cleaning). PAPI data coding, entry, management and cleaning alone consumed a total of 812 h (101.5 person-days).

Fig. 4

Time performing data management tasks per household: Paper vs. Electronic

Cost efficiency

The actual combined expenditures in personnel fees and transportation, data management, training, procurement of equipment, and printing amounted to a per province total of 59,958 USD spent for the combined electronic clusters, and 35,762 USD spent for the combined paper clusters (Table 1). Field team personnel costs (fees and transportation) comprised the majority of these costs at 26,100 USD in both paper and electronic clusters, covering six interviewers, one data editor, and one supervisor for each province. These teams of eight conducted surveys in both the paper and electronic clusters of each province. The largest cost difference resulted from the investment required to electronically program the survey, increasing the cost for CAPI clusters by 19,634 USD. Costs to develop the survey questionnaires into a word processor were not factored into this analysis. Training costs for CAPI staff exceeded those of the PAPI staff by 2826 USD because they received both PAPI and CAPI training. This was necessary because CAPI staff also performed paper-based surveys in the remaining 32 provinces of Afghanistan, and these skills certainly contributed to the quality and efficiency of the work performed by survey staff in the CAPI clusters. Equipment costs of 2157 USD for CAPI cluster tablets and charging devices were not required in the paper clusters, which only had an additional 516 USD of printing costs for the paper forms not needed in the electronic clusters. Per household the CAPI method cost 91.12 USD, while PAPI methods cost 54.43 USD per household.

Table 1 Cost comparison of PAPI and CAPI data collection and management

When initial costs associated with CAPI are removed (one-time survey programming and equipment costs), the total cost for CAPI would have been 38,346 USD. This however does not account for efficiencies which would be gained over time for data cleaning and management, and includes one replacement tablet, power bank, an internet package, umbrella and other equipment. The per household costs would also be more comparable to the PAPI costs at 58.28 USD.

Are electronically collected data of higher quality than data collected on paper?Data completeness

Differences in data quality were evaluated from the perspective of data completeness. In each module we calculated the number of missing data points out of all possible required data points for the forty-nine tracer variables only. For the pre-cleaned data (as defined above), the paper-based dataset was missing a slightly higher percentage of tracer data points (2.0%: 1202/59,424) compared to the electronically collected dataset (1.6%: 911/55,576) (Fig. 5a).

Fig. 5

a Pre-cleaned dataset completeness for Paper and Electronic. b Final dataset completeness for Paper and Electronic

After data cleaning, the electronically collected dataset was still missing slightly fewer tracer variable data points (1.1%: 626/55,466) than the paper dataset (2.0%: 1223/59,980) (Fig. 5b). In both the pre-cleaned and final datasets paper-based data only modules on women’s health and immunization were lacking any tracer data. The electronically collected dataset missed tracer data points in all modules, but primarily in the women’s and immunization modules. One question related to children’s vaccination cards was unexpectedly routinely skipped, accounting for more than 95% of the missing data points in the immunization module.

How acceptable did field enumerators and respondents find electronic data collection compared to paper based data collection?Clarity and ease of use

Data collectors were interviewed about clarity and ease of using electronic versus paper forms for data collection. Most enumerators preferred electronic data collection over paper due to substantial time savings, user-friendliness, improved data security, and less conspicuity when traveling. These benefits of CAPI were cited in all three focus group discussions. Automated skip logic was the most frequently mentioned reason for preferring CAPI. This removed the necessity to manually decide which questions need to be asked based on previous responses, as the electronic program did this for data collectors. Many enumerators mentioned that this was a helpful time-saver during data collection.

Enumerators also appreciated the logic constraints of CAPI, and that certain fields did not make it possible to enter impossible answers, for example an age which is too high or too low. In terms of practical convenience, some enumerators reported that tablets were preferable to paper when it was raining, as papers would sometimes become wet and difficult to read, while a tablet is easier to protect from rain. Data collectors also enjoyed that tablets were lighter to carry than large piles of paper questionnaires.

Some limitations around CAPI were noted. A few data collectors reported that previously entered information appeared to sometimes be erased when they scrolled back in the questionnaire to correct responses. One enumerator noted that when the tablet memory was full, tablets would turn off, and sometimes tablets would just turn off for no apparent reason. Several enumerators mentioned that when they only recognized mistakes after multiple incorrect or irrelevant responses were entered into the tablet, it was burdensome to remove all incorrect data from the electronic form. With paper it is easy to simply disregard some incorrect responses by crossing them out and writing the correct answer on the spot. CAPI moves up this type of data cleaning to the field rather than the data management office.

Interactions with household respondents

Many of the enumerators mentioned that household respondents were skeptical about safety of their data when they would be entered into tablets. Some household respondents were concerned that the tablets may be used to photograph them or record their voices. This was more of a concern among women than among men. However, they also mostly reported that these concerns were easily resolved by enumerator reassurance about electronic data confidentiality. Conversely some household respondents (generally younger) felt more secure about their information being entered into a tablet rather than paper, because they understood that it would be more difficult for people to get a password to open up the questionnaires on tablets, than it would be to read responses on paper. 54% (80/148) of enumerators in a follow-up survey reported that they would not feel safe and comfortable if they had to use tablets for household data collection. Among those who reported not feeling safe, 22% (n = 18) mentioned Taliban concerns, 33% (n = 26) mentioned concerns around insecurity, 19% (n = 15) mentioned that as tablets are known to collect GPS coordinates, this makes them feel vulnerable to retaliation from either Taliban or communities, and 33% (n = 26) reported that household and community members would be hesitant to accept this practice. Among those working in provinces in which there was Taliban occupancy in 2019 (n = 88), 47 (53%) anticipated not feeling safe collecting data in households in the future.

Safety concerns

When asked which method of data collection they would prefer to use for a hypothetical next survey, most enumerators indicated that CAPI is preferred when possible. It was perceived as much more comfortable than PAPI, more convenient, and by some respondents it was perceived to be more safe even in insecure areas. One can more easily conceal tablets whereas it is often more difficult to hide a large stack of paper questionnaires. The focus group discussions revealed that some enumerators felt unsafe carrying tablets though due to fears that people may steal them. The two-question survey administered to field staff trainees in all 34 provinces provided more insight into security concerns: 48% (72/148) of enumerators reported that they did not feel safe and comfortable when they used tablets for data collection in a recent health facility survey. Among the 72 respondents who reported not feeling safe, 79% (n = 57) say they are concerned about repercussions from Taliban who view use of smart devices with suspicion due to their capacity to capture GPS coordinates, and subsequent future retaliation from either Taliban or communities. 46% (n = 33) mentioned concerns around general insecurity or theft. Among those working in provinces with Taliban occupancy (n = 88), 38 (43%) reported feeling unsafe using tablets to collect data in health facilities. Enumerators who reported feeling unsafe using tablets in a health facility were 4.5 times more likely (95% CI 2.9–7.2) to report anticipate feeling unsafe doing so in a household setting. A few enumerators also mentioned that collecting data electronically in the future depends on the skills of the survey staff. One enumerator mentioned that the only advantage of using paper in the future is that communities more quickly/ easily recognize and accept PAPI.