Strengths and limitations of computer assisted telephone interviews (CATI) for nutrition data collection in rural Kenya

Despite progress in fighting undernutrition, Africa has the highest rates of undernutrition globally, exacerbated by drought and conflict. Mobile phones are emerging as a tool for rapid, cost effective data collection at scale in Africa, as mobile phone subscriptions and phone ownership increase at the highest rates globally. To assess the feasibility and biases of collecting nutrition data via computer assisted telephone interviews (CATI) to mobile phones, we measured Minimum Dietary Diversity for Women (MDD-W) and Minimum Acceptable Diet for Infants and Young Children (MAD) using a one-week test-retest study on 1,821 households in Kenya. Accuracy and bias were assessed by comparing individual scores and population prevalence of undernutrition collected via CATI with data collected via traditional face-to-face (F2F) surveys. We were able to reach 75% (n = 1366) of study participants via CATI. Women’s reported nutrition scores did not change with mode for MDD-W, but children’s nutrition scores were significantly higher when measured via CATI for both the dietary diversity (mean increase of 0.45 food groups, 95% confidence interval 0.34–0.56) and meal frequency (mean increase of 0.75 meals per day, 95% confidence interval 0.53–0.96) components of MAD. This resulted in a 17% higher inferred prevalence of adequate diets for infants and young children via CATI. Women without mobile-phone access were younger and had fewer assets than women with access, but only marginally lower dietary diversity, resulting in a small non-coverage bias of 1–7% due to exclusion of participants without mobile phones. Thus, collecting nutrition data from rural women in Africa with mobile phones may result in 0% (no change) to as much as 25% higher nutrition estimates than collecting that information in face-to-face interviews. We selected Baringo and Kitui Counties in Kenya as study sites, as these counties differ in socioeconomic and environmental conditions, mobile phone access and network coverage (S1 Table). Baringo County (0° 28’ N, 35° 59’ E) is characterized by mixed crop-livestock farming systems in the highlands and pastoralism in the lowlands, and generally receives adequate rainfall for agriculture throughout the county [41]. In Baringo, approximately 52% of the population is below poverty line [42], 30% of children are stunted [43], and 50% of the population owns a mobile phone [16]. Kitui County (1° 22’ S, 38° 23’ E) is generally at lower elevation than Baringo and is dominated by agro-pastoralism with sorghum, millet, and small livestock. There is inadequate rainfall for agriculture in the easternmost parts of the county. Kitui has higher rates of poverty (60%), child stunting (46%), and lower mobile phone ownership (25%), than Baringo (see above references). Study locations within each county were selected through a combination of purposeful and random sampling of administrative units. We included all districts within each county (Kitui has two districts and Baringo four). Within each district in Kitui, we purposefully selected the divisions with the highest and lowest number of households. From the two districts in Baringo with the highest population, we purposefully selected the divisions with the lowest number of households, while from the two districts with the lowest population, we selected the divisions with the highest number of households, for a total of four divisions in each county. Within each division, we randomly chose two sublocations, for a total of 32 sublocations representing the economic and geographic variation within each county. We tested data collection mode with two internationally-validated nutrition indicators: Minimum Dietary Diversity for Women (MDD-W) [44] and Minimum Acceptable Diet (MAD) for infants and young children [45] (Table 1). MDD-W accesses the micronutrient adequacy in the diet of women of reproductive age, a critical predictor of both maternal and child nutrition. MAD assesses the adequacy of Infant and Young Child Feeding (IYCF) based on both dietary diversity (MDD) and meal frequency (MMF). We also included a sociodemographic indicator, the Kenya Progress Out of Poverty Index (PPI) [46] as a wealth proxy to assess differences in mode effect on MDD-W and MAD by wealth. The indicators can be collected in short surveys of approximately five minutes, do not rely on pictorial demonstrations of food groups, and are calculated on a scorecard methodology based on respondents’ answers to several questions. The indicators differ in target population, type of data generated in question response, and conversion of scores to population prevalence (Table 1). To meet the threshold for adequate dietary diversity, participants must have consumed at least five food groups out of ten in the past 24 hours for MDD-W and four out of seven for MDD. Recommended MMF is satisfied when either (a) breastfed infants less than nine months old eat at least twice a day, (b) breastfed infants older than nine months eat at least three times per day, or (c) non-breastfed infants regardless of age consume milk at least twice per day and other foods at least four times per day. Both MDD and MMF criteria must be satisfied for MAD. Finally, PPI raw scores are converted into below poverty-line (defined as $1.25/day) likelihoods using nonlinear conversion tables [46]. Thus, we were able to examine the equivalence between modes at three levels: population prevalence, individual indicator score, and responses to indicator subquestions. Sector, target population, survey length, data type, and conversion of score to prevalence for indicators in this study. 1 MMF is considered a conditional step function because the inflection point of the function depends on the age and breastfeeding status of the child. 2 The type of function used to convert PPI score to poverty likelihood depends on the definition of poverty used (e.g. national poverty line, $1.25/day, etc.). Two separate surveys were used concurrently for different target populations: MDD-W, PPI, and basic demography for women of reproductive age, and MAD, PPI and basic demography for adult caretakers of children aged 6–23.99 months. We used a test-retest design with four treatment arms (T1-T4) to evaluate the effect of data collection mode on nutrition indicators (Fig 1). In the main treatment arms T1 and T2, participants were interviewed with both CATI and F2F modes. We included two control arms to the experimental design. For the first control arm (T3), participants were interviewed via F2F mode in both rounds to understand potential learning and/or temporal effects. A fourth arm (T4) of F2F interviews for respondents with no phone access was included for MDD-W, to better understand non-coverage bias in conducting studies via CATI. Phone access was defined as owning a mobile phone or having access to one via intrahousehold sharing and was determined by asking potential participants. Test-retest mode experiment on two nutrition indicators, MDD-W and MAD. Survey consisted of four treatment arms: two main treatment arms testing for mode differences, arm 3 controlling for temporal effects and arm 4 controlling for non-coverage bias. Target sample sizes are indicated. Participants were systematically assigned to treatment arms during the test round, whereby participants meeting the inclusion criteria for the indicators were alternately assigned to T1 and T2, and every sixth house visited per day was assigned to T3. For participants in T1 (CATI first), phone numbers were collected and participants were called the next day. For participants in T2 and T3, F2F interviews were conducted immediately. In the retest round respondents were re-interviewed using the other mode, e.g. participants in T1 were relocated and interviewed F2F, while T2 participants were called via CATI. Participants in T3 were reinterviewed using F2F (Fig 1). The retest round occurred approximately nine days (8.7 ± 4.4 days) following the first survey, accounting for activities such as market days that may have altered diet choices when possible. G*Power [47] was used to calculate sample sizes to detect a mode bias of 1.5 in population prevalence using McNemar’s Exact Test for paired nominal data, assuming 25% of participants would switch indicator status (above or below threshold) between rounds. Although attrition rates in mobile surveys can vary dramatically with mode and location, for the sample size calculations we assumed a 15% attrition rate as CATI and F2F have low attrition rates, and studies with only two rounds have lower attrition rates than panel surveys [14]. For the power analyses, we used alpha = 0.2 and beta = 0.05 to minimize the rate of false equivalences. The resulting target sample size was 1000 participants, which we split evenly into the T1 and T2 arms. Control group sizes were chosen as 12.5% or 125 respondents for F2F controls, and 20% or 200 respondents for no-phone controls. Thirty-two field enumerators (16 men and 16 women) and eight phone (3 men and 5 women) operators were recruited from local populations in Baringo and Kitui Counties to ensure familiarity with local languages, diets, and context. Enumerators and operators were trained together for two days to standardize survey methodology and interpretation of responses. To collect phone numbers and conduct F2F interviews, enumerator teams working simultaneously in Kitui and Baringo visited each identified sublocation in sequence. In each sublocation, enumerators sampled households semi-randomly, as enumerators traveled between households on foot, but were located in different parts of the sublocation, and did not sample adjacent households. Household sampling continued until 70 suitable households were identified for participation in the experiment. Only one participant was interviewed per household. Both enumerators and operators used the same survey instrument, Enketo (Ona, Nairobi, Kenya and Washington, DC, USA), based on the Open Data Kit platform [48], to collect survey data. Operators used Enketo’s web form on desktops in United Nations World Food Programme’s in-house call center in Nairobi to conduct CATI interviews, while enumerators used a similar offline platform on tablets for the face-to-face surveys in the field. All data were uploaded daily to a centralized ODK server from where raw data was then extracted and analyzed. We use F2F results as the “control” scores and CATI scores as the “treatment”, as F2F is the standard mode of nutrition data collection. We evaluated effect of data collection mode in several ways. First, paired t-tests evaluated mean score as a function of mode for participants that received both CATI and F2F modes. At the population level, Kolmogorov-Smirnov tests compared distributional differences (mean, variance, skew and kurtosis) of indicator scores between modes. Equivalence Tests [49] against varying levels of difference assessed if differences by mode were clinically significant (large enough in magnitude to alter a nutritionist’s interpretation of the resultant data). Linear mixed-effects models were used to examine survey methodology effects on nutrition scores, such as mode bias, enumerator bias [50], or temporal effects. Using a top-down approach [51], we first used the most complex fixed effects of interest (a three-way interaction between survey mode, round and enumerator gender) and found the optimal structure for the random effects (such as county and enumerator). Then, using the resulting random effects, we determined the optimal fixed-effects structure. For model selection we computed both Akaike’s Information Criterion (AIC) and the Bayesian Information Criterion (BIC). Both AIC and BIC reward model explanatory power and penalize model complexity, but BIC also accounts for the number of observations in the dataset [51]. Where AIC and BIC disagreed on the best-fit model, we chose the model with the lowest AIC, in order to guard against false negatives (e.g. declaring there in no effect of survey mode when there may be). Differences in responses to indicator component questions at the individual level were examined via McNemar’s Exact Test and paired t-tests for categorical and continuous data respectively. Resulting p values were corrected for multiple testing using the false discovery rate method [52]. Non-coverage bias was assessed by comparing dietary diversity, PPI, and demographic data between the phone access and no phone access groups in the retest round for MDD-W. The magnitude of the non-coverage bias, or the relative change in population-level estimates of dietary diversity by only surveying women with mobile phones, was estimated as Where Y-1 and Y-2 are the mean MDD-W scores for women with and without phone access, respectively, and (N2N1) is the proportion of women without access to mobile phones [53]. We estimated mobile phone access in our population from published sources [16, 35]. All data analyses were conducted in R [54]. The study protocol received research clearance and ethical approval from Kenya’s National Commission for Science, Technology and Innovation (NACOSTI), as well as the London School of Hygiene and Tropical Medicine (LSHTM). All methods were performed in accordance with NACOSTI and LSHTM guidelines. For all participants, oral informed consent was obtained by the enumerator and/or operator before beginning each survey. All efforts were made to ensure confidentiality of the participants. The data were stored in a password-protected computer and made accessible only to the core study team members. The analyses are presented in an aggregate format, phone numbers have been deleted, and all data has been anonymized. No incentives or remunerations were given for participation in this study.