Improving the Quality of Adult Mortality Data Collected in Demographic Surveys: Validation Study of a New Siblings’ Survival Questionnaire in Niakhar, Senegal

Background:In countries with limited vital registration, adult mortality is frequently estimated using siblings’ survival histories (SSHs) collected during Demographic and Health Surveys (DHS). These data are affected by reporting errors. We developed a new SSH questionnaire, the siblings’ survival calendar (SSC). It incorporates supplementary interviewing techniques to limit omissions of siblings and uses an event history calendar to improve reports of dates and ages. We hypothesized that the SSC would improve the quality of adult mortality data.Methods and Findings:We conducted a retrospective validation study among the population of the Niakhar Health and Demographic Surveillance System in Senegal. We randomly assigned men and women aged 15-59 y to an interview with either the DHS questionnaire or the SSC. We compared SSHs collected in each group to prospective data on adult mortality collected in Niakhar. The SSC reduced respondents’ tendency to round reports of dates and ages to the nearest multiple of five or ten (“heaping”). The SSC also had higher sensitivity in recording adult female deaths: among respondents whose sister(s) had died at an adult age in the past 15 y, 89.6% reported an adult female death during SSC interviews versus 75.6% in DHS interviews (p = 0.027). The specificity of the SSC was similar to that of the DHS questionnaire, i.e., it did not increase the number of false reports of deaths. However, the SSC did not improve the reporting of adult deaths among the brothers of respondents. Study limitations include sample selectivity, limited external validity, and multiple testing.Conclusions:The SSC has the potential to collect more accurate SSHs than the questionnaire used in DHS. Further research is needed to assess the effects of the SSC on estimates of adult mortality rates. Additional validation studies should be conducted in different social and epidemiological settings. © 2014 Helleringer et al. This study was approved by the Columbia University Medical Center institutional review board (Protocol AAAI9159) and by the Ethics committee of Senegal’s Ministry of Health and Social Action (SEN 12/11). Study participants provided informed consent in writing prior to participating in the study. This is a retrospective validation study of SSH data collected during surveys in the lower-middle-income country of Senegal. It estimates the accuracy of two survey instruments (the SSC and the standard DHS questionnaire, both described in detail below) by comparing the SSH data they yield to a prospective dataset on adult mortality, the Niakhar HDSS (reference dataset). We first selected and enrolled a stratified sample of individuals who had ever been registered by the Niakhar HDSS. In doing so, we oversampled individuals who had experienced at least one adult death among their maternal siblings, according to the HDSS. We then randomly allocated study participants to either an interview with the DHS questionnaire (control group) or to an interview with the SSC (experimental group). Finally, we compared the quality of SSH data obtained in each study group. The study focused on the population of the Niakhar HDSS site, located 120 km southeast of Dakar, Senegal’s capital. The population covered by the HDSS lives in 30 villages comprising ≈44,000 inhabitants as of 1 January 2013. Most of the population belongs to the Sereer ethnic group, with Wolof, Toucouleur, and Laobe minorities. The main language is Sereer but many people also speak Wolof, the most common language in Senegal. The main religious groups in the area are Muslim (≈80%) and Christian (≈20%). Households in Niakhar live traditionally on one food crop (millet) and one cash crop (groundnuts). They also raise a few cattle. The climate is typical of the sub-Sahel. The three largest villages in the area include health facilities, weekly markets, daily buses to Dakar, and several shops. The educational level is low: 50% of men and 75% of women in the HDSS population have never attended primary school. High levels of mobility, both permanent and temporary, also characterize the area. A large proportion of Niakhar residents move to Dakar, where they seek employment. A more detailed description of the Niakhar HDSS is given elsewhere [21]. Activities of the Niakhar HDSS started in 1962 in eight villages of the Niakhar area and were later expanded to 30 villages in 1983. An initial baseline census was carried out in 1962, followed by another census in 1983, when the study area was expanded. Since these censuses, data on demographic events have been collected from household informants during household visits. Study interviewers use a printed roster of household residents and inquire about the vital status of each household member, as well as possible changes in marital status and births since the previous household visit. New household members (including in-migrants) are added to the roster. Each individual who ever resided in the study area since the start of the HDSS has been assigned a unique ID number, under which HDSS data are stored. From 1962 to 1987, household visits were conducted yearly. From 1987 to 1997, household visits were conducted weekly because of requirements of vaccine trials conducted in the Niakhar area [22]–[24]. Between 1997 and 2007, household visits were conducted every 3 mo, and between 2008 and 2013, every 4 mo. In the Niakhar HDSS dataset, the date of birth of a population member is ascertained in one of two ways. It is recorded prospectively if s/he is born in the HDSS population between two household visits. It is assessed retrospectively if s/he was already present in the population at the time of the initial household census (i.e., 1962 or 1983) or if s/he first entered the HDSS population after birth, by migration. The age of population members is thus known with varying degrees of precision and certainty depending on the way they entered the HDSS dataset. The date of other vital events (e.g., deaths and migrations) is ascertained through household visits, when changes in household membership are recorded. The household informant is asked to report the day and month when a death/migration occurred. The dates of vital events are thus known with varying degrees of precision, depending on the frequency of household visits. For each event, the age at the time of the event is calculated as date of event minus date of birth of the individual. Migrants who move to another household of the HDSS study population are assigned a new residence and continue being part of the longitudinal follow-up under their same ID number. On the other hand, migrants who move outside of the HDSS area (e.g., Dakar) are lost to follow-up: they are not tracked, and their relatives are not asked to report their vital status. As a result, it is not known whether they are still alive at any time after their migration. If a migrant returns to the HDSS area after some time outside, s/he is reassigned his/her previous ID number so as to avoid duplication of individuals in the HDSS dataset. The Niakhar HDSS allows researchers to identify sibships (i.e., brothers and sisters having the same biological mother) among individuals who have ever resided in the HDSS population. The identification of sibships is possible because every population member is potentially linked to his/her biological mother through a mother ID number. The mother ID number is attributed either at the time of birth (if the mother gave birth in the HDSS area) or the first time an individual enters the HDSS population (i.e., initial census or after in-migration). We used these data to identify the sibships of potential respondents and measure the rate of omissions of siblings’ deaths in each study group (see below). For some members of the HDSS population, the mother ID number may be missing because their biological mother may never have been a member of the HDSS population or because the information reported during household visits was not sufficient to establish a link between mother and child. Similarly, some sibships may be only partially identified if, for example, some of the siblings were born outside of the HDSS area. We used the standard DHS questionnaire (i.e., the maternal mortality module). It consists of a single name-generating question, asking respondents to list all their maternal siblings (i.e., siblings with the same biological mother), starting from the oldest. Then, for each nominated maternal sibling, respondents are asked (1) whether the sibling is male/female, (2) whether s/he is still alive, (3) if alive, how old s/he currently is, (4) if deceased, how old s/he was when s/he died, and (5) how long ago (in years) did the death occur. For female siblings deceased above age 12 y, the DHS instrument also includes questions about the circumstances of the death, e.g., whether she died during pregnancy, at the time of delivery, or within 2 mo after the end of a pregnancy or childbirth. The SSC includes four major modifications of the DHS questionnaire. First, interviewers were instructed to sensitize respondents to the issue of misreporting prior to beginning the SSH interview. To do so, interviewers used a standardized script stressing the importance of accurate recall. Second, instead of asking respondents to list their maternal siblings by birth order, the SSC asked respondents to list their maternal siblings in the order that they came to mind (“free recall”). This modification was adopted because studies in social psychology indicate that imposing sorting constraints in name-generating questions limits the number of persons reported during surveys about social relationships [25],[26]. Third, the SSC used supplementary interviewing techniques designed to stimulate the recall of potentially omitted siblings. These techniques have been used previously in sexual networks research and in retrospective studies of dietary intake [27]–[31], for example. They included nonspecific prompting, reading back the list of nominated siblings, and recall cues. Nonspecific prompting involves asking respondents whether there are other maternal siblings they may have omitted to list. Interviewers were encouraged to prompt nonspecifically after the initial list of freely recalled siblings was obtained. At that time, interviewers were also instructed to read back the list of nominated siblings to the respondent slowly, starting from the sibling nominated last. This procedure gives survey respondents an additional opportunity to recall their siblings [29]. Recall cues were designed using findings from a validation study of SSH data conducted in Bandafassi, a HDSS site in southeastern Senegal [32]. During this study, we found that maternal siblings were more likely to be omitted during SSH interviews if they were deceased, had migrated away from the community of origin, had a different biological father than the respondent, or had not co-resided with the respondent [19]. We thus developed a set of recall cues that addressed these factors. The interviewer introduced cues to the respondent one at a time, asking him/her whether s/he may have omitted one or more maternal siblings who correspond to a cue. Finally, we adopted an event history calendar approach to collecting data on ages at, and dates of, vital events that affected the siblings of a respondent. Event history calendars are designed to help respondents accurately report the timing of past events [33]–[36]. Such calendars reduce recall error and significantly increase data reliability [36]–[41], but they have not been used in the collection of SSHs. The event history calendar we designed is a large grid (A3 paper) bracketed by years from 1953 to 2013. It is composed of three sections: a landmarks section, a respondent section, and a siblings section. Landmarks are events that are likely to be remembered by most respondents. These include, for example, major political events (e.g., independence of Senegal), natural disasters (e.g., droughts), and sporting events (e.g., Senegal reaching the quarterfinals of the 2002 World Cup). Because SSH data are most frequently collected during national surveys, we included only national events as landmarks, rather than also including more local events. For example, we did not include electrification of a respondent’s village or the death of a local chief in this section, even though these are events that are likely well remembered by study respondents. The respondent section recorded events having affected the respondent him/herself in four key domains: residence, marriages, births, and schooling. The sibling section was used to (1) reorder the list of all nominated siblings by birth order and (2) elicit the date of birth—and possibly date of death—of each sibling. Events recorded in the landmarks and respondent sections aimed at anchoring the reporting of events that affected siblings. Compared to the DHS questionnaire, the structure of the SSC is more flexible and lends itself to repeated probing and crosschecking of answers. The SSC (in French) is available as Questionnaire S1. Both the SSC and the DHS questionnaire included a small number of questions on the socio-demographic characteristics of respondents (e.g., schooling, religion). Study participants were selected among individuals who had ever been registered by the Niakhar HDSS. Individuals were eligible if they were aged 15 to 59 y on 1 January 2013 according to the HDSS dataset and they had at least one known sibling in the HDSS dataset. We excluded members of the HDSS population who had no known siblings or who had a missing/invalid/inconsistent mother ID number in the HDSS dataset. The SSH data these individuals may report during a retrospective SSH survey cannot be validated against the HDSS data. Study participants included individuals who resided in the Niakhar HDSS area at the time of the survey, individuals who were temporarily absent from their residence in the HDSS area, and individuals who had permanently migrated outside of the HDSS area prior to the survey. The latter two groups were traced to their new places of residence following methods used in migration studies conducted in sub-Saharan countries (e.g., [42]). Recruitment was conducted on the basis of household visits. Participants who were still resident of the Niakhar HDSS were visited up to three times by the study team. For participants who had migrated outside of the HDSS area or were temporarily absent due to seasonal migration, we first conducted a short “migration inquiry” with members of their last known HDSS household. We asked a household informant about their destination (region, city/village, neighborhood) and tried to obtain at least one contact phone number for the migrant. The form used to collect this information (in French) is shown in Questionnaire S2. Because of resource constraints, we could not attempt to trace all absent residents and migrants selected for study participation. Some former members of the HDSS population had moved abroad (e.g., The Gambia) or to areas of Senegal that are hard to reach (e.g., Valley of the Senegal River, Casamance). We thus delineated a “tracing area” within which the study team sought to contact absent residents and migrants for enrollment. This tracing area included Dakar and its suburbs, Mbour and surrounding areas, as well as the area within an 80-km radius of Niakhar. If the migrant or absent resident was reported to reside in the tracing area, members of the study team sought to get in touch with him/her by phone and schedule an appointment to conduct the interview in person. The study enrollment process is summarized in Figure 1. First, we identified all individuals who met eligibility criteria among individuals ever registered by the Niakhar HDSS. The selectivity resulting from age- and sibship-related criteria is described in Table S1. Then, we identified sibships in which at least one adult death had occurred over the course of the HDSS and sibships in which there was no known adult death among known siblings. We selected all sibships with adult death(s) for inclusion (n = 592), then we selected an additional 500 sibships at random among sibships in which all siblings registered by the HDSS were still alive. Since adult mortality is a relatively rare event, this stratified sampling was necessary to ensure sufficient reports of adult deaths during the validation study. In total, 1,092 sibships formed our “primary sample.” Among these sibships, we selected one participant to interview at random among eligible members. Among the 592 sibships of the primary sample in which there was at least one adult death among siblings, we sampled an additional participant at random among the remaining eligible individuals. This additional sampling was designed to enable an assessment of the inter-sibling reliability of adult mortality data collected in each study group (“reliability sample”). In total, 488 additional participants were selected through this procedure (Figure 1). The same interviewer could not interview two individuals from the same sibship. Finally, we replaced sampled respondents who had migrated outside of the tracing area by interviewing another randomly selected member of their sibship. If there were no potential replacements in a respondent’s sibship, the respondent was not replaced. Similarly, we replaced sampled respondents who were deceased or incapacitated by another member of their sibship. In total, 150 replacements were thus added to the study sample (Figure 1). Participants selected in the primary sample were randomized 1∶1 to an interview with the DHS questionnaire (DHS group) or the SSC (experimental group). In the primary sample, randomization was stratified by gender, residence at last HDSS visit (in HDSS area or elsewhere), and composition of the respondent’s sibship (at least one adult death versus no adult death among siblings). Randomization was conducted using computer-generated random number sequences in Stata 12. Participants selected as part of the reliability or replacement samples were automatically assigned to the same interview questionnaire as their sibling in the primary sample. As a result of the selection processes in the reliability and replacement samples, the final numbers of participants allocated to each study group were not perfectly equal (e.g., some respondents to be replaced did not have another eligible sibling, see Figure 1). In a number of methodological trials of survey questionnaires or interviewing techniques, control and experimental groups are surveyed concurrently either by the same study team or by different study teams (e.g., [37],[43],[44]). This approach to data collection raises potential contamination concerns: interviewers may be tempted to use the methodological innovations of the experimental questionnaire during control interviews. For example, they may informally incorporate some of the recall cues in control interviews. If separate study teams conduct control and experimental interviews concurrently, (1) contamination may still occur if study teams communicate with each other, and (2) interviewer effects may confound study results if teams are small (<30 interviewers per team). We decided to conduct control and experimental group interviews consecutively with the same team of interviewers. After a short training (see below), interviewers first collected the control group data using the DHS questionnaire. They were then trained in the use of the SSC. Finally, they collected the experimental group data using the SSC. This approach to data collection ensured that control interviews occurred without knowledge of the methodological innovations introduced in the SSC. All study data and outcomes were measured in one visit, there was no follow-up after the SSH interview. Neither the interviewer, nor the study participants were blinded, but interviewers did not have access to prior HDSS information about the sibship of the respondents they interviewed (e.g., whether the respondents belonged to a sibship where adult deaths had occurred). We recruited interviewers with prior experience of DHS data collection in Senegal. We did so for two reasons: first, to ensure that the SSC would be feasible for interviewers with qualification and skills comparable to those of interviewers typically employed during DHS; second, we hypothesized that hiring interviewers with DHS experience would enhance the external validity of our validation study data. We thus contacted the Senegalese National Agency of Statistics and Demography, which implements the DHS in Senegal. We asked officers in charge to recommend ten potential interviewers who had previously worked as interviewers during one of the DHS conducted in the country. After a short selection process (based on an aptitude test and interview), we selected eight interviewers. Seven interviewers had participated in the 2010/2011 Senegal DHS, whereas one interviewer had participated only in the 2005 DHS. All interviewers spoke Wolof, but only one interviewer was fluent in Sereer. Training for each questionnaire lasted 3 d. The study investigators (S.H., A.M.K., L.D., and B.M.) first reviewed each item on the questionnaire with the interviewers, addressing questions and discussing translations of key terms. Then interviewers worked in pairs on a series of role-playing exercises, during which they interviewed each other. Study investigators provided feedback on interviewing techniques and corrected errors. Finally, each interviewer conducted 2–4 practice interviews with inhabitants of the town of Niakhar (outside of the HDSS), who were not part of the study sample. Two of the study interviewers also served as team supervisors. They were asked to (1) track completion of the study sample, (2) check completed questionnaires for errors (e.g., missing data, incoherent ages), and (3) provide feedback to interviewers and study investigators. At least one of the study authors was present in Niakhar at all times during the course of the study. Data collected in each study group were double-entered using Epi Info. The analyses we report here are exploratory, previously unplanned analyses of the validation study data. They focus on (1) metrics that are commonly used by demographers in evaluating the quality of SSH data and (2) simple sibship-level comparisons between the reference HDSS dataset and the SSH data collected using the SSC versus DHS questionnaires. We chose to conduct such analyses because they permit a direct comparison between our validation study results and data quality assessments conducted with data from national DHS [20], hence allowing an assessment of the external validity of study results. In contrast, the planned analyses of the validation study data focus on outcomes measured at the sibling level by record linkage, i.e., matching the report of a particular sibling's survival obtained through SSH to the record of that same sibling's survival in the HDSS dataset. Record linkages permit precise measurement of the proportion of siblings who are added/omitted during SSHs (planned primary outcome), as well as measurement of age and date errors (planned secondary outcomes) in reports of SSH [19]. But record linkages can rarely be implemented outside of a small number of HDSS populations because they require detailed lists of a respondent's siblings as well as sufficient identifying information about each sibling. Analyses of planned study outcomes are still ongoing and will be reported later. All study group comparisons reported below were conducted on an intention-to-treat basis, i.e., with participants included in their randomly assigned study group. During the course of the study, however, five participants assigned to the SSC group were interviewed with a DHS questionnaire by error (see Figure 1). We thus also conducted an as-treated analysis in which we reclassified these five participants as members of the DHS group (i.e., the questionnaire with which they were actually interviewed). The as-treated analysis yielded results similar to those of the intention-to-treat analysis. We first measured the proportions of missing data on current age of live siblings and age at death and date of death of deceased siblings. We tested for differences in the extent of missing data between study groups using χ2 tests of the association between categorical variables. Next, we measured age/date “heaping” in SSH reports, i.e., respondents' propensity to report ages and dates ending in round numbers (e.g., multiples of five or ten). Heaping is an oft-used indicator of deficiencies in demographic data [45]–[48]. We measured age and date heaping in each study group by computing the following heaping ratio for each age/date: where N(a) represents the number of nominated siblings at age/date a. This heaping ratio is an indicator of the excess number of deaths reported to have occurred at a certain date or age. For assessment of the external validity of study results, we also investigated heaping patterns in the Senegal 2010/2011 DHS. We tested whether the SSC was more accurate in recording adult deaths among the maternal siblings of a survey respondent than the DHS questionnaire. We measured accuracy by the concordance—at the sibship level—between SSH data reported in each study group and the Niakhar HDSS dataset. For example, let us consider a respondent who has one adult sister who died in the last 15 y according to the HDSS. The SSH reported by this respondent is said to be “concordant” if the respondent also reported an adult death of one of his sisters during that time frame. The SSH is “discordant” if the respondent did not report any adult death or reported only adult deaths among his/her brothers. On the other hand, among respondents whose known adult sisters were all still alive according to the HDSS, an SSH was deemed “concordant” if the respondent did not report any adult death among his/her sisters. We thus defined the sensitivity of SSH data as the proportion of concordant SSHs among respondents with at least one adult death among their sisters/brothers according to the HDSS dataset. Sensitivity was measured separately by gender of the deceased. We measured the sensitivity of each questionnaire in capturing adult deaths that occurred in the 15 y prior to the survey. We selected this reference period (15 y) because this corresponds to the reference period used in analyses of the global burden of disease relying on SSH data [3]. The specificity of SSH data was defined as the proportion of concordant SSHs among respondents with no adult death among their sisters/brothers in the past 15 y according to the reference HDSS dataset. We measured the specificity of SSH data because we were concerned that the SSC may lead to false reports of deaths among siblings during SSHs. Such false reports may happen because of repeated probing and prompting (e.g., additions of cousins or other relatives of the respondents). We evaluated the amount of time required to complete each questionnaire (in minutes). If the SSC requires significantly more time to complete than the DHS instrument, it may be impractical to adopt in large-scale nationally representative surveys such as the DHS. We tested for differences in the specificity and sensitivity of SSH data between study groups (SSC versus DHS questionnaire). To do so, we used logistic regressions controlling for stratification variables (i.e., gender of the respondent, his/her residence, and sibship composition), as recommended for analysis of stratified randomized trials [49]. Based on the observed values of sensitivity/specificity for each questionnaire, we calculated the true proportion of respondents with at least one adult death among their adult sisters/brothers. We used the formula [50] where is the true proportion of respondents with at least one adult death among their sisters, and is the proportion of respondents reporting at least one adult death during their SSH interview. We let vary from 0% to 50%, and we estimated the extent of bias in estimates of as . We then conducted subgroup analyses in which we tested whether the effects of the SSC on the sensitivity/specificity of SSH data varied across different respondent characteristics. These characteristics included: gender, age (<25 y old, 25–34 y old, 35–44 y old, ≥45 y old), place of interview (in Niakhar HDSS versus in the tracing area), education (no schooling versus primary schooling versus secondary schooling or higher) and religion (Muslim versus Christian). To do so, we used the Mantel-Haenszel test of the homogeneity of odds ratios across strata of a classifying variable [51], as recommended in guidelines for subgroup analyses [52],[53]. In sibships where two siblings were interviewed, we measured agreement in reported SSHs using Cohen's Kappa [54]. Finally, we compared the average duration of interviews between study groups using non-parametric tests of differences in median. All tests of statistical significance were adjusted for the clustering of respondents within sibships. We initially planned to contact 698 respondents, but we managed to enroll close to 1,200 participants because of shorter interviewing times and higher interviewer productivity (i.e., daily number of interviews) than expected. The study was designed to measure medium effects of the SSC on the primary study outcome, i.e., the proportion of adult siblings omitted during SSH interviews assessed through record linkages (see Text S1). Since the study was not designed to measure the indicators we report here (e.g., sensitivity), we conducted tests of the equivalence of sensitivity/specificity indicators between the SSC and DHS questionnaires. Based on the available sample size, this approach allows identifying an equivalence interval (p−Δ; p+Δ) that likely contains the true difference in sensitivity/specificity between SSC and DHS questionnaires. We used the two one-sided tests approach to calculate this interval [55],[56].