Impact evaluation of the free maternal healthcare policy on the risk of neonatal and infant deaths in four sub-Saharan African countries: A quasi-experimental design with propensity score Kernel matching and difference in differences analysis

Objective Despite the huge financial investment in the free maternal healthcare policy (FMHCP) by the Governments of Ghana and Burkina Faso, no study has quantified the impact of FMHCP on the relative reduction in neonatal and infant mortality rates using a more rigorous matching procedure with the difference in differences (DID) analysis. This study used several rounds of publicly available population-based complex survey data to determine the impact of FMHCP on neonatal and infant mortality rates in these two countries. Design A quasi-experimental study to evaluate the FMHCP implemented in Burkina Faso and Ghana between 2007 and 2014. Setting Demographic and health surveys and maternal health surveys conducted between 2000 and 2014 in Ghana, Burkina Faso, Nigeria and Zambia. Participants Children born 5 years preceding the survey in Ghana, Burkina Faso, Nigeria and Zambia. Primary outcome measures Neonatal and infant mortality rates. Results The Propensity Score Kernel Matching coupled with DID analysis with modified Poisson showed that the FMHCP was associated with a 45% reduction in the risk of neonatal mortality rate in Ghana and Burkina Faso compared with Nigeria and Zambia (adjusted relative risk (aRR)=0.55, 95% CI: 0.40 to 0.76, p<0.001). In addition, infant mortality rate has reduced significantly in both Ghana and Burkina Faso by approximately 54% after full implementation of FMHCP compared with Nigeria and Zambia (aRR=0.46, 95% CI: 0.36 to 0.59, p<0.001). Conclusion The FMHCP had a significant impact and still remains relevant in achieving Sustainable Development Goal 3 and could provide lessons for other sub-Saharan countries in the design and implementation of a similar policy. The data used in this study were obtained from 11 separate Demographic and Health Surveys (DHS) and 1 Malaria Indicator Survey (MIS). The DHS and MIS are nationally representative cross-sectional surveys which include common questions about the year of birth and survival status of all births to women of reproductive age (15–49 years). The DHS and MIS data sets are freely available and could be downloaded at the DHS website (http://dhsprogram.com) after completing the online data request registration form. With the exception of Burkina Faso that could not provide DHS but MIS data for 2014, each country contributed three different DHS data sets that were conducted between 2000 and 2014. That is, we used the pre-baseline data from 2001/2003 to 2007/2008; baseline data 2007/2008 and end-line data 2013/2014. The unit of analysis in this study is the children of women born in 5 years (0–59 months) preceding the survey. Detailed distribution about number of live births in 5 years preceding the survey, number of women aged 15–49 interviewed, total number of women aged 15–49 in the country at the time of the survey, year of survey and survey response rate for eligible women, NMR and IMR per 1000 live births and cumulative incidence rate per 1000 person-years at risk can be found in online supplementary appendix table S1 A. bmjopen-2019-033356supp001.pdf Patients and the public were not involved. The primary outcomes of interest were IMR and the NMR. In this analysis, the IMR is defined as the probability of dying between birth and first birthday whereas NMR is defined as the probability of dying between birth and the first month of life.13 All deaths that were recorded within the first 28 days after birth were coded as 1 or otherwise 0 in defining a binary indicator variable for neonatal mortality. For infant mortality, deaths within 1 year after birth in the 5 years preceding each survey were coded as 1 otherwise 0 to define a binary indicator variable for infant mortality. Countries that have abolished at least 80% of user fees for institutional delivery in SSA between the periods of 2007 and 2014 and have DHS or MIS data readily available were included in this study as intervention countries. That notwithstanding, these countries should have conducted DHS between the periods of 2000 and 2008. This was necessary to test the parallel trend assumption which is a requirement for the validity of DID design and its estimate. There were only two countries that implemented user fee reforms for maternal healthcare between 2007 and 2008. Ghana and Burkina Faso met these inclusion criteria and therefore were qualified as intervention countries. Although Zambia and Nigeria conducted DHS between 2000 and 2014, both countries did not have a universal exemption on user fees for institutional births during the study period and therefore were qualified to be used in the comparison groups. A similar study based on quasi-experimental design has provided a detailed explanation as to why Zambia, Cameroon and Nigeria could represent a valid comparison group compared with other countries in SSA in evaluating the impact of FMHCP on intermediate-term and long-term health outcomes.11 Cameroon was excluded as a comparison country in this study because there was no survey conducted in 2007/2008 which represents the full policy implementation year. The choice of the selected covariates in assessing risk factors of child survival was based on the analytical framework for the study of child survival in developing countries by Mosley and Chen.14 Specifically, we extracted data and performed the estimation of the propensity scores by using the following variables: household ownership of bednets, child’s age and gender, mother’s age at the time of the survey, mother’s education level, household wealth, sex of the household head, urban or rural area of the household, birth order, multiple births and household size and household access to improved water and sanitation. We defined a household as having access to an improved water source if it has any of the following: piped water into the dwelling, yard or plot; public tap or standpipe, tube well or borehole; a protected dug well or protected spring; rainwater or bottled water. There is a direct correlation between access to an improved water source and infant survival.15 This analysis defines a household as having an improved sanitation if it has any of the following types of toilet facilities, and if this facility is not shared with another household: a flush or pour flush to piped sewer system, septic tank or pit latrine; a ventilated improved pit latrine; a pit latrine with a slab or a composting toilet. There is an inverse relationship between access to improved sanitation and infant mortality. Increasing access to improved sanitation is associated with lower levels of infant mortality.15 The estimation of the propensity scores was based on the binary logistic regression model that adjusted for the complex survey design structure of the data set (weighting, stratification and clustering). Since the study pooled data from different surveys, the women’s standard weights were denormalised. This was achieved by dividing the women’s standard weight by the women survey sampling fraction, that is, the ratio of the total number of women aged 15–49 interviewed in the survey year over the total number of women aged 15–49 in the country at the time of the survey. The total number of women aged 15–49 interviewed in the survey year was obtained from the DHS data sets, while the total number of women aged 15–49 years in the country at the time of the survey was obtained from our world in data (https://ourworldindata.org/). Complex survey design characteristics (weighting, stratification and clustering) were adjusted in all the analyses. In particular, we used the sampling weights in the estimation of the propensity score model and also used the sampling weight times the Kernel weight obtained from the repeated cross-section as the weight variable in the final outcome analysis. This analytic technique has been shown to produce unbiased treatment effect estimates that are generalisable to the original survey target population.16 The Kernel function used in the weight estimation was Epanechnikov and the bandwidth selection was based on cross-validation of the means of covariates.17 To determine the impact of the policy on NMR and IMR, we performed a Propensity Score Kernel Matching with DID analysis using a modified Poisson regression model with robust standard errors. We estimated the average treatment effect (ATE) using propensity scores with Kernel matching adjustment and inverse probability of treatment weighting (IPTW). The data for this study originated from multistage complex surveys and to assess the impact of the intervention, there is a need to replicate random assignment. In experimental study design with random assignment, treatment groups (countries with FMHCP) and control groups (countries with no such policy) are similar on all background characteristics (observed and unobserved) as a consequence of the randomisation, allowing for straightforward comparison of outcomes. In contrast, in complex surveys, the intervention and comparison individuals may differ significantly on background characteristics. Thus, any difference in outcomes (NMR and IMR) between the two groups may be due to these background covariates or to the intervention itself. Matching procedures, followed by regression adjustment on the matched sample, can often be a stronger approach for estimating causal effects than regression on an unmatched sample.18 The DID design is a known quasi-experimental method that is used frequently in policy evaluations to compare changes over time in a group unaffected by the policy intervention (comparison countries) with the changes over time in a group affected by the policy intervention (intervention countries) and attributes the ‘DID’ to the effect of the policy.19 Several sensitivity analyses were conducted to determine the robustness of our results. We tested whether the policy impact estimate is robust to the type of model specification using logit, probit and Cox proportional hazard models with robust standard errors. For the Cox model, the time-to-death with survival status as a censoring indicator was modelled. Finally, we tested whether the impact estimate is robust to different weighting procedures. First, we employed IPTW given by wi=Tiei+1-Ti1-ei where ek is the estimated propensity score for individual k and Ti is the treatment status indicator variable. The IPTW serves to weigh both the treated and control groups up to the full sample, in the same way, that surveys sampling weights weigh a sample up to a population.20 We also applied weighting by the odds to estimate the ATE on the treated (ATT) given by wi=Ti+(1-Ti)ei1-ei. The DID design relies on the parallel trend assumption. This assumption stated that in the absence of the intervention (FMHCP), there would be no statistically significant difference in the trend of NMR and IMR between the intervention and the comparison countries. We relied on DHS data conducted between the years 2000 and 2008 to test this assumption. P values less than 0.05 were considered as statistically significant. Data cleaning and analysis were conducted using Stata V.15 (StataCorp).