The effect of performance-based financing on illness, care-seeking and treatment among children: An impact evaluation in Rwanda

Background: Performance-based financing (PBF) strategies are promoted as a supply-side, results-based financing mechanism to improve primary health care. This study estimated the effects of Rwanda’s PBF program on less-incentivized child health services and examined the differential program impact by household poverty. Methods: Districts were allocated to intervention and comparison for PBF implementation in Rwanda. Using Demographic Health Survey data from 2005 to 2007-08, a community-level panel dataset of 5781 children less than 5 years of age from intervention and comparison districts was created. The impacts of PBF on reported childhood illness, facility care-seeking, and treatment received were estimated using a difference-in-differences model with community fixed effects. An interaction term between poverty and the program was estimated to identify the differential effect of PBF among children from poorer families. Results: There was no measurable difference in estimated probability of reporting illness with diarrhea, fever or acute respiratory infections between the intervention and comparison groups. Seeking care at a facility for these illnesses increased over time, however no differential effect by PBF was seen. The estimated effect of PBF on receipt of treatment for poor children is 45 percentage points higher (p∈=∈0.047) compared to the non-poor children seeking care for diarrhea or fever. Conclusions: PBF, a supply-side incentive program, improved the quality of treatment received by poor children conditional on patients seeking care, but it did not impact the propensity to seek care. These findings provide additional evidence that PBF incentivizes the critical role staff play in assuring quality services, but does little to influence consumer demand for these services. Efforts to improve child health need to address both supply and demand, with additional attention to barriers due to poverty if equity in service use is a concern. Rwanda’s PBF incentives targeting prenatal care, facility delivery, immunizations, and growth monitoring were designed to provide the best start in life for infants and ensure their healthy development. Using population-based survey data, we first tested the hypothesis that PBF reduced the prevalence of childhood illness and examined whether the effect of PBF on illness was uniform across wealth status at a population level. Our second analysis tested the hypothesis that the implementation of PBF increased the probability of a caregiver seeking facility-based curative care services for a sick child. Lastly, among those who sought curative care, we tested the hypothesis that the probability of receiving medications and/or rehydration therapy was positively influenced by adoption of PBF. In 2005, the Government of Rwanda adopted a national PBF program with a phased implementation plan, facilitating an evaluation by the government in partnership with the World Bank [8]. Under the direction of the original program evaluators and concomitant with a Government decentralization and redistricting process, geographic areas nationwide were grouped on population density, rainfall and livelihood. Eight groups were created with on average 2–4 districts each; ten districts, among them the three districts surrounding Kigali, were excluded from this process due to previous PBF piloting. Districts within these eight groups were randomly allocated to early implementation between January 2006 and November 2007 (intervention) and to delayed implementation (comparison) beginning in April 2008 [8]. However, following this initial allocation, some facilities with prior PBF experience were found in the comparison group. For programmatic reasons the government requested uniform scale-up of PBF across a district, such that if a facility within a comparison district was already implementing PBF then the entire district should implement PBF. Thus the initial random allocation to early and delayed implementation had to be modified. Based on suspected early exposure to PBF, five districts with minimal exposure were reassigned from comparison to intervention group. One district was excluded from the evaluation due to extensive exposure. In summary nationwide, PBF was scaled-up in 12 early implementation districts, seven districts were allocated to late implementation, and 11 were excluded due to previous pilot work. Health facility catchment areas mapped closely to the new administrative districts such that when an intervention district adopted PBF, the district population theoretically gained access to intervention sites. This design allowed for comparisons over time between the early implementers or intervention districts and delayed implementers or comparison districts. National household survey data, collected independently from the PBF intervention, provided pre- and post-implementation measures for selected child health outcomes. Data from the 2005 Rwanda DHS (henceforth 2005) and the 2007–08 Rwanda Interim DHS (henceforth 2008) provided individual and household socio-demographic characteristics and health indicators for child health, including reported illness followed by care-seeking and treatment received for reported illness. The 2005 survey used a multi-stage national sampling frame and selected 462 primary sampling units (PSUs) [14] based on census enumeration areas, with field work completed from February 2005 through July 2005. A subset of 250 of these DHS 2005 PSUs were resampled for the 2008 DHS from December 2007 through April 2008 [15]. Geographic coordinates were available for 246 PSUs, facilitating the creation of a panel dataset of matched PSUs from 2005 and 2008. The 11 PBF pilot districts were excluded from the analysis. Longitudinal data from a total of 150 PSUs were thus used in the analysis, with 86 PSUs from the 12 intervention districts and 64 PSUs from the seven comparison districts. The panel dataset included 5781 children less than 5 years of age at the time of each survey who lived in either an intervention (3307) or comparison district (2474). Slightly over half of the children were from the 2008 survey, 3157 (54.6 %). Three primary outcomes were studied: prevalence of childhood illness, care-seeking at a health facility for reported illness, and treatment received among those who sought care at a facility. In this analysis, care-seeking was reported as success in actually being seen by a provider at a facility, therefore it does not include those who may have tried to see a provider at a facility and failed. Receipt of treatment among those who sought care was defined as receiving some medication for the condition. Data for reported cases of diarrhea, fever, and symptoms of acute respiratory infections (ARI), care sought for these episodes, and treatment received were collected in 2008; treatment information for ARI was not collected in 2005. Across both survey years, illness with diarrhea, fever, or ARI in the preceding two weeks was reported for fewer than 30 % of children; subsequent care-seeking was sought for fewer than 40 % of the ill children, effectively reducing the sample for the analysis of whether treatment was obtained. To maximize the data available, reported illnesses were combined as described below. In the DHS, caregivers were asked if any child in the home was ill with diarrhea, fever, and/or a cough with short, rapid breathing (symptoms of ARI) in the previous two weeks. Responses were combined into two dichotomous illness variables: illness with diarrhea, fever and/or ARI; and illness with diarrhea and/or fever, excluding ARI. This allowed the creation of data subsets for those ill including ARI (n = 2073) and those ill excluding ARI (n = 1742). Questions regarding treatment received were asked only of the latter group in both survey years. Caregivers who reported a sick child were subsequently asked whether advice or treatment was sought from any source. All those who reported seeking advice from a public or private hospital, health center, clinic, or health post were coded as having sought care at a health facility. Among those seeking care for diarrhea and/or fever, a series of follow-up questions were also asked to identify any treatment or medications administered, either at home or a facility. A dichotomous variable for treatment received was constructed to indicate whether (a) a child with diarrhea received oral rehydration salts, was recommended home fluids, increased fluids, and/or antibiotics; or (b) if a child sick with fever received a fever reducer and/or an antimalarial. All other responses were coded a not having received treatment. The key independent variables were residence in a PBF intervention district and household wealth quintile. Assignment to the PBF intervention group was based on the district in which the survey PSU was located; hence all children from the same PSU were assigned identical PBF status. Household wealth scores based on asset ownership and housing characteristics were created separately for households in the 2005 and 2008 study samples. Polychoric principal component analysis (PCA) was used to calculate a wealth score that maximized the contribution of binary and categorical variables [16]. The choice of assets for the wealth score was based on the economic context in Rwanda and data availability. Assets for 2005 included television, radio, telephone, bicycle, and land ownership; housing characteristics included electricity, drinking water, toilet facility, cooking fuel, and flooring material. Three assets were excluded due to perfect prediction with other assets: refrigerator, motorcycle, and car. For 2008, land ownership data was not collected, car and motorcycle ownership were combined as a single variable, and refrigerator was excluded, again for reasons of perfect prediction. The first component of the polychoric PCA was used to create the wealth index score, explaining 59 % of the variance for 2005 and 57 % for 2008. Households were divided into quintiles based on their wealth index score; the wealth quintile was assigned to each child living in the household. A difference-in-differences (DID) estimation strategy was used to evaluate the program effect of PBF on the three primary outcomes: probability of childhood illness, facility care-seeking and treatment received. The DID strategy estimates the change in outcome for the intervention and comparison groups over time and takes the difference between the two trends to determine the average effect of PBF, written as: A linear probability model with individual, maternal, and household covariates included to reduce residual variance and improve efficiency was used for estimation. Community fixed effects were subsequently included to control for time-invariant unobserved community differences. The DID with community fixed effects specification is written as: where subscripted indexes were defined as i = individual, j = community or PSU, and t = time (2005 or 2008). Terms in the model include the vector of covariates (X), a dummy variable for time period 2007–08 (Y08 = post-implementation), and a dummy program variable for PSUs located in districts with performance based financing (PBF = 1 for intervention district, 0 for comparison). The primary coefficient of interest was β3, which captured the effect of the PBF program on the outcome Y. By subtracting the differences over time between program and non-program areas, the unobserved time-invariant community fixed effects (μj) were differenced out. To identify heterogeneous effects by poverty, the wealth quintiles were collapsed into a dichotomous variable with the two poorest quintiles groups together as poor and the two upper quintiles grouped as non-poor. Children from the middle wealth quintile were dropped from the model. Adding an interaction term between year, PBF, and poverty to the model allows one to examine the change in probability of an outcome across three dimensions: i) change in outcome over time, between 2005 and 2008; ii) change in outcome between the intervention and comparison groups; and iii) change in outcome between the poorest children and the least poor children. The model specification is shown below. where subscripted indexes were defined as i = individual, j = community, and t = time. A binary variable for poverty was added (POV = 1 for the poor; =0 for the non-poor). The primary coefficient of interest is the triple interaction term (β7), which captures the difference in the impact of the program between the poor and the non-poor. Due to concurrent scale-up of a national community based insurance program, interaction terms between insurance status and PBF residence, and insurance with wealth quintiles were also tested. For each outcome a basic linear probability model (LPM) and an LPM with community fixed effects were estimated. Robust standard errors were clustered at the district level where treatment was assigned. The LPMs with fixed effects were used to calculate the DID and DID with and without the poverty interaction. The full basic and fixed effects models are presented in appendices A and B [see Additional file 1]. Lastly, the fixed effects models were run with and without “choice” variables (insurance, prior facility delivery) to identify potential bias in estimates due to inclusion of these variables that may arguably introduce endogeneity to the model. No significant or substantial differences in program effect were found with or without these choice variables. The study, based exclusively on secondary analyses of publicly-available data, was reviewed and deemed exempt by the University of North Carolina (UNC) Institutional Review Board. All analyses were completed in Stata SE 11.2 (StataCorp, College Station, TX).