Evaluation of the ethiopian millennium rural initiative: Impact on mortality and cost-effectiveness

Main Objective: Few studies have examined the long-term, impact of large-scale interventions to strengthen primary care services for women and children in rural, low-income settings. We evaluated the impact of the Ethiopian Millennium Rural Initiative (EMRI), an 18-month systems-based intervention to improve the performance of 30 primary health care units in rural areas of Ethiopia. Methods: We assessed the impact of EMRI on maternal and child survival using The Lives Saved Tool (LiST), Demography (DemProj) and AIDS Impact Model (AIM) tools in Spectrum software, inputting monthly data on 6 indicators 1) antenatal coverage (ANC), 2) skilled birth attendance coverage (SBA), 3) post-natal coverage (PNC), 4) HIV testing during ANC, 5) measles vaccination coverage, and 6) pentavalent 3 vaccination coverages. We calculated a cost-benefit ratio of the EMRI program including lives saved during implementation and lives saved during implementation and 5 year follow-up. Results: A total of 134 lives (all children) were estimated to have been saved due to the EMRI interventions during the 18- month intervention in 30 health centers and their catchment areas, with an estimated additional 852 lives (820 children and 2 adults) saved during the 5-year post-EMRI period. For the 18-month intervention period, EMRI cost $37,313 per life saved ($42,366 per life if evaluation costs are included). Calculated over the 18-month intervention plus 5 years post-intervention, EMRI cost $5,875 per life saved ($6,671 per life if evaluation costs are included). The cost effectiveness of EMRI improves substantially if the performance achieved during the 18 months of the EMRI intervention is sustained for 5 years. Scaling up EMRI to operate for 5 years across the 4 major regions of Ethiopia could save as many as 34,908 lives. Significance: A systems-based approach to improving primary care in low-income settings can have transformational impact on lives saved and be cost-effective. © 2013 Curry et al. All research and consent procedures were approved by the Institutional Review Board at the Yale School of Medicine and the Ethiopian Federal Ministry of Health. Since data was collected in the aggregate from health facilities, no written or verbal informed consent was needed for this study. The Ethiopian Millennium Rural Initiative (EMRI) is an 18-month systems-based intervention to improve the performance of 30 primary health care units (PHCUs) in rural areas of Ethiopia. The initiative was designed using mechanisms that, if successful, could be implemented across all health centers nationally by the Ethiopian Federal Ministry of Health (FMOH). The intervention was implemented by the Clinton Health Access Initiative (CHAI), working collaboratively with the FMOH in 4 of the more populous regions of the country: Amhara, Oromia, Tigray and Southern Nations, Nationalities, and Peoples’ Region (SNNPR). EMRI was rolled out in a series of 3 separate and overlapping phases (10 PHCUs in each of Phases I, II and III). Each phase lasted 18 months, with the exception of 5 PHCUs in Phase III, which received only 15 months of intervention due to delays in program startup. Each PHCU includes a health center (HC) (staffed by approximately 15 health workers including a medical director, nurses, pharmacists, and other health workers), 6 to 7 health posts (HP) (each staffed by 2 health extension workers (HEWs) overseen by a health extension worker supervisor), and a team of approximately 180 volunteer community health workers (VCHWs) who are unpaid but trained and supervised by HEWs to conduct health education and community mobilization activities in the catchment area. Health centers and health posts are owned by the government and overseen by the woreda health office, which reports to the Regional Health Bureau (RHB). The woreda health office may be co-located in the health center or located within an hour of the health center. EMRI addressed the following broad areas of PHCU organization: 1) Physical infrastructure, including water, electricity, condition of buildings, and equipment, 2) Human resources capacity, including training and community health worker engagement, 3) Services, including prevention of mother-to-child transmission of HIV (PMTCT) programs, HIV testing during directly observed therapy for tuberculosis (TB DOTS), HIV testing at the health post level, antenatal care, postnatal care and under-5 services, and 4) Systems, including health care financing, supply chain management, health management information system (HMIS), and referral. The impact of EMRI on lives saved was assessed using monthly data on 6 key indicators: 1) antenatal coverage (ANC), 2) skilled birth attendance coverage (SBA), 3) post-natal coverage (PNC), 4) HIV testing during ANC, 5) measles vaccination coverage, and 6) pentavalent 3 vaccination coverage. We also tracked the impact of the EMRI intervention on basic PHCU infrastructure, including access to water and electricity and adherence to recommended staffing levels per FMOH guidelines (Refer to Appendix S1 for detailed indicator definitions). We tracked PHCU performance monthly on each of these indicators throughout the 18-month intervention period. In addition, we tracked monthly performance for 12 months following the intervention for Phases I and II PHCUs to understand sustained performance levels after the intervention was completed. Data were collected by PHCU staff and corroborated through on-site review of data reports and PHCU registers by the research team every quarter. To assess EMRI impact on maternal and child survival, we used Spectrum Policy Modeling System software to determine the number of deaths averted. The Spectrum Policy Modeling System software (Version 4.43) is a suite of 9 policy models prepared by the USAID Health Policy Initiative in collaboration with the Futures Group International. The Spectrum software has been peer-reviewed and is the software used by UNAIDS to prepare annual reports on national and global HIV/AIDS estimates [20]. Three of the models in the Spectrum software were used in conjunction to estimate impacts on mortality at the population level: 1) Lives Saved Tool (LiST): A model to estimate the impact on child and maternal survival following implementation of programs to increase coverage of various health interventions, 2) Demography (DemProj): A model to estimate population projections and demographic indicators over time used simultaneously with LiST in order to provide inputs for calculations of impact on mortality, and 3) AIDS Impact Model (AIM): A model used simultaneously with LiST to simulate the AIDS epidemic and estimate the number of new infections, people living with HIV/AIDS, and AIDS deaths. The LiST model, which projects changes in maternal and child survival based on changes in the coverage of different child health interventions, requires inputs derived from the data we obtained on the 6 key indicators: ANC, SBA, PNC, HIV testing during ANC, measles vaccination coverage and pentavalent 3 vaccination coverage (Refer to Appendix S2 for a detailed description of the inputs to the model). To generate estimated performance on the key indicators at each point in time, which underpinned the input values for the LiST model, we employed two-part linear regression for the health centers in each Phase of EMRI. Using separate models for each health indicator, we regressed the health outcome (e.g., ANC care, HIV testing) on time in months, using months 0–18 of the intervention. We repeated this process using data from the 12 months after the intervention (i.e., months 19–30) for Phases I and II. We tested for alternative functional forms but found no significant departure from linear trend, so we employed linear regression. We did find the interaction between phase and time to be significant and, therefore, modelled each phase independently rather than combining data from all phases in a single analysis. Rather than modelling continuous performance change over time, the LiST model requires annual estimates of each indicator. Therefore, we used estimates drawn from the middle of each program year to represent average annual performance (6 months for Year 1, 18 months for Year 2). Because EMRI was 18-months, we attributed only half of the Year 2 estimates lives saved to EMRI. In addition, we used estimated performance at 30 months to approximate post-EMRI (i.e., months 18–30) levels of performance of Phase I and II health centers. Because no follow-up data were available for Phase III health centers, post-EMRI performance was estimated using the 18-month level of performance. We calculated the impact of EMRI by program end, as well as the additional impact if health center performance levels were maintained for an additional five years in the current EMRI catchment areas. Because lessons learned from each phase were incorporated into subsequent interventions, we believe that Phase III represents the most evolved EMRI model. Therefore, we also estimated the total lives that could be saved if the EMRI program were scaled nationally and sustained for 5 years at Phase III performance levels. We calculated a cost-benefit ratio of the EMRI program including both the lives saved during the 18-month EMRI intervention and a 5-year follow-up period in the catchment areas of the 30 health centers: Additionally, we calculated the cost-benefit ratio if evaluation funding were included. We estimated the value of the lives saved assuming 40 disability-free years of life for a child and 20 disability-free years of life for an adult. We selected the approach of valuation of years of lives saved using average GDP per capita per year for valuation, a method applied by the World Bank [21], [22], as a conservative approach to estimating the benefits of the intervention. An alternative approach, the Value of Statistical Life methodology, would aggregate individuals’ willingness to pay to reduce the risks of premature death. Although this method is increasingly being applied in middle- and high-income settings, it has not been extensively used in low-income countries, and data needed for such valuation was not available. We monetized the value using the annual Ethiopian gross domestic product (GDP) per capita ($357) per disability-free year saved, and we used a discount factor of 3% to calculate the present value of lives saved [23]. We then applied the World Health Organization benchmarks [24] by which an intervention that costs less per disability-free life year saved than the GDP per capita per year is considered to be cost effective; this approach is highly conservative in assigning benefits to a saved human life as a person’s ability to generate income.