The impacts of quality improvement on maternal and newborn health: Preliminary findings from a health system integrated intervention in four Ethiopian regions

Background: Quality improvement (QI) methods are effective in improving healthcare delivery using sustainable, collaborative, and cost-effective approaches. Systems-integrated interventions offer promise in terms of producing sustainable impacts on service quality and coverage, but can also improve important data quality and information systems at scale. Methods: This study assesses the preliminary impacts of a first phase, quasi-experimental, QI health systems intervention on maternal and neonatal health outcomes in four pilot districts in Ethiopia. The intervention identified, trained, and coached QI teams to develop and test change ideas to improve service delivery. We use an interrupted time-series approach to evaluate intervention effects over 32-months. Facility-level outcome indicators included: proportion of mothers receiving four antenatal care visits, skilled delivery, syphilis testing, early postnatal care, proportion of low birth weight infants, and measures of quality delivery of childbirth services. Results: Following the QI health systems intervention, we found a significant increase in the rate of syphilis testing (ß = 2.41, 95% CI = 0.09,4.73). There were also large positive impacts on health worker adherence to safe child birth practices just after birth (ß = 8.22, 95% CI = 5.15, 11.29). However, there were limited detectable impacts on other facility-usage indicators. Findings indicate early promise of systems-integrated QI on the delivery of maternal health services, and increased some service coverage. Conclusions: This study preliminarily demonstrates the feasibility of complex, low-cost, health-worker driven improvement interventions that can be adapted in similar settings around the world, though extended follow up time may be required to detect impacts on service coverage. Policy makers and health system workers should carefully consider what these findings mean for scaling QI approaches in Ethiopia and other similar settings. In partnership with the Federal Ministry of Health, the program established governmental district-wide learning collaboratives and provided them with structured, systematic, QI and relevant clinical skills training. The pilot phase implemented four learning collaboratives (one in each primary hospital catchment area that included all government health facilities (health centers and their corresponding health posts) in its referral network), beginning in September 2016 and ending in September 2018. Each collaborative formed quality improvement teams (QITs) that work with support from government leadership at multiple levels (e.g., woreda, zonal, regional) within each site. Each health center QIT included health extension workers (HEWs) from its linked health posts. Primary hospitals are the first point of contact with physicians and provide care for complications including caesarian section, and blood transfusions. Health centers are nurse/health officer-led and provide primary health care services, including uncomplicated deliveries. Health posts are managed by a health extension worker and provide basic health services at the lowest administrative level. QIT participants included facility heads, maternal and neonatal clinical staff, data officers, and health extension workers. The pilot phase was implemented in one collaborative in 4 agrarian and 1 pastoralist (data collection ongoing) region to represent diversity in the country. These data are from the first four completed collaboratives, including 30 QITs. QITs attended four structured learning sessions over 18 months for training in QI, experience sharing, and peer learning, followed by the implementation of team-initiated QI ‘change ideas’ and troubleshooting. In between the learning sessions, intensive coaching visits were made by project staff to supervise and mentor the QITs (see Fig. 1 for intervention components). The results presented in this paper use data from the pilot phase, including between 9 and 13 months of pre-intervention data per facility, and follow outcomes until December 2018, totaling 878 facility months across all pilot facilities. The QI intervention was considered to reach full implementation between the 2nd and 3rd learning session, the time at which change ideas were developed, tested, and monitored. Additionally, staff knowledge acquisition of QI methods and strategies would not be sufficient to conduct the aforementioned interventions until this timepoint. The Ethiopia Quality Improvement Intervention Components We consider the intervention as having three ‘active ingredients’, including the clinical and QI trainings done at the collaborative start, the change ideas tested by the QITs, and coaching visits provided to support clinical quality and coach QITs. We characterize the main ingredient of the intervention using the change ideas tested within each QIT. QITs developed change ideas targeting these maternal and child health indicators. Multiple change ideas targeting one or more priority areas were defined and tested within each QIT team at their respective facilities. Information for each change idea was systematically documented into a project monitoring database and included the date initiated, the implementation strategy, and specific goals, targets, and timelines as part of QI coaching. This change idea data was extracted and dichotomized so that, if any change was developed and tested for a particular target indicator, the facility was coded as having tested a change in that month. We also created an overall category of any change tested in any category over the intervention period as well as a continuous count of the total changes tested across all categories. Coaching included observing clinical care and supporting health care workers’ clinical skills, motivating QITs, supporting facility leadership in fostering team communication and identifying problems, and supporting developing and monitoring testable solutions to address gaps in care. Daylong coaching visits were made to QITs one to two times per month over the course of the support phase (between 11 and 15 months). Additionally, in order to simultaneously ensure outcome data quality and strengthen the routine data reporting system (the government health management information system (HMIS)), all facility data were extracted from the facility paper registers. This data was validated by comparing and reconciling with HMIS reports as a part of data quality improvement efforts. Outcome data were extracted from facility registers from May 2016 until at least 6 months following the fourth (and last) learning session. In this paper, we present results of the intervention on the following maternal and newborn health outcomes: skilled delivery (the proportion of births attended by a skilled birth attendant); four antenatal care visits (ANC) completed (the proportion of pregnant women who have four ANC visits by 36 weeks of pregnancy); syphilis testing (proportion of ANC users who have been tested for syphilis); neonatal complications (proportion of cases treated for sepsis and asphyxia); early PNC (the proportion of women who receive postnatal care (PNC) within 48 h of delivery); and proportion of low birth weight infants. Infants placed in Kangaroo Mother Care were also examined as an outcome, but this data is only captured at the hospital level, and thus are not included in the longitudinal analyses. Outcome variables were calculated using census-derived population estimates to calculate the denominators of number of expected pregnancies and number of expected live births as per the definitions in HMIS (Commission; 2008). We also explore the extent to which the QI intervention improved compliance to three bundles of essential birth practices for safe childbirth. These included: On-Admission Safety Bundle; Before Pushing Safety Bundle, and Just After Birth Safety Bundle). Bundle components can be found in supplementary Table S.2. These bundles outline essential components to the standard of routine maternal care, and were derived from the WHO Safe Childbirth Checklist which had been adopted by the Ethiopia Federal Ministry of Health and introduced to health care facilities at the time of program initiation [19]. The Checklist was introduced as a job aid for clinical care provision as part of the QI initiative in the first learning session, and implemented in line with similar studies in LMIC [20]. Bundle adherence was assessed through the triangulation of three methods. First, monthly retrospective medical record charts of 30 randomly selected births from a facility were reviewed for documentation of bundle elements; second, senior program officers observed all births that occurred during a coaching visit and checked if each element was conducted; and third, each paper copy of the safety birth checklist were assessed for completeness. Adherence was considered achieved (and coded as ‘1’) if all element of the particular bundle were met for a given birth. If any element of the three bundles was not performed, adherence was not achieved (coded as ‘0’). Facilities kept monthly logs of the proportion of births with 100% adherence to a given bundle. We expect that the type of facility and geography may affect the magnitude, speed, and type of change that is possible following QI changes, particularly given the large amount of facilities included in the intervention. For example, some regions have richer resource chains, more highly skilled or experienced staff, or environments more conducive to change compared with others. To address some of these differences, we control for selected covariates at the facility level, collected from a baseline assessment. These included facility type (health center vs. hospital), its catchment population, the number of staff working within each facility, and the geographic region of each facility. A baseline survey of each health facility assessed the presence or absence of essential pharmacy supplies, medicines, and laboratory testing equipment required to provide minimal acceptable services related to maternal care and child delivery. From each of these identified indicators, we created a ‘medication index’ to reflect this baseline facility quality (see supplementary material) and include this as a covariate in all models. First, we compared the pre and post-intervention means for our outcomes. The quasi-experimental design of the intervention, whereby each facility serves as its own control over time, allows us to leverage an interrupted time series (ITS) approach [21]. This analytic strategy has been employed in public health intervention evaluations with access to systematic longitudinal data [22]. ITS uses a segmented multivariable regression to detect whether the intervention (e.g., the change ideas tested) is associated with a significant trend shift in the outcome variable of interest (e.g., proportion of women receiving postnatal care within 48 h of delivery, etc.). The ‘interruption’ (e.g., the intervention), was considered present after the project met full implementation so that the pre-trend uses approximately 13 months of data per facility and the post-trend uses about 20 months, allowing us to account for seasonality effects. The core equation to be estimated using GEE was: Y is the outcome of interest, t is the time period, f is the facility, and CTf represents pre/post intervention (0 if before full implementation and 1 if after) or a change category tested in facility f. The vector X represents the facility and covariates included in the model (facility type, catchment population, the geographic woreda, and the baseline medicine index). The immediate impact of the ‘interruption’ is indicated by β2 and determines whether there is a one-time jump in the outcome value after full implementation of the intervention or change. β3 indicates the longer term impact or trend, indicating whether there is a change in the slope of Y after full implementation of the intervention or change (the difference in slope from before to after). We examine both of these effects in order to understand the immediate impact of the intervention as well as if this effect was maintained and sustained in the 15–23 months following the intervention’s full implementation. In addition, β2 + β3 yields the overall effect of the change category when time equals 1. We also include a marker of intervention intensity, number of coaching visits, to explore if this ‘ingredient’ had an independent impact, particularly because this dimension of the intervention is adjusted as the program scales nationally. Changes targeting different categories were each modeled in separate multivariable time series regressions. A non-linear trend is accommodated through a quadratic term in t multiplied by the change variable and models also include controls for time and time squared (not shown in the equation).