Effects of the EQUIP quasi-experimental study testing a collaborative quality improvement approach for maternal and newborn health care in Tanzania and Uganda

Background: Quality improvement is a recommended strategy to improve implementation levels for evidence-based essential interventions, but experience of and evidence for its effects in low-resource settings are limited. We hypothesised that a systemic and collaborative quality improvement approach covering district, facility and community levels, supported by report cards generated through continuous household and health facility surveys, could improve the implementation levels and have a measurable population-level impact on coverage and quality of essential services. Methods: Collaborative quality improvement teams tested self-identified strategies (change ideas) to support the implementation of essential maternal and newborn interventions recommended by the World Health Organization. In Tanzania and Uganda, we used a plausibility design to compare the changes over time in one intervention district with those in a comparison district in each country. Evaluation included indicators of process, coverage and implementation practice analysed with a difference-of-differences and a time-series approach, using data from independent continuous household and health facility surveys from 2011 to 2014. Primary outcomes for both countries were birth in health facilities, breastfeeding within 1 h after birth, oxytocin administration after birth and knowledge of danger signs for mothers and babies. Interpretation of the results considered contextual factors. Results: The intervention was associated with improvements on one of four primary outcomes. We observed a 26-percentage-point increase (95% CI 25-28%) in the proportion of live births where mothers received uterotonics within 1 min after birth in the intervention compared to the comparison district in Tanzania and an 8-percentage-point increase (95% CI 6-9%) in Uganda. The other primary indicators showed no evidence of improvement. In Tanzania, we saw positive changes for two other outcomes reflecting locally identified improvement topics. The intervention was associated with an increase in preparation of clean birth kits for home deliveries (31 percentage points, 95% CI 2-60%) and an increase in health facility supervision by district staff (14 percentage points, 95% CI 0-28%). Conclusions: The systemic quality improvement approach was associated with improvements of only one of four primary outcomes, as well as two Tanzania-specific secondary outcomes. Reasons for the lack of effects included limited implementation strength as well a relatively short follow-up period in combination with a 1-year recall period for population-based estimates and a limited power of the study to detect changes smaller than 10 percentage points. Trial registration: Pan African Clinical Trials Registry: PACTR201311000681314 We used a quasi-experimental, plausibility design to evaluate EQUIP [29] (Fig. 1) and compared two purposefully selected districts in each of Southern Tanzania and Eastern Uganda; a detailed description of the intervention and study design is presented elsewhere [23, 30]. Briefly, the two pairs of districts were selected from areas where the research team had established research collaborations and (1) the districts were rural, so results would be relevant to other rural districts in the two countries and (2) the districts were of comparable size with similar health infrastructure. However, in Uganda, the government split our comparison district 3 months after our decision was made in December 2010. Of the two new districts, we selected the newly created district Namayingo, as this was the most similar to the intervention district in terms of geographical features (both border Lake Victoria). This meant that the comparison district in Uganda lacked a hospital, had a smaller population than the intervention district and had a less developed health infrastructure (Table 1). Trial design. (asterisk) Estimates per year using the birth rate observed by continuous household survey Population and health system characteristics aWindisch et al. National and district expenditure, p112 (ref [43]) bInformation presents availability of the respective equipment and supply at the day of the health facility survey. 1st round of health facility survey took place from November 2011 to February 2012 and the sixth round from January to April 2014 cRelates to an average spanning over the six rounds of data collection as no variation was observed The intervention districts of Tandahimba and Mayuge mainland and the two comparison districts of Newala and Namayingo, in Tanzania and Uganda, respectively, were predominantly rural. In Tandahimba, Tanzania, 32 public and faith-based health facilities offered maternal and newborn care for a population of 227,514. In Mayuge, Uganda, 33 public and faith-based facilities served a population of 412,500 (a ratio of 1.4 and 0.8 facilities, respectively, per 10,000 population). We did not include private for-profit facilities, as the few facilities operating in the study areas did not provide maternal and childbirth services (Additional file 1: webannex 1 maps). In each country, we compared time trends of coverage and health care quality indicators and mothers’ knowledge of danger signs between intervention and comparison districts using independent continuous facility and household surveys. The plausibility design also included the regular assessment of contextual factors likely to affect maternal and newborn health other than the study intervention, as recommended by Victora et al. [31]. Community members and health staff were not masked to the intervention. The survey team worked independently and was trained to be as neutral as possible but was not masked to the intervention area. However, the survey team was unaware of the primary and secondary outcomes that were chosen for evaluation. We based the QI approach on the collaborative model of improvement [11], which is a short-term, rapid-learning method to improve quality in a focused area based on PDSA cycles and clearly defined and agreed upon indicators for monitoring [32]. We described our methodology in more details in our protocol paper and in the annexes of our protocol paper [23, 33]. In intervention districts, we implemented two strategy components (1) collaborative QI with (a) district health managers, (b) health facility staff and (c) community health workers and (2) continuous household and health facility surveys, with results communicated to district health managers, health facilities and communities using report cards once every 4 months [26]. In comparison districts, we implemented continuous household and health facility surveys for evaluation, with results communicated to district health managers using a written report once per year [30]. For the QI strategy, the main health facility participants were health staff working in the area of maternal and newborn health. Community participants were volunteers, either selected by the community, often on grounds of prior experience as community volunteers (Tanzania) or chosen from active village health teams (in Uganda). The district QI team was composed of the council health management team (Tanzania) and the district health team (Uganda). Every 3 to 4 months, we invited both health facility and community members to participate in learning sessions to review progress. Learning sessions were either joint or separate depending on the chosen improvement topic. These sessions, which typically lasted 1 day, introduced and reminded participants about QI techniques, including the PDSA cycles, and new topics for improvement and also provided a platform to review progress and allow teams to learn from each other. Separate learning sessions were held with district managers (Additional file 1: webannex 2 table). During action periods, which were times between learning sessions when the teams implemented the improvement work, the QI teams were mentored regularly by EQUIP project staff and district managers. In Tanzania, health facility and community QI teams were mentored on average two to three times each quarter, half of the time in the form of “cluster meetings”, where health facility and community QI teams met together locally. Similarly, in Uganda, two to three coaching and mentoring visits for health facility and community QI teams were done per quarter. The learning sessions were supported with feedback from the survey results presented in the form of report cards covering selected indicators (Additional file 1: webannex 5 report cards). The EQUIP team met with district health managers 11 and 12 times over the 30 months of EQUIP implementation in Tanzania and Uganda, respectively. A piloting period took place in both intervention districts from June to October 2011. In Tanzania, full implementation, after the pilot and roll-out periods between June 2011 and February 2012, was achieved in 31 primary health facilities, one hospital and 157 villages (including each roughly 250–500 households each) from March 2012 to April 2014. In Uganda, gradual implementation was done from November 2011 to February 2013, and QI teams were active in 31 health facilities and 72 parishes (each comprising almost 1000 households) from February 2013 to April 2014 (Additional file 1: webannex 3 timeline of assessment and implementation). QI teams selected improvement topics from a QI charter, a planning tool where key areas where improvement is needed are outlined (Additional file 1: webannex 4, improvement charter) which used the WHO guideline of recommended essential interventions, commodities and guidelines as a basis [3]. During the course of the project, prioritisation of which essential intervention to address first were frequently chosen based on discussions between project staff and district mentors and taking into account continuous survey results, which were summarised in report cards. QI teams implemented and tested various change ideas, such as new strategies for birth preparation counselling (e.g. going through birth preparation checklists) or changing implementation routines to address defined problems (e.g. having a delivery tray including oxytocin prepared at all times, see more details Additional file 1: webannex 6 vignettes). The community QI teams implemented a variety of change ideas (Table 2), including home visits to pregnant women, community discussions and the establishment of community savings funds. The improvement topics ‘facility delivery’, ‘uterotonics within one minute’ and ‘post-partum care’ were introduced early on in both countries. In Tanzania, the 1-day training, ‘Helping Babies Breathe’ was offered in March 2013 during one learning session for all intervention facilities preceding the national roll-out in the region. Health facility QI teams used job aids, timely and improved ordering of drugs and supplies, sensitization to and improved counselling of clients and better communication with district managers to improve implementation. In Tanzania, QI teams also started to access funds collected as part of the national cost-sharing strategy and from community health funds, which had been accumulating funds without being used. District QI teams worked on improving (1) the human resource situation, such as requesting new staff or staff to be transferred from the central level, (2) the drug supply from the Medical Stores Department to facilities and (3) the supervision of district managers of primary care facilities and communities to improve quality of care. The change ideas were implemented and tested during a 1–2-month period using locally generated data (Additional file 1: webannex 7 run charts) and widely implemented if internal monitoring suggested improvements. Improvement topics ANC Antenatal care, EDD Expected date of delivery, AMSTL Active management of the third stage of labour, PNC Postnatal care, CHWs Community health workersHFs Health facilities, BCG Bacillus Calmette–Guérin, CME Continuous medical education, QIT Quality improvement team,QITs Quality improvement teams *the scale-up of the topic took place between November 2011 and February 2012, the intervention was implemented up to April 2014 In intervention districts, two external medical and social science experts were employed to facilitate learning sessions and mentoring and coaching. Additionally, in each district, three staff members from the health and community sectors, and a varying number of government-salaried staff working at the lower level of the district, were involved in mentoring and coaching and were remunerated through daily allowances as per government guidelines. In Tanzania, these mentors supported one district and 32 health facility QI teams, as well as about 300 village volunteers organised in 10 cluster QI teams. In Uganda, the mentors supported one district, 30 health facility and 61 parish QI teams (see annex II of [23]). Our primary coverage outcomes were (1) the percentage of women delivering in a health facility and (2) breastfeeding within 1 h after delivery, as assessed through the continuous household survey using reports from women of reproductive age with a live birth in the 12 months before the survey (Table 3). Effects of EQUIP on coverage, quality and knowledge of danger signs aFirst round November 2011 to February 2012 included 111 and 91 women with a live birth in the year prior to the survey in intervention and comparison districts in Tanzania and 238 and 272 in Uganda, respectively. The sixth round January 14–April 14; ~relates to the second round April 2012 to July 2012 included 106 and 101 women with a live birth in the year prior to the survey in intervention and comparison districts in Tanzania and 281 and 199 in Uganda, respectively. bAssessed through multiplying household survey coverage estimates of facility delivery in women with a live birth in the last year prior to the survey to reports of health workers on implementation practices in surveyed facilities using the last event module during the same time period. We included 409 last event questionnaire in Tanzania and 291 in Uganda. cKnowledge of all three critical danger signs in pregnancy (severe vaginal bleeding, oedema of face/hands and blurred vision) and four in newborns (convulsions, difficult breathing, lethargy/unconsciousness and very small baby) dInfection prevention items included clean running water, disinfectant, soap and gloves The primary quality outcome indicator was the proportion of births in which an uterotonic drug was administered within 1 min after delivery. This indicator was constructed by multiplying women’s reports of the place of birth assessed through household surveys with reported health workers practicing on the usage of uterotonics in surveyed facilities. The health worker’s report was based on a last event module that asked staff providers about their practices in a narrative and non-threatening way [30]. As this measurement mode is not validated in low-resource settings, we did observations of delivery practices in selected facilities to validate health worker reports of implementation practices [34]. The primary knowledge indicator was the proportion of women who knew danger signs both in pregnancy and for newborns, as measured amongst mothers who gave birth in the year before the continuous household survey. We used open-ended questions to assess knowledge of danger signs, which was defined as recalling all three critical signs of severe vaginal bleeding, oedema of the face/hands and blurred vision/headache in pregnancy and all four critical signs in newborns (convulsions, difficult breathing, lethargy/unconsciousness and very small baby) [23]. The secondary outcomes were seven indicators that were constructed to reflect the improvement work such as post-partum care and clean birth kits (Tables 2 and ​and33). Assuming a design effect of 1.4 and 10% refusals, the size of the survey was calculated to provide at least 80% power to detect small absolute increases (fewer than 10 percentage points) between the beginning and end of the intervention period for outcomes across the continuum of care, 10-percentage-point increases each year of the intervention for most outcomes (including institutional delivery and immediate breastfeeding) and larger increases more frequently. See Marchant et al. [30] for a detailed discussion. We implemented continuous household cluster and health facility surveys in the intervention and comparison districts. The details are presented elsewhere [30]. Briefly, questionnaires were developed based on well-established sequences of questions as used in Demographic and Health Surveys and Service Provision Assessments [35] and also drew upon earlier work in the study areas [5, 36]. Data collection was organised in 5-month cycles, comprising a 4-month ‘round’ of field work and a 1-month break for aggregated analysis and planning for the next cycle. Based on a sampling frame of the lists of sub-villages with the total number of households (Tanzania) and parish-level lists (Uganda), we sampled, each month, and in each district, 10 clusters (comprised of 300 households) using probability proportional-to-population-size sampling. We thus included no repeated sample of the same women over time but 24 independent probability samples of household clusters to represent each district each month while. We systematically selected each cluster of 30 households from a household list using a fixed fraction (total number of households in the sub-village divided by 30) [30]. After completion of 4 months of data collection, data were aggregated (1200 households and an estimated 152 women with a recent birth, per district) for analysis, and report cards were prepared (Additional file 1: webannex 5 report card). Each 4-month round also included a census of all health facilities to assess readiness and also included a last event module whereby the birth attendant for the last birth recorded in the health facility register was identified and interviewed about the care they had given during that birth. The contextual factors were assessed using pre-defined indicators based on the health system building block framework to inform the plausibility analysis. The selection of indicators was informed by the work of Victora et al. [31]. Systematic investigation into concurrent context changes is needed to draw any conclusion when randomisation is not feasible [29, 31]. The detailed methodology of our context analysis is described elsewhere [23, 33]. In brief, we included indicators of financing, leadership, human resources, drugs and supplies, health information and service delivery [37]. Data were collected using (1) district reports, (2) interviews every fourth month with the district health managers and (3) the continuous household and health facility surveys. A structured questionnaire addressed to the district medical officer and his or her team collected information on the district planning process including participation of civil society, implementation of district plans and other aspects of leadership and governance [23]. Qualitative information on leadership and governance were triangulated with information available in district reports. Analysis used a qualitative and explanatory approach. We used an approach adapted from the analytic methods to model data points over time (see Additional file 1: webannex 8 statistical methods). For each of the six time points of the continuous survey rounds, we calculated the difference in the indicator estimate between the intervention and comparison district [38]. We used meta-regression to fit a regression line through the resulting six data points over the 30 months of data collection and used this regression to estimate the difference-of-difference value between intervention and comparison districts from the baseline (first data collection round) to the end line (last data collection round). Estimates were adjusted for the sampling method (using svy commands in STATA). The delta method was used to estimate the variance of the difference-of-differences measure and present confidence intervals [39]. Rather than using a time-series approach based on autoregressive integrated moving average (ARIMA) models, we chose a simpler and more transparent analytical method that we thought was more appropriate for our small number of data points over time. We did, however, perform a sensitivity analysis with ARIMA models to investigate correlation over time and found no significant impact on the results for our primary outcomes. We did not adjust the models for potential confounding factors. We used information on potential confounders qualitatively [29]. Descriptive tabulation was done for the outcomes selected to present the context. Analysis was done using STATA 13 (StataCorp, Texas, USA). Ethical clearance was obtained from the institutional review boards of Ifakara Health Institute, the Tanzania Commission for Science and Technology, the Uganda National Council of Science and Technology, Makerere University School of Public Health and the London School of Hygiene and Tropical Medicine (LSHTM). This activity underwent human subjects review process at CDC and was approved as not being engaged in human subjects research. Advocacy and sensitization meetings with district and sub-district authorities were held at the start of the project. Communities and health facilities were informed about the survey by a survey team member 1 day prior to the interview, using information sheets in the local languages. Written informed consent to participate in the surveys was obtained from household heads, women, facility in-charge and health staff interviewed.