Service delivery reform for maternal and newborn health in Kakamega County, Kenya: study protocol for a prospective impact evaluation and implementation science study

Background: Maternal and neonatal mortality remain elevated in low and middle income countries, and progress is slower than needed to achieve the Sustainable Development Goals. Existing strategies appear to be insufficient. One proposed alternative strategy, Service Delivery Redesign for Maternal and Neonatal Health (SDR), centers on strengthening higher level health facilities to provide rapid, definitive care in case of delivery and post-natal complications, and then promoting delivery in these hospitals, rather than in primary care facilities. However to date, SDR has not been piloted or evaluated. Methods: We will use a prospective, non-randomized stepped-wedge design to evaluate the effectiveness and implementation of Service Delivery Redesign for Maternal and Neonatal Health in Kakamega County, Kenya. Discussion: This protocol describes a hybrid effectiveness/implementation evaluation study with an adaptive design. The impact evaluation (“effectiveness”) study focuses on maternal and newborn health outcomes, and will be accompanied by an implementation evaluation focused on program reach, adoption, and fidelity. Kakamega County is located in Kenya’s Western Region, approximately 50 km from the Lake Victoria port of Kisumu. With a population of approximately 1.9 million, the county’s maternal mortality rate was 316 per 100,000 live births and the neonatal mortality rate was 19 per 1,000 live births in 2014, when the most recent data is available [12]. There were approximately 70,000 births in the county in 2018; 35% of which occurred at home, 28% in primary care facilities (dispensaries (Level 2) and health centers (Level 3)) and 37% in hospitals (Level 4 and 5 facilities) [13]. There are 205 government health facilities in the county, 10 of which are Level 4 hospitals. Maternal and newborn care is free in most public health facilities throughout Kenya via the “Linda Mama” program; additionally, Kakamega County introduced a conditional cash transfer program (“OparanyaCare”) for antenatal, delivery, postnatal care and immunizations to poor and vulnerable women. SDR for maternal and newborn health is a multi-component health system reform. In Kakamega, SDR is being implemented by the County Department of Health, with support from the non-governmental organization (NGO) Jacaranda Health. Financing is from the County government and is supplemented by the Bill and Melinda Gates Foundation. An initial feasibility assessment study examined the existing capacity, facility accessibility, political will and potential benefits of SDR in Kakamega County [13]. Based on this assessment, the County identified several components for a health system reform. First, it selected ten hospitals to serve as delivery hubs across the county. These were all public facilities; typically sub-county hospitals (level 4 hospitals in the Kenyan system). Before the intervention, the delivery hubs all conducted 1,000 or more deliveries per year, but not all had comprehensive emergency capabilities. The first component of SDR is a preparation phase, in which delivery hubs are strengthened (Fig. 1) and prepared for the higher patient volumes that will result from the SDR policy change of encouraging delivery at delivery hubs. This includes deployment of additional staffing, as well as improvements to infrastructure and equipment. For example, designated delivery hub hospitals in Kakamega are scheduled to be staffed with additional nurses, doctors, and specialists: In addition to their pre-existing complement of medical officers, the current plan is for each delivery hub in Kakamega to have a surgeon, a pediatrician, an obstetrician-gynecologist, and a specialist medical officer posted to the facility to bolster delivery care. Service Delivery Redesign Theory of Change On the infrastructure side, each delivery hub will have operating theaters built or improved, maternity wards expanded, and newborn units built or expanded. Facility capacity to deliver blood transfusions will also be augmented. Delivery hubs are further from most women’s homes than local clinics; traveling greater distances to deliver can be difficult, especially at night. In Kakamega’s SDR reform, this issue will be addressed with the participation of the NGO Rescue.co, who contracts with private transport providers to provide emergency transport for pregnant women. When delivery hubs are upgraded to a level that will enable higher patient volumes and improved quality of care, and complementary interventions such as emergency transport and facility financing reforms are in place, the government of Kakamega plans to communicate to all health workers, and all health system users, that policy regarding place of delivery has changed, and women are now expected to deliver at designated delivery hub hospitals. To this end, the last component of the intervention is this public communication element, which will be undertaken through media campaigns and public outreach meetings, as well as direct communication from health workers to antenatal care (ANC) clients. The main research question of the impact evaluation (effectiveness) component of this study is the impact of SDR in Kakamega County on maternal and newborn health, measured using a composite indicator of maternal and newborn health. The secondary research question examines the impact of SDR on patient satisfaction and perceived quality of care, maternal complications, out-of-pocket (OOP) expenditures, and utilization of care. The main research objective for the implementation component of the evaluation will be to measure the implementation fidelity and reach, identify key obstacles to successful implementation as well as facilitating factors, and measure possible unintended consequences of SDR policy on health system functioning. This protocol describes two study designs for the impact evaluation component of the SDR evaluation: the first is a multi-phase stepped wedge design, for the planned situation in which SDR is progressively scaled up across all of Kakamega County’s sub-counties in three phases. However, due to the multi-year, complex nature of the SDR intervention, we also pre-specify a smaller scale, nested “phase 1” study design which will be carried out for two reasons. First, a study based on phase 1 will be conducted to track a set of implementation indicators in order to assess policy implementation. As a pragmatic, real world health system evaluation, findings about implementation and take-up from this first phase can be used strengthen phase 2 and phase 3 implementation. Second, in the event that only the first phase of the reform is fully implemented, the full set of outcomes will be analyzed after phase 1 using a difference-in-difference design. In the sections below we first discuss the full SDR reform impact evaluation (“effectiveness evaluation”), and then discuss the phase 1 sub-study. Then, we discuss the implementation science component of the study. This study will use a prospective, observational stepped wedge design. Kakamega County is planning to implement a phased rollout of SDR by sub-county, in three phases. Kakamega has 12 sub-counties: the reform will be implemented first (starting in 2022) in a group of 3 sub-counties. After a period of implementation in these “phase 1” sub-counties, a second set of 4 sub-counties will implement the reform. After another period of implementation in these phase 2 sub-counties, a final set of 5 sub-counties will implement the reform (Fig. 2). This phased approach parallels the structure of a stepped wedge trial in that delivery and post-delivery outcomes for mothers and newborns can be compared within each group of sub-counties that implement SDR in the same implementation phase before and after the roll-out, as well as across counties in the same phase of rollout. Analytically, the analysis is also similar to that of a stepped wedge cluster randomized trial; the difference is that the order in which the clusters are exposed to treatment was decided not via randomization, but was decided by Kakamega County policymakers. Phases of Service Delivery Redesign implementation by sub-county in Kakamega County (Source: authors) While noting the planned three phase design, we also have included an adaptive element to the evaluation study, due to the complex nature of the intervention: SDR is a multi-element, health systems intervention, requiring significant investment in infrastructure, human resources, and equipment. Furthermore, much of the investment is being executed using the county’s public budget (in addition to contributions from the Gates Foundation). Given the public component of SDR spending and implementation through public sector institutions, the research team cannot fully anticipate the timing of the various elements of the stepped wedge design. Therefore in this trial we specify a secondary, interim design which leverages the first phase of SDR, described further below. Embedded within the planned stepped wedge design is a difference-in-difference interim study design with the intervention occurring in phase 1 sub-counties (equivalent to the first “step” or wedge in the planned stepped wedge sequence). This multi-stage design will enable the research team to evaluate interim results of the reform after phase 1. The design will also enable a full evaluation to be conducted even in the event that only one phase of the policy is implemented. In the event that the reform is only implemented in phase 1 subcounties, the research team will conduct an interim study after phase 1 implementation is completed, while also conducting a difference-in-difference study comparing outcomes in phase 1 subcounties to phases 2 and 3 subcounties over the full two year evaluation period. The research team has begun to enroll study participants from a sample of 72 facilities which provide ANC across all 12 sub-counties of Kakamega (enrollment started in February 2022) (Fig. 3). The selected facilities were stratified by sub-county and randomly selected proportional to volume of antenatal care visits, excluding facilities located within Kakamega Town (defined as three most densely populated wards in the vicinity of Kakamega Town). Sampled facilities were divided evenly between those in the three Phase 1 intervention sub-counties (36 facilities) and the other nine sub-counties (36 facilities). All private facilities were excluded, as the main SDR intervention components are not planned to take place in these facilities. Finally, facilities with the extremely low ANC volumes (< 6 new ANC clients per month) were excluded from the sample. These represent the lowest quartile of facilities by ANC volume in the county. Participant flow diagram In each sampled facility, survey firm staff working with the research team seeks to enroll all pregnant women who present for antenatal care over a two year period (6 months of baseline and 18 months of implementation).1 Based on administrative data from 2021, we estimate that this sampling strategy will yield a sample of 59,753 mothers across the 72 facilities over two years. Approximately 98 percent of pregnant women in Kakamega County receive at least one ANC visit (National Bureau of Statistics et al. 2015). The approximate annual number of births in Kakamega County is 61,442, so over the two years of data collection approximately half of all pregnant women will be enrolled in the study. Any woman who reports that she intends to give birth outside at Kakamega County at the time of enrollment will be excluded. In addition, women who state upon enrollment that they have already made firm plans to deliver at a referral facility due to complications or previous caesarean sections will be excluded. Finally, all pregnant women below 15 years of age will also be excluded. The data collection strategy is informed by the SDR theory of change (Fig. 1), which has multiple pathways. According to the reform logic, in sub-countries where SDR has been implemented, pregnant women will change their care-seeking behavior by shifting their place of delivery from home or primary care clinics to delivery hub hospitals. The extent of this shift in care-seeking behavior will be measured by the main survey of enrolled pregnant women, hereafter referred to as the pregnancy registry. A second component of the theory of change is that delivery hubs, having been upgraded with better equipment, infrastructure, and staff, will deliver high quality intrapartum, postpartum, and newborn care. For the subset of women and newborns with life-threatening conditions (obstructed labor, sepsis, postpartum hemorrhage, low birthweight, birth asphyxia), timely and appropriate treatment will save lives, resulting in less maternal and newborn mortality and morbidity. We will measure both elements of this theory of change. Place of delivery and ultimate health outcomes will be measured through the pregnancy registry. Quality of care provided will be measured through facility and provider surveys, direct observations of deliveries, and women’s reports of quality from a subset of the main survey. The main data collection tool to measure the primary outcomes will be the pregnancy registry survey, which includes both enrollment at ANC and post-delivery follow-up surveys. All eligible, consenting pregnant women presenting for ANC in the 72 participating facilities (except the excluded categories mentioned above) will be enrolled.2 At this visit, basic demographic information, anticipated delivery date, preferred and previous delivery locations, self-reported and formally diagnosed antenatal complications, and detailed contact and tracking information will be collected. Participants will be contacted by phone 7 days after their anticipated delivery date and again 28 days after their expected delivery date. In these interviews, details regarding maternal health and child survival and mother’s delivery experience will be collected, as well as mothers’ receipt of post-natal care, satisfaction with delivery care, and OOP costs for delivery including transportation costs. In-person follow up visits will be conducted (using tracking information collected at enrollment) if respondents cannot be reached by phone. A subset of 2,000 respondents from the pregnancy registry will be invited to take part in a more detailed follow up with multiple interviews over the course of their pregnancy and immediate postpartum period. Interviews for this survey (known as the longitudinal survey) will take place in a random subset (40 out of 72) of sampled facilities. Approximately 3% of all enrolled women will be randomly sampled within the pregnancy registry at enrollment for participation in the longitudinal survey. This survey will be conducted, in person, at the antenatal care facility at enrollment; over the phone during the month 8 of pregnancy, 7 days after her anticipated delivery date, and 28 days after delivery; and again in person at the participant’s home 2 months after delivery. These surveys will allow deeper investigation of topics including: previous pregnancies, quality of antenatal, delivery and post-natal care, delivery location intention at various points in pregnancy and reasons for same, travel modality and time to delivery hubs, socioeconomic status, awareness of SDR, and opinions about the health system. Basic anthropometrics and biomarkers (height/weight, blood glucose, hemoglobin and blood pressure) and will also be collected at enrollment and at the final follow up. For newborns, anthropometrics will be captured at the final follow up visit. Facility audits and provider assessments will be conducted in planned delivery hubs and all other level 4 and level 5 hospitals (n = 14) four times over the study period: once six months before the start of phase 1, once at the start of each of the three waves of SDR implementation. These surveys will measure capacity for deliveries, structural quality, and process quality of care of each delivery hub before and after re-design is implemented. For example, the survey will measure the number of maternity beds, the presence of a newborn unit, the number and functionality of operating theaters, presence of staff (including all medical doctors by specialty, as well as clinical officers, nurse-midwives, nurses, and other cadres), the availability of 24 h care, the presence of both basic and comprehensive emergency obstetric signal functions including procedures, equipment, and medicines. These surveys will also measure provider knowledge and attitudes both before and after the redesign, with particular focus on the management of common obstetric complications, and will also assess working conditions, motivation, and awareness of SDR implementation. All providers who are working in the maternity or newborn care units the day of the facility assessment will be invited to participate in the survey. Enumerators who are trained in doing deliveries (i.e. nurses or midwives) will conduct continuous observations of deliveries for one week in each delivery hub (n = 10) at four times over the study. Observations will be used to assess the technical and interpersonal quality of care from the time women are admitted to the labor ward until approximately 6 h after her delivery or her discharge from the facility. Observations will be conducted 6 months before the start of phase 1, and once at the start of each of the three waves of SDR implementation. Given current volumes in delivery hubs from administrative data, an estimated 1,500 deliveries will be observed over the course of the study. Facility and provider surveys will also be conducted in a random sample of 40 primary care (i.e. non-delivery hub) facilities will take place twice: during the baseline period and at the end of phase 1. The purpose will be to examine the capacity and readiness of antenatal care providers and facilities, their skills, job satisfaction, and the resources, drugs, and equipment that they have to do their work, and how this changes over the initial period of SDR implementation. All providers that provide antenatal, postnatal, or maternity care who are working the day of the facility assessment will be invited to participate. The primary outcome of this study is a composite measure of maternal and neonatal mortality. This composite was selected, following similar trials (e.g. Semrau et al. 2017) because SDR is hypothesized to improve both maternal and neonatal outcomes, and because the individual components of this measure would likely not occur frequently enough to allow us to reliably detect changes within the population of Kakamega County. Compared to the composite used in similar trials [7], we omit the “maternal near miss” element from the composite outcomes, due to the difficulty is accurately assessing near miss based on retrospective self report [14]. We estimate a baseline prevalence of the composite indicator of 4% in Kakamega County. This composite measure reflects severe maternal, fetal, and neonatal adverse events that occur around the time of childbirth relative to the total number of birth events. For this study, a birth event is defined as the period from the time of maternal admission for childbirth (or the onset of labor, for home deliveries) until the completion of 28 days after birth. A birth event considers the mother and baby to be a single dyad given that their health outcomes are many times interdependent. For this reason, multiple gestations will also be considered to be a single birth event. A birth event may result in a stillbirth and/or a live birth. Specific outcomes comprising the composite measure are: The composite outcome will be assessed through the two follow up phone surveys from the pregnancy registry, and in person surveys when respondents cannot be reached by phone. Secondary outcomes of the study include patients’ satisfaction with care received and confidence in the health system, the location of care (health center versus hospital), OOP expenditures, quality of delivery care, travel time, self-reported maternal complications, and equity on all of these dimensions between women. Women’s satisfaction with the care that they receive will be measured in the pregnancy registry, and with more detail, in the longitudinal survey. Total OOP expenditures will be measured using seven questions capturing expenditures on transportation, overnight lodging, medical fees, registration, drugs and supplies, medical tests, food, and any other costs. These will be summed to create an OOP expenditures indicator. Quality of care will be measured using both women’s perceived quality as well as through observations of care in delivery hubs. Perceived quality ratings will include women’s ratings on technical quality and respectful care quality for delivery as well as for ANC and postnatal care (PNC). Using the observations of delivery care, technical quality of routine care will be assessed through a 20-item index capturing the processes of intrapartum and immediate postpartum care [15]. Respectful care will be measured using a 10 item index measuring provider actions, such as whether or not a provider explains procedures before proceeding [16]. Care utilization will be measured with both the facility utilization rate, the delivery hub utilization rate, as well as ANC and PNC utilization, using the pregnancy registry. Additional data from the Kenyan Health Information System (KHIS) will be used to compare delivery hub utilization rates between enrolled and non-enrolled pregnant women during the study period. Maternal complications will be defined through the self report of one of the following: fits (in absence of history of epilepsy), loss of consciousness for > 1 h, high fever with chills and foul-smelling discharge, blood transfusion, hysterectomy, intensive care unit admission, mechanical ventilation, or hospital stay of > 7 days. Gaps in access and outcomes by education, proximity to delivery hubs, and household socio-economic status will be compared for the satisfaction, utilization, and cost outcomes described above in implementation sub-counties versus control sub-counties. For the implementation science component of the study we will measure coverage, adoption, and fidelity of SDR implementation using survey, administrative, and program data on activities and outputs. We will track indicators on program reach (e.g. percentage of primary care clinics where women are routinely advised to deliver at delivery hubs; percentage of women who report being advise to deliver in delivery hubs); adoption (e.g. percentage of women delivering at delivery hubs; percentage of delivering at health centers; labor and delivery quality of care; ANC and PNC utilization and quality; C-section rate); and intervention fidelity (maternity ward and newborn unit expansions in delivery hub hospitals; specialist physician and nurse staffing levels; maternity and newborn staff knowledge levels; availability of critical inputs such as blood for transfusions). Given the size and scope of the policy reform, it is likely that SDR will affect the health system in indirect ways, beyond the hypothesized effects on maternal and newborn health services. Complex system-level reforms can have both positive and negative broader systemic effects. Therefore we will measure potential system-level effects of SDR as secondary implementation outcomes (Table ​(Table11). Implementation evaluation indicators MNCH: Maternal, Newborn and child health, HIV: Human Immunodeficiency virus, TB: Tuberculosis For the impact evaluation evaluation component, we performed simulation-based power calculations for the composite outcome in the stepped wedge evaluation design to determine the number of facilities to sample. We excluded private facilities, facilities in Kakamega town, and facilities with less than or equal to 6 first antenatal care visits per month. There were 94 facilities in control sub-counties and 59 in treated sub-counties. We assumed that the baseline composite outcome was 0.0406. Given that direct observation of the primary composite health outcome was not available at baseline, we estimated it using three strategies and selected the most conservative estimate (40.6 events per 1000 births) based on a prospective cohort study of maternal death, stillbirth and neonatal deaths with a site in Western Region of Kenya encompassing 31,118 births [17]. We then generated outcome data across Kakamega’s health facilities by: where k indicates sub-county, Y is composite outcome occurrence for delivery i in for a women living in sub-county k and trt takes value 1 or 0 depending on if the cluster is receiving the intervention during that phase. We also include a random intercept γ0k for each sub-county drawn from a normal distribution with standard deviation 0.20, which was based on an assessment of expected variation in composite outcome rates between sub-counties. To determine the appropriate number of facilities to sample, we calculated the power varying the number of facilities sampled from the treated and control sub-counties. We generated 1,000 simulated datasets followed the planned sampling design by first stratifying by sub-county and then sampling facilities with probability proportional to the number of antenatal care visits in 2021. Over the full study (following the stepped wedge design in which all sub-counties ultimately receive the SDR reform), we determined that we would have 89% power to detect a 12.5% reduction in composite outcome (a drop from 0.0406 at baseline to 0.0355) for 72 facilities, roughly corresponding to 60,000 births. To optimize power in the difference-in-difference Phase 1 study, we determined that equal allocation of 36 facilities in Phase 1 sub-counties and 36 facilities in Phase 2 and 3 sub-counties (18 facilities in Phase 2 and 18 facilities in Phase 3 sub-counties) would result in 86% power to detect a 30% reduction in composite outcome (with 72% power to detect a 25% reduction). For our secondary outcome of satisfaction, we determined that 2,000 women from the longitudinal study sample would result in > 80% power to detect a 0.15-standard deviation increase for both evaluation designs. The primary analysis will assess the impact of SDR on the composite outcome comprised of maternal mortality, neonatal mortality, and stillbirth. We will perform both an intention to treat and per protocol analysis. Intention to treat analysis: We will fit the model in Eq. (1), as well as analogous linear models. A participant’s treatment status will be determined based on her sub-county of residence. If she lives outside of Kakamega County, it will be determined by the location of the facility where she is enrolled. Since the order in which subcounties were selected for SDR was not random, but rather was a decisions taken by Kakamega County policymakers, it is possible that respondents will differ on observable characteristics which may be correlated with the study’s outcomes of interest. We will account for sources of confounding by controlling for mother’s education, age, parity, transport time to nearest delivery hub, season of pregnancy, and perceived pregnancy risk in our regression model. Self-reported pregnancy risk is captured by the following question “How concerned are you about complications during this pregnancy?” with a response scale of 1 (no concern) to 5 (very concerned) and will be coded as a binary variable indicating high risk (choice of >  = 4). Distance will be measured by approximate travel time from a woman’s village to the delivery hub. Additional confounders from external data sources such as rainfall or temperature data may be incorporated in the model, depending on availability. Per protocol analysis: We will compute the proportion of women living outside of intervention areas that are delivering at another sub-county’s updated delivery hub. If this happens often (> 5% of deliveries in non-intervention facilities), we will also estimate the effect of delivery in an updated delivery hub on the composite outcome (instead of the effect of living in a treated sub-county). In addition to the confounders listed for the intention-to-treat analysis, we will adjust for whether the woman comes from an intervention area. Account for missing outcomes data: We expect there to be loss-to-follow-up after delivery resulting in missing values for the composite outcomes. The survey team will attempt multiple phone-based and in-person tracking options before coding an outcome as missing. We will use multiple imputation to impute missing outcome values. Secondary outcomes include satisfaction with care, respectful care index, out-of-pocket expenditures, location of delivery, travel time, maternal complications, and equity. These outcomes will be analyzed as above, with the exception the respectful care index and travel time outcomes, which will be restricted to the random sample of women in the longitudinal cohort. We will also analyze equity for selected outcomes (e.g. quality of care, OOP cost). In this analysis we will include an interaction term between delivering at an updated delivery hub and the equity markers to assess baseline disparities in outcomes and test if these disparities changed during the intervention. Phase 1 interim analysis: After six months of data has been collected following the implementation of improvements in Phase 1, we will conduct a study on interim outcomes to understand implementation of the reform and how it has affected women’s decision-making about delivery care. We will examine the effect of policy exposure on the same outcomes as in the full evaluation. The main analytical difference is that this analysis will be conducted using a difference-in-difference design rather than a stepped-wedge design as we will only observe the intervention among women living in Phase 1 sub-counties. Otherwise, the analytical approach will be the same for the Phase 1 study as for the full SDR evaluation. Data collection will be monitored in an ongoing basis using the interim analysis described in Additional file 1: Appendix 1.