Evaluating implementation effectiveness and sustainability of a maternity waiting homes intervention to improve access to safe delivery in rural Zambia: A mixed-methods protocol

Background: In low-income countries such as Zambia, where maternal mortality rates are persistently high, maternity waiting homes (MWHs) represent one potential strategy to improve access to safe delivery, especially for women living in remote areas. The Maternity Homes Access in Zambia project (MAHMAZ) is evaluating the impact of a MWH model on women’s access to safe delivery in rural Zambia. There is a growing need to understand not only the effectiveness of interventions but also the effectiveness of their implementation in order to appropriately interpret outcomes. There is little evidence to guide effective implementation of MWH for both immediate uptake and to promote sustainability in this context. This protocol describes a study that aims to investigate the effectiveness of the implementation of MAHMAZ by not only documenting fidelity but also identifying factors that influence implementation success and affect longer-term sustainability. Methods: This study will use mixed methods to evaluate the implementation effectiveness and sustainability of the MAHMAZ intervention. In our study, “implementation effectiveness” means to expand beyond measuring fidelity to the MWH model and includes assessing both the adoption and uptake of the model and identifying those factors that facilitate or inhibit uptake. Sustainability is defined as the routine implementation of an intervention after external support has ended. Quantitative methods include extracting data from existing records at the MWHs and health facilities to analyze patterns of utilization, and conducting a routine health facility assessment to determine facility-level factors that may influence MWH implementation and woman-level outcomes. We will also conduct an experience survey with MWH users and apply a checklist to assess fidelity to the MWH model. Qualitative methods include in-depth interviews and focus group discussions with MWH users, community members and other stakeholders. Qualitative data will be analyzed using an integrated framework drawing constructs from the Consolidated Framework for Implementation Research and the Conceptual Framework for Sustainability. Discussion: The findings from this evaluation will be shared with policymakers formulating policy affecting the implementation of MWH and may be used as evidence for programmatic decisions by the government and supporting agencies in deciding to take this model to scale. Trial registration: NCT02620436, Registered 3 December 2015, Prospectively registered (clinicaltrials.gov; for the overarching quasi-experimental impact study). The overall aim of this study is to generate evidence on the implementation effectiveness and sustainability of the MAHMAZ project and reasons for variation in order to inform the interpretation of the outcomes of the main trial. The specific objectives of our study include: The MAHMAZ cRCT is taking place in three districts in the Southern Province (Choma, Kalomo, and Pemba) and one district in Eastern Province (Nyimba) of Zambia. All study districts are primarily rural. Choma district has a total population of 247,860, with 76% living in rural areas. Kalomo, which has the largest rural population in Southern Province, has a total population of 258,570, with about 93% living in rural areas [20]. During the 2010 census Pemba was part of Choma district. Nyimba district has a total population of 77,359, with 91% living in rural areas [21]. A total of 22 health facilities in the districts were eligible for the intervention based on criteria to ensure a basic level of quality, including transfer time to nearby facilities that provide comprehensive emergency obstetric and newborn care (2 h or less), delivery volume (≥150 per year), and either 1) the ability to perform basic emergency obstetric and newborn care (BEmONC) signal functions (five out of seven), which are a set of interventions to care for the mother and newborn during intrapartum care [22] or 2) had at least one skilled attendant on staff, routinely practices active management of third stage of labor, and had no stock outs of oxytocin or magnesium sulfate in the preceding 12 months. The 20 facilities farthest from the referral hospital were selected and matched into pairs based on travel time to referral hospital and delivery volume. Pairs were then randomly assigned to the intervention or control group. Detailed information about the setting, randomization process, and selection criteria can be found in the trial protocol [10]. MWHs were also constructed at two of the five main referral hospitals in the study districts. The MWHs at referral hospitals do not include all of the intervention elements of the Core Model. They are being assessed separately and therefore are not detailed in this implementation protocol. We will use a mixed-methods approach and collect longitudinal, cross-sectional data at both the MAHMAZ intervention rural health center sites (n = 10) and the matched control sites (n = 10) before and during MAHMAZ project implementation, and for a short period (approximately 4 months) after cessation of project support. Our study is guided by an integrated framework drawing from the Consolidated Framework for Implementation Research (CFIR) by Damschroder et al [19] and the Conceptual Framework for Sustainability of Public Health Programs (the Sustainability Framework) put forward by Scheirer and Dearing [18]. CFIR synthesizes constructs from multiple key implementation frameworks that are hypothesized to influence implementation. The Sustainability Framework posits the relationships between factors affecting sustainability and sustainability outcomes within a broader socio-political environment. Because these frameworks address different but related key drivers of program success – implementation and sustainability – elements from each underpin our study. We discuss these frameworks in more detail in the analysis section below. Figure 1 summarizes data collection tools (Fig. 1). The four categories of quantitative tools, represented in the outer circle, include those designed to assess MWH utilization and activities; facility delivery and health outcomes; fidelity and quality of the MWHs and the health facilities; and the potential for sustainability. Qualitative tools, represented in the inner circle (in-depth interviews [IDIs], focus group discussions [FGDs], MWH leadership records, and project records) serve to triangulate and provide depth and context to the quantitative data (Additional files 1, 2, 3, 4, 5, 6, 7, 8 and 9). Table 2 links the evaluation questions, data collection activities and data sources by objective. Table 3 describes each tool, its original data source, and the frequency of data collection. The maximum estimated sample size for this evaluation is 14,400 (Table 4). Qualitative data may have a lower sample size as we will stop during any given round when we reach saturation or predictability. The record review is estimated as utilization is likely to vary across sites. If we approach the maximum sample size because of underestimating utilization, we will amend the protocol. Routine implementation data by category Objectives, Questions, Data Sources for the MAHMAZ Implementation Effectiveness Protocol a. At intervention sites, what is the proportion of MWHs that meet each component of the Core Model? b. What proportion of intervention sites are operating according to the project standard operating procedures? Quantitative MWH Register Others Register MWH Activity Log MWH Experience Survey Core Model Checklist Qualitative Project records a. What are the barriers and facilitators to implementation of the: 1. Core Model? 2. Governance and management models? 3. IGAs? Qualitative IDI with MWH governance and management structures IDI with IGA staff/volunteers IDI with health facility/system staff a. How does satisfaction with staying at the MWH compare in intervention sites relative to control sites? b. To what degree are the intervention MWH perceived as responsive to community standards and needs? c. What are the essential features and characteristic of the MWH as perceived by both women and communities? d. What are continued barriers to accessing and utilizing MWH after the intervention? Quantitative MWH Experience Survey Qualitative IDIs with MWH governance and management structures FGDs with pregnant/ recently delivered women (RDW), men with children < 1, community el ders, Safe Motherhood Action Group members (S MAGs) a. How does capacity to handle obstetric emergencies change over time at the facility? b. How do staff perceptions of care related to maternal health change over time? c. What role does the MWH have in shaping staff perceptions of maternal health care? Quantitative Health Facility Assessment Qualitative IDI with health facility/system staff a. How does the utilization of the MWH change over time? b. How do referral patterns, census and demographic of those utilizing health facilities change over time? c. What are the perceived changes in service utilization at both MWH and health facilities over time? Quantitative Delivery Register Maternity Ward Admissions Register (for referrals) or ad hoc referral register Postnatal Care Register MWH Register Qualitative IDI with health facility/system staff IDI MWH governance and management structures Project records a. What external factors may have influenced the implementation and outcomes observed? b. How does the external and policy environment of the health system influence the utilization and uptake of MWH for delivery? Qualitative IDI with health facility/system staff IDIs with MWH governance and management structures FGDs with pregnant/ RDW women, men with children < 1, elders, SMAGs Project records a. What operational and financial systems are in place and functional at baseline? b. What is the contribution of each strategy in the overall financial sustainability of the MWH, and how does this change over time? c. Which strategies and managements models are perceived as viable for long-term sustainability? What differences exist between communities? d. What is the effect of each strategy/management model on the utilization of the MWH? Quantitative MWH Main Cashbook IGA Sales Journal Monthly Goat Stock MWH Register Qualitative IDI with health facility/system staff IDI with MWH governance and management units FGDs with pregnant/ RDW, men with children < 1, elders, SMAGs Project records MAHMAZ Implementation Evaluation Data Collection Tools and Timing of Administration 1) Provincial & District Government 2) Health Facility Staff 3) MWH Governance 4) MWH Management 5) IGA Elicit information on the challenges, strengths, and perceived impact of the maternity home on the health facility staff and perceived patient behavior in terms of utilization, bypassing, etc. Also elicit perceptions of MWH within the health system and changes in prioritizing, financing, and planning for MWHs. Themes will include perspectives on governance and management challenges and successes, progress toward financial and operational sustainability, perspectives on community ownership of the MWH, MWH responsiveness to community needs/acceptability, and ideas for improvements. Elicit information on challenges and successes of the IGAs and perceived impact of the IGAs on the MWHs. 1) Pregnant or women with a child under 1 year 2) Men with a child under age 1 3) Community elders or mother-in-law 4) SMAG members Maximum Estimated Sample Size by Data Collection Method for the MAHMAZ Implementation Effectiveness Protocol In order to assess the degree to which each individual intervention was implemented according to the project plan and to document any adjustments and/or adaptations, we will extract data from MWH Registers, conduct MWH Experience Surveys and the Core Model Checklist, and review project records. On a monthly basis, we will systematically extract and aggregate data from the registers created for this project (MWH Register, Others Register, and Shelter Activity Log) at intervention and control health facilities, on MWH utilization and educational classes provided at the homes. On a monthly basis, the Monitoring & Evaluation (M&E) research assistants will visit each site and will administer the MWH Experience Survey. This is a primarily quantitative instrument that captures women’s general perceptions of and experience in the MWH (or other facility-affiliated structure in control sites) as related to the Core Model domains; opportunity costs incurred while staying at the MWH; and intended future use. The MWH Experience Survey captures the domains of quality that the community members identified as important to them in the formative evaluation [9]. The survey captures whether or not a particular element of the Core Model: 1) was available to the waiting mother at the time of say; 2) was utilized by the waiting mother at the time of stay; and 3) the waiting mother’s perception of quality of each Core Model element. These data are important for monitoring whether or not the service is well-received by our population of interest and what changes could be made to improve implementation. At each site, study staff will use the MWH register to randomly select a sample of up to six women who have been at the MWH/existing structure for at least three consecutive nights on the day of the visit and have not previously participated in the survey. We will implement a Core Model Checklist to determine the degree to which each of the sites is implementing elements of the Core Model and how standard of care compares at control sites. The Core Model Checklist includes the same domains and Core Model elements as the Experience Survey so as to facilitate comparison and be confident in our assessment of fidelity. The Core Model Checklist will be administered by study staff via an electronic data capture system installed on tablets on a monthly basis at the MWH and will collect information on infrastructure, equipment and supplies, functioning of the management structures, and linkages with the associated health facility. Study staff will observe the MWHs/existing structures and surrounding areas to assess quality and document any quality problems. Study staff will also document whether MWHs/existing structures are adhering to standard operating procedures and implementing elements of the Core Model. Lastly, project records will be reviewed regarding construction for MWHs and IGAs, outfitting of MWHs and IGAs, timing of trainings, training attendance logs and materials covered. Additionally, general project mentorship activities will be reviewed for implementation timelines and how well they adhered to initial plans. Pre/post training tests provide documentation of the effectiveness of the governance, financial management, and IGA skills trainings. Final make-up of the governance and management, financial management, and IGA structures will be compared to the original plan, including documents developed through stakeholder participation, such as terms of reference for the governance committee, standard operating procedures for the MWH, and financial management guidelines. We will use the qualitative data to identify barriers and facilitators to implementation. The study staff will conduct IDIs semi-annually with selected MWH governance committee members and MWH management unit members after the MWHs open (Additional files 3 and 4). These IDIs will be conducted in10 intervention sites, with at least 2 and up to 4 members per site per round (2 governance/2 management). The interviewer will use a semi-structured guide developed from the core frameworks to prompt respondents to discuss each area and probe responses. Questions will examine perspectives on governance and management challenges and successes, progress toward financial and operational sustainability, community ownership, responsiveness to community needs and acceptability, ideas for improvements, and how the IGAs are functioning. We will also conduct semi-annual IDIs with IGA staff/volunteers starting after the IGAs become operational. Questions will gather perspectives on challenges and successes of the IGA, daily operations of the IGA, financial management structures, linkages with the MWH, and progress toward financial and operational sustainability of the MWH. These IDIs will be conducted in 10 intervention sites, with at least 1 and up to 2 members per site per round. The interviewer will use a semi-structured guide to prompt respondents to discuss each area and probe responses (Additional file 5). Lastly, we will conduct IDIs with the health facility staff and district/provincial staff to understand challenges, strengths, and perceived impact of the Core Model on the health facility staff and perceived patient behavior in terms of utilization, bypassing, etc. At control sites, we will ask questions in similar domains, but as per the standard of care. We will also conduct IDIs among the district health staff to understand perceptions of MWHs generally, how they have impacted the health system, and changes in prioritizing, financing and planning for MWHs. We will also ask about general principles of governance and management of MWHs (Additional files 1 and 2). We will use the MWH Experience Survey (see Objective 1), and semi-annual IDIs with governance committee and management unit members (see Objective 2) to assess the extent to which implementation of the Core Model is perceived as responsive to community standards of acceptability. In addition, at baseline (2016), midline (2017) and endline (2018), study staff will conduct FGDs among 1) pregnant or recently delivered women (RDW), 2) men with a child under age 1, 3) community elders/mothers-in-law, and 4) Safe Motherhood Action Group (SMAG) members, a cadre of community health worker. The FGDs are designed to elicit perspectives on: 1) the quality of the mothers’ shelters, 2) community ownership, 3) barriers and facilitators to access to care, 4) sustainability, and 5) ways to improve (Additional files 6, 7, 8 and 9). We will capture basic demographic information on each participant, including past use of an MWH. The team will use a guide to prompt respondents to discuss each topic and probe responses. At control sites, FGD guides will be structured using the same domains but tailored to the standard of care. We will assess changes to health facility-level factors over the course of implementation through the quantitative Health Facility Assessment tool and qualitative IDIs with health facility/systems staff. The Health Facility Assessment tool measures the capacity of the health facility to manage obstetric complications. Questions focus on infrastructure and equipment, including delivery and postnatal beds, the number and qualifications of staff, the ability of and frequency that staff have performed essential obstetric and newborn procedures, and stock outs of essential obstetric-related medicines. We will monitor capacity over time and identify if changes are associated with demand that may be facilitated by the implementation of the MWH. We anticipate respondents for the health facility assessment will be the health facility in-charge, or another staff with knowledge of facility capacity. To complete the assessment, project staff will consult facility registers and health facility staff. We will also conduct semi-annual IDIs with health facility staff and health system staff (see Objective 2). We will use quantitative and qualitative data to assess how service utilization patterns at both MWH and rural health centers may change over time. We will routinely collect health facility data at the 20 sites beginning at the start of the calendar year, approximately 9 months prior to implementation of the intervention. On a monthly basis, study staff will extract key variables about delivery and neonatal outcomes, referrals and postnatal care from routine data systems including the Delivery Register, Maternity Ward Admission Register (or ad hoc referrals register), and Postnatal Care Register. We will also extract the MWH utilization from the MWH Register monthly from the start of the intervention at all 20 sites (see Objective 1). Secondly, using the Safe Motherhood Number (SMN), a unique code that a woman receives when visiting government facilities during her pregnancy, each woman will be tracked from admission to the MWH through delivery and postnatal visits, to understand utilization of MWH and health facility services and maternal and neonatal health outcomes. We will compare this information with data captured in Objective 1. To develop a contextual history of factors that may have influenced implementation or outcomes at the community, district, provincial, and national levels, we will analyze qualitative data sources, including IDIs with health facility/systems staff (see Objective 2), IDIs with MWH governance and management structures (see Objective 2), and FGDs with relevant community members (see Objective 3). Throughout the project, we will also document, through project logs and meeting minutes, other events that occur that may affect the patterns of utilization or the effectiveness of our intervention, such as openings of new health facilities or redistricting of current ones. To identify the key features of entrepreneurial strategies and management models developed to support the finances and operations of the Core Model, and to assess its effectiveness, we will use routinely collected monitoring data from registers, activity logs, training attendance logs, and the Core Model Checklist. From intervention sites, we will extract data from registers implemented by the project at the IGA (Main Cash book, Sales Journal and Monthly Goat Stock) to assess the revenue and expenses of the IGAs and MWH, quantity of products or services sold at the IGAs and stock at the goat rearing sites. For each type of IGA implemented, we will capture the costs of implementation, the MWH revenue over time from IGAs, community donations, or health facility contributions, and the proportion of MWH operating costs covered. Additionally, we will use program data to determine decisions made and changes occurring in MWH operating costs over time to understand what mix of operating costs and revenue streams create a financially sustainable MWH. We will triangulate findings with data from the semi-annual IDIs conducted with governance and management structures, IGAs staff/volunteers, and health facility staff (see Objectives 2 and 3). For all qualitative data collection, all participants will be screened for eligibility and study staff will obtain informed consent. Study staff conversant in English and the appropriate local language, and trained in qualitative research methods and human subjects protection, will conduct the IDIs and FGDs. IDIs and FGDs will be recorded and transcribed verbatim, and translated to English when necessary. Each data collection method will use its own set of inclusion and exclusion criteria as different participants will be enrolled depending in the method (Table 5). Respondent Inclusion and Exclusion Criteria by Data Collection Method • Mother utilizing the maternity home for at least 3 consecutive nights • Mother utilizing the home for maternal health services (ANC, labor or PNC visit) • Currently pregnant or under 6-weeks post-partum • >/= 15 years of age (emancipated minor) • Has responded to the survey at a previous visit for the same maternal health services (ANC, labor or PNC) • Utilizing the home for reasons unrelated to maternal health • Unwilling or unable to provide informed consent MWH Governance Committee & Management Unit Members; IGA Officer and SMAG Tailor • Is a current governance committee/management unit member at an intervention site OR is responsible for overseeing MWH operations at a control site OR is an IGA Officer OR is a SMAG • Is capable of responding to the domains • >/= 18 years of age • Is not currently affiliated with governance or management of MWH any of the study sites • Unwilling or unable to provide informed consent • Respondent is a SMAG member or other potential SMAG member including TBA, CHW, or any individual currently volunteering at the health centre in an MCH capacity, Elder or MIL in a study catchment area • Or is a pregnant woman or with a child under the age of 1 • Or is a male with a child under the age of 1, • >/= 15 years of age • Has previously participated in a FGD in the same year at another site or in another category (i.e.: SMAG and recently delivered) • Is not currently affiliated with any of the study sites • Unwilling or unable to provide informed consent • Is a current health facility staff member at any one of the study sites (intervention, control, CEMONC) • Is knowledgeable in maternal health capacity at the health facility • Or is knowledgeable in health facility financial records • Or is a district or provincial health officer in the study districts • >/= 18 years of age • Is not currently affiliated with any of the study sites • Unwilling or unable to provide informed consent • Is recorded in MWH Register, Others Register or Activity Log between January 2016 – December 2018. • Is recorded in BEmONC facility-based data: Delivery Register, Maternity Ward Admission Register, Postnatal Care Register, Facility Transfer Log, Ante-natal care register (where applicable) and other improvised registers as necessary between January 2016 – December 2018.