Comparison of maternal and child health service performances following a leadership, management, and governance intervention in Ethiopia: a propensity score matched analysis

Background: Leadership, management, and governance (LMG) interventions play a significant role in improving management systems, enhancing the work climate, and creating responsive health systems. Hence, the Ethiopian Ministry of Health with the support of the USAID Transform: Primary Health Care project has been implementing LMG interventions to improve performances of primary healthcare entities. The purpose of this evaluation was to compare maternal and child health service performances and overall health system strengthening measurement results of primary health care entities by LMG intervention exposed groups. Methods: The study used a cross-sectional study design with a propensity matched score analysis, and was conducted from August 28, 2017, to September 30, 2018, in Amhara, Oromia, Tigray, and Southern Nations, Nationalities, and Peoples’ (SNNP) regions. Data collection took place through interviewer and self-administered questionnaires among 227 LMG intervention exposed and 227 non-exposed health workers. Propensity score matched analysis was used to balance comparison groups with respect to measured covariates. Results: The mean overall maternal and child health key performance indicator score with standard deviation (± SD) for the LMG intervention exposed group was 63.86 ± 13.16 and 57.02 ± 13.71 for the non-exposed group. The overall health system strengthening score for the LMG intervention exposed group (mean rank = 269.31) and non-exposed group (mean rank = 158.69) had statistically significant differences (U = 10.145, z = − 11.175, p = 0.001). In comparison with its counterpart, the LMG exposed group had higher average performances in 3.54, 3.51, 2.64, 3.00, 1.07, and 3.34 percentage-points for contraceptive acceptance rate, antenatal care, skilled birth attendance, postnatal care, full immunization, and growth monitoring services, respectively. Conclusion: There were evidences on the positive effects of the LMG intervention on increased maternal and child health services performances at primary healthcare entities. Moreover, health facilities with LMG intervention exposed health workers had higher and statistically significant differences in management systems, work climates, and readiness to face new challenges. Therefore, this study generated evidence for integrating LMG interventions to improve the performance of primary healthcare entities and maternal and child service uptake of community members, which contributes to the reduction of maternal and child deaths. This study employed a cross-sectional study design with a propensity matched score analysis that measured the effects of the LMG intervention on maternal and child health service performance by LMG exposure statuses [19–21]. The PSM approach was used to explain possible differences at baseline variables between LMG exposed and non-exposed groups. The PSM aimed to balance the LMG intervention exposed and non- exposed groups with respect to measured baseline covariates, to achieve a comparison with reduced selection bias [16–18]. Hence, any observed differences on maternal and child health service coverages can demonstrate the effects of the LMG intervention. The study was conducted between August 28, 2017 and September 30, 2018, in Amhara, Oromia, SNNP, and Tigray regions of Ethiopia. The Ethiopian health system is comprised of primary, secondary, and tertiary level institutions. The primary level of the health tier system encompasses district hospitals which cover 60,000–100,000 people and four to five health centers with an average targeted population of 25,000 people. In addition, each health center oversees an average of five satellite health posts with a target population of 3500 to 5000 people. The primary healthcare entities are mandated to provide preventive, curative, promotive, and rehabilitative health services to about 100,000 people. The secondary and tertiary levels manage complex heath conditions for larger size populations and include general and referral hospitals [4]. USAID Transform: Primary Health Care works closely with regional state health bureaus and aims to contribute to the successful achievement of the national goal of preventing child and maternal deaths (PCMD). To realize this aim, the project implements LMG related interventions that create conducive work environments, strengthen management systems, and create responsive healthcare providers and improve their capacity towards resource mobilization, allocation, and utilization [7]. within 4 years, (2017–2019) – through the direct support of the project – 184 districts and 519 health centers had LMG trained healthcare workers [7]. The trainees went on to develop 656 performance improvement projects and implemented them in their respective primary healthcare entities [22]. The LMG intervention is dedicated to capacitating health workers with leading, managing, and governing practices. The USAID Transform: Primary Health Care project, in collaboration with regional health bureaus, randomly selected primary healthcare entities for enrollment into the intervention. Based on the course syllabus, trainees were informed about priority health areas, resource management and health service delivery management, and the concept of leadership, management, and governance for a result model of performance improvement [23]. The LMG interventions included didactic sessions lasting 6 days and implementation of performance improvement projects, supplemented with three to four coaching sessions and participation in knowledge sharing events. During the six-day classroom training sessions, each primary healthcare entity, represented by three to four health workers, committed to work together for about 9 months. Each LMG team then developed maternal and child health service-related performance improvement projects. While developing the projects, teams used the ‘challenge model’ tool [5]. This tool helped the teams to identify their challenges through reviewing their organizational mission and strategic priorities, develop a shared vision, define a measurable result, assess their current situation to take accurate baseline measurements, identify obstacles and root causes (using a fishbone analysis, the why technique, or workflow analysis), define challenges in light of the root causes and select priority actions, develop an action plan that estimates the human, material, and financial resources needed as well as timelines for implementing the actions, and implement the plan and monitor and evaluate progress towards achieving the desired results [5, 23]. Some of the performance improvement projects developed and implemented by the trainees aimed to increase the proportion of institutional deliveries, the proportion of modern contraceptive users, and the proportion of well-baby service utilization (Additional file 1). Upon returning back to their workplaces, the LMG core teams were instructed to organize a consensus building meeting on their performance improvement project, with participation from all health workers. Hence, staff of primary healthcare entities with shared visions worked together to achieve optimum results. During the implementation of each performance improvement project, each team received three to four onsite coaching sessions from experts [23] (Additional file 2). The coaches applied the observe, ask, listen, feedback and agree (OALFA) technique. At the end of the performance improvement projects, LMG intervention exposed health workers were invited to present their findings and lessons in knowledge sharing events organized at district level. Two groups were targeted, namely: LMG intervention exposed and non-exposed health workers employed in primary healthcare entities providing maternal and child health services within the four regions of Ethiopia. Health workers who volunteered to participate and that were working within the USAID Transform: Primary Health Care projects’ targeted primary healthcare entities were enrolled in this study. The national average performance scores against selected maternal and child health indicators of primary healthcare entities were found to be at 65% [24]. With the assumption of training directives, LMG trained and capacitated healthcare workers improved the score of primary healthcare entities to 80% or more on the same key performance indicators (KPIs), and believing the presence of heterogeneity across regions and between primary healthcare entities, design effect of 2 was considered [25, 26]. The required sample size with a power of 90% to detect the effects of the LMG intervention for exposed (n1 = 272) and non-exposed (n2 = 272) groups was 544. The researchers used a database as a sampling frame to select primary healthcare facilities using systematic random sampling methods. Following this step, data collectors randomly chose one of three key persons using a lottery method, namely: head of health centers, maternal and child health core process owners, and health center – health post linkage focal persons. The questionnaires were prepared after a thorough review of relevant literature and the national leadership, management, and governance in-service training materials for hospitals and health center managers [3, 23, 27]. The questionnaires were developed in English (Additional file 3) and translated into local languages (i.e., Amharic, Afan Oromo, and Tigrigna), then translated back into English. Data were collected from LMG intervention exposed and non-exposed healthcare providers using interview and self-administered structured questionnaires. Sixteen data collectors, each with clinical, health management, social science, or public health training were recruited. Data collectors and supervisors were trained on ethical principles, data collection tools, and interviewing techniques. Before the actual data collection began, all tools were piloted and amended accordingly. Six maternal and child health key performance indicators endorsed by the Ministry of Health [24], and three categories of immediate outcomes on the leading, managing, and governing model for results measurements were considered in the study [5, 23]. The detailed characteristics of selected KPIs are discussed after which the features of the health system strengthening variables are presented. Data were extracted from health facility routine health information management systems. Six maternal and child health program performance KPIs were selected as dependent variables. These indicators were: (1) contraceptive acceptance rate (CAR), (2) antenatal care (ANC), (3) skilled (institutional) birth attendance (SBA), (4) postnatal care (PNC), (5) full immunization coverage for infants under the age of 1, and (6) growth monitoring for children 2–59 months old. Each indicator was scored from 0 to 100 percentage points. The average of all six indicators was considered as an overall maternal and child health performance score. Eighteen questions, with a 10-point Likert scale [28] were used to measure responses to statements presented on: strengthened management systems (9 items), enhanced work climate (5 items), and responsiveness or capacity of the health system to overcome new challenges (4 items). The sample questionnaire for each category is presented in Table 1. The respondents were LMG intervention exposed and non-exposed health workers operating within primary healthcare entities. Sample questionnaires The box presents sample items from composite scales, representing the three categories namely: management systems, work climates, and responsiveness Data on possible independent variables were collected. Individual characteristics of the study participants which included age, gender, profession, educational level, service years, and salary were captured. In addition, primary healthcare entity characteristics such as access to roads, distance from zone capital in kilometers, and catchment population were collected. The data were analyzed using the Statistical Package for Social Sciences (SPSS) version 25 [29]. A PSM analysis was conducted using the R-plugin for propensity score matching for SPSS [30] [31]. The internal reliability of the tools was assessed using Cronbach’s alpha values [32]. According to Bland et al. [33], if the Cronbach’s alpha value score is more than 0.7, the scale can be considered reliable. The tools have 18 questions divided into three categories, namely: strengthened management systems (9 questions), enhanced work climate (5 questions), and the capacity to respond to challenges (4 questions). The reliability test results were 0.839 for work climate, 0.895 for strengthened management systems, and 0.886 for the capacity to respond to changes, which showed the scale used was internally consistent and reliable. A statistical test using multi-collinearity analysis through determining the variance inflation factor (VIF) was run to check the tools’ divergent validity [20]. According to Menard [34], if the VIF reported value exceeds 10, it implies the associated regression coefficients are poorly estimated because of multi-collinearity. In this study, the collinearity test VIF results were: 1.018 for work climate, 2.94 for strengthened management systems, and 6.443 for capacity to respond to new challenges and no VIF value exceeded 10. Hence, there was no observed multi-collinearity affecting the regression coefficients. PSM is a single score used to balance the LMG intervention exposed and non-exposed groups on observed covariates [30]. In this study, the covariates included in the PSM model are the socio-demographic characteristics of health workers including gender, age, profession, salary, marital status, and tenured service years. In addition, the researcher included the characteristics of the targeted primary healthcare entities, regions, catchment populations, distance from zone capitals in kilometers, and access to all weather roads in the PSM model. During matching, a standardized difference of covariates between the two groups of less than 10% was taken as adequate. Using a logistics regression estimation (logit model), the PSM was analyzed by assigning 1 for LMG intervention exposed and 0 for non-exposed groups, with nearest neighbor matching, caliper of 0.2 and matching ratio of 1 to 1. The relative multivariate imbalance L1 was 0.926 before matching and 0.916 after matching [35]. Only ‘place of work’ had a large imbalance after PSM with |d| = 0.80, which violates the recommended criteria standard error difference of |d| < 0.25 (Additional file 4). Finally, using PSM, data from 454 (227 LMG intervention exposed and 227 non-exposed) participant groups were used to analyze and compare maternal and child health service performances of primary healthcare facilities (Additional file 5). Descriptive statistics were employed using frequencies and tables. The homogeneity of the data was checked using Levene’s test for equality of variances. The results revealed that maternal and child health coverage data were found to be homogeneous at p > 0.05. While the data on strengthening management system, enhanced work climate, and capacity to respond to new challenges were found to be non-homogeneous at p < 0.05 (Additional file 6). An independent sample t-test was analyzed to check statistical difference on maternal and child health service performance coverage between LMG intervention exposed and non-exposed groups. In addition, a non-parametric test, (Mann-Whitney U test) was conducted to check presence of significant differences in mean rank LMG intervention exposed and non-exposed groups [31]. Ethical clearance was obtained from four regional institutional review boards. Ethical clearances were granted by the Amhara Public Health Institute, (Ref. No. HRTT02/137/2018), the Oromia Regional State Health Bureau, (Ref. No. BEFO/HBTPH/1-8/476), the SNNP Regional State Health Bureau, (Ref, No. PLMG-19/8407), and the Tigray Regional State Health Bureau’s (Ref. No. 453/1418/10) institution review boards and research ethics committees. Data were collected after getting full informed written consent from each participant and facility manager. Privacy, anonymity, and confidentiality were maintained throughout the data collection, analysis, and report writing activities. This study has no known risk and no payment was made to participants.