Implementing a social accountability approach for maternal, neonatal, and child health service performances in Ethiopia: A pre-post study design

Background: Social accountability approaches, such as the community scorecard (CSC), can improve the performance of health systems in low-income countries by providing a mechanism for obtaining and incorporating community input. This longitudinal study assessed the effects of CSCs implemented by primary health care units (PHCUs) on health system performance in Ethiopia. Methods: This study used a pre-post design and was conducted from October 2018 to September 2019 in 159 PHCUs in 31 districts in Amhara and Southern, Nations, Nationalities and Peoples’ regional states. The data were extracted from a routine health information management system database at baseline, midterm, and endline stages over 12 months for statistical analysis. The effects of implementing CSCs on health system performance were evaluated against selected key performance indicators (KPIs). Results: The CSC measurement results were based on input from 38,556 community representatives. The mean CSC score with standard deviation (6SD) was 60.8%612.5%, 66.3%610.8%, and 70.6%610.0% at baseline, midterm, and endline, respectively. The mean KPI score was 54.9%617.4%, 61.9%615.1%, and 67.6%614.6% at baseline, midterm, and endline, respectively. The average CSC and KPI values were positively correlated (r>0.37). Using a nonparametric Friedman’s test, we found a statistically significant difference in CSC and KPI scores at baseline, midterm, and endline (P=.001). Post hoc analysis with Wilcoxon signed-rank tests was conducted with a Bonferroni correction and the results showed higher CSC and KPI values from baseline to midterm and from midterm to endline (P<.017). Conclusions: The use of CSCs in Ethiopia contributed to the health system’s performance in terms of maternal and child health services. The responsiveness of health workers and utilization of basic health services by community members were found to increase significantly with CSC use. We recommend continued implementation of the CSC intervention at PHCUs. This study was conducted in South Wollo and Kembata Tembaro administrative zones of Amhara and SNNP regional states, respectively. The USAID Transform: Primary Health Care project provides technical, financial, and other resource support to 91 districts in Amhara and 84 in SNNP, as well as to districts in other regional states.20 The project supported 31 districts and 159 PHCUs to start and implement the CSC intervention for over 12 months as a social accountability tool for performance management.21 Ethiopia has a 3-tier health care delivery system. Level 1 is the district (woreda) level, composed of primary hospitals that cover 60,000–100,000 people, health centers serving 15,000–25,000 people, and their satellite health posts covering 3,000–5,000 people, connected to each other through a referral system. The primary hospitals, health centers, and health posts form PHCUs. Districts are subdivided into kebeles (villages), the lowest administrative units. Level 2 includes general hospitals covering 1–1.5 million people. Level 3 includes specialized hospitals covering 3.5–5 million people.16 In the last 3 decades, the country has expanded access to primary health care through 17,187 health posts; 7,245 private health facilities; 3,724 health centers; and 266 hospitals. In addition, more than 151,053 health professionals are serving communities.20 To institutionalize accountability and transparency as a tool for performance management in the health system in both Amhara and SNNP regional states of Ethiopia, 632 participants attended the CSC training of trainers, including 155 from district health offices and 477 from PHCUs. A 3-day classroom theoretical orientation and practical sessions were facilitated in July 2018 at the respective capital cities of the regional states. The CSC orientations were given to 4,053 client councils within the targeted 159 health centers in South Wollo and Kembata Tembaro administrative zones in August 2018.21 The client councils were informed of the 6 standards of the CSC: (1) compassionate, respectful, and caring health workforce; (2) patient waiting time; (3) availability of services, biomedical equipment, and pharmaceutical supplies; (4) health facility infrastructure; (5) ambulance service and management; and (6) clean and safe health facility, citizens' rights, service providers' duties, facilitation techniques, counting and organizing scores, observation skills, verification tools, report submission, and provision of feedback. In addition, every 3 months, the client councils were actively engaged in facility-community interface meetings, presented the results of the CSCs and feedback of community members, addressed issues raised by town hall meeting participants, and closely monitored the implementation of developed action plans. The project supported all client councils with 1-page job aids, reporting forms, and a minute book. From September 2018 to December 2019, USAID Transform: Primary Health Care provided technical, financial, and other resource support to its targeted districts. Some forms of this support were providing team-based strategic problem-solving trainings for health care providers with the formation of performance improvement and quality improvement projects, enhancing the capacity of performance management team members through offering the use of data for decision-making trainings, providing performance improvement subgrant funding for primary health care entities, and organizing and facilitating community-facility interface meetings. In addition, the project's staff and experts from zone health departments provided follow-up visits and on-site coaching for all PHCUs and district health offices on a quarterly basis. During facility visits, the coaching team facilitated the exploration of CSC measurements with feedback and gave opportunities for PHCU management staff to systematically analyze the root causes of any issues, propose prioritized solutions, and develop doable action plans. In addition, the coaches revised the concepts of social accountability, provided feedback on performance, and supported the client councils and health care providers. The coaches then submitted a copy of agreed measurement reports, identified gaps, and developed action plans with district health offices and to zone health departments. Baseline data were extracted from the period of October 10–20, 2018. At 6 and 12 months after the intervention periods, secondary data were collected during April 10–20, 2019, as midterm assessments, and during October 10–22, 2019, as endline measurements by the data collection teams.21 For this study, the investigators used a longitudinal pre-post interventional study design.22 The necessary data were collected from October 2018 to September 2019. We used quantitative methods to measure the effects of CSC implementation on health-seeking behavior, health service utilization, and responsiveness of health care providers. The study participants were identified from the Amhara and SNNP regional states in Ethiopia. Two administrative zones where the project had provided technical and other resource support for 12 months or more were purposively selected. All 31 districts (woredas) and 159 PHCUs were included in selected zone administrations. Documents and records from each primary health care entity were reviewed at the 3 points of the assessment. The required routine health management information system data on CSC and on maternal and child health services were extracted from 159 PHCUs' RHIMS database. To ensure data completeness, accuracy, consistency, and reliability, 8 data managers and 2 supervisors were trained for 3 days. The training covered the objective of the pre-post interventional study, ethical issues, quantitative data extraction methods, and piloting all tools and ethical principles. During the real data collection, all investigators actively monitored completeness and consistency of data on a daily basis. Data were extracted using structured and pretested forms. Dependent variables: The average measures, from 0% to 100% on 10 key performance indicators (KPIs) were reviewed and collated: (1) contraceptive acceptance rate, (2) syphilis screening among antenatal care (ANC) clients, (3) skilled delivery services coverage, (4) postnatal care coverage, (5) full immunization coverage, (6) under 2 years growth monitoring coverage, (7) proportion of available essential or tracer drugs, (8) proportion of clean and safe health facility standards met, (9) proportion of available laboratory and diagnostic services, and (10) patient flow and service organization (Table 1).18,21 The effects of CSC implementation on the health system's performance were measured at baseline, midterm (after 6 months), and endline (after 12 months). Description of Key Performance Indicators and Community Scorecard Scoring Standards to Assess Effects of Implementing a Social Accountability Approach on Improving Health System Performance in Maternal and Child Health Services in 2 Regional States in Ethiopiaa Independent variables: The data extraction forms dedicated to capture information on CSCs were adopted from the nationally endorsed CSC reporting tools. The forms captured the summary of the following 6 standards: (1) compassionate, respectful, and caring health workforce; (2) patient waiting time; (3) availability of services, biomedical equipment, and pharmaceutical supplies; (4) health facility infrastructure; (5) ambulance service and management; and (6) clean and safe health facility, reported as an average score from a 5-point Likert scale measurement (Supplement 1), that is, 5=very good to 1=very low. The overall summary of CSCs was reported as a percentage from 0% to 100.0% on 3 occasions17 (Table 1). The data were collated using a Microsoft Excel sheet and exported to Statistical Program for Social Science (SPSS IBM V 20) software for analysis. To ensure consistency and reliability, the data were double entered by experienced data encoders. The PHCU scores of CSCs and KPIs were reviewed for completeness and consistency. Quantitative data analysis methods were used, which included descriptive statistic frequencies, mean, median, interquartile ranges, and standard deviations. To determine the presence of a linear relationship between baseline, midterm, and endline scores, a Pearson Product-Moment Correlation technique was employed. After checking the assumption of the nonparametric test, we used the Friedman23 test, which includes (1) a single group measured on 3 or more different occasions, (2) a group that is a random sample from population, (3) use of a continuous level of dependent variables, and (4) samples that do not need to be normally distributed. Post hoc analysis with the Wilcoxon signed-rank tests was conducted with a Bonferroni correction applied and a statistical test result with a P-value of <.017, indicating presence of a significant difference between CSCs and KPIs at baseline, midterm, and endline measurements. The ethical clearances of this study were granted by the JSI Institutional Review Board (IRB), and the Amhara Public Health Institute and SNNP Regional State Health Bureaus' IRBs. The research protocol of this pre-post interventional study was granted certification from the Amhara Public Health Institute (reference number: M/T/SH/D/03/435) and the SNNP Regional State Health Bureau Research and Ethics Review Committees (reference number NS 12/36/22). In addition, the study protocol was reviewed at the JSI Research & Training Institute, Inc. IRB, which determined that this activity was exempt from human subjects' oversight (reference number IRB no. 19-16E). To maintain the confidentiality of collected data, anonymity was maintained throughout the research process. Permission to use data from zone health departments and PHCUs were obtained through formal written requests. Informed individual written consent was taken from each study subject. Both quantitative and qualitative data were collected in aggregate forms. All PHCUs were encouraged to use the data for evidence-based decision making and to be responsive to the demands of rights holders. The summary of CSCs and KPIs were submitted to district health offices, zone health departments, and regional health bureaus at the 3 points of the assessment. Throughout the research process, the investigators maintained national and international ethical principles.