Does a complex intervention targeting communities, health facilities and district health managers increase the utilisation of community-based child health services? A before and after study in intervention and comparison areas of Ethiopia

INTRODUCTION: Ethiopia successfully reduced mortality in children below 5 years of age during the past few decades, but the utilisation of child health services was still low. Optimising the Health Extension Programme was a 2-year intervention in 26 districts, focusing on community engagement, capacity strengthening of primary care workers and reinforcement of district accountability of child health services. We report the intervention’s effectiveness on care utilisation for common childhood illnesses. METHODS: We included a representative sample of 5773 households with 2874 under-five children at baseline (December 2016 to February 2017) and 10 788 households and 5639 under-five children at endline surveys (December 2018 to February 2019) in intervention and comparison areas. Health facilities were also included. We assessed the effect of the intervention using difference-in-differences analyses. RESULTS: There were 31 intervention activities; many were one-off and implemented late. In eight districts, activities were interrupted for 4 months. Care-seeking for any illness in the 2 weeks before the survey for children aged 2-59 months at baseline was 58% (95% CI 47 to 68) in intervention and 49% (95% CI 39 to 60) in comparison areas. At end-line it was 39% (95% CI 32 to 45) in intervention and 34% (95% CI 27 to 41) in comparison areas (difference-in-differences -4 percentage points, adjusted OR 0.49, 95% CI 0.12 to 1.95). The intervention neither had an effect on care-seeking among sick neonates, nor on household participation in community engagement forums, supportive supervision of primary care workers, nor on indicators of district accountability for child health services. CONCLUSION: We found no evidence to suggest that the intervention increased the utilisation of care for sick children. The lack of effect could partly be attributed to the short implementation period of a complex intervention and implementation interruption. Future funding schemes should take into consideration that complex interventions that include behaviour change may need an extended implementation period. TRIAL REGISTRATION NUMBER: ISRCTN12040912. The Ethiopian Government initiated the OHEP intervention in 26 districts of Amhara, Southern Nation, Nationalities and Peoples, Oromia and Tigray regions with an approximate population of 3.5 million (figure 1). Intervention districts were selected by the government of Ethiopia and implementing partners for having both a relatively low utilisation of primary child health services and the availability and ability of partners to support implementation. The implementers were four non-governmental organisations (PATH, UNICEF, Save the Children and Last 10 Kilometres/John Snow Inc.). The intervention started in 2016 and lasted for a duration of 2.5 years and had three components: 1) community engagement, 2) primary care level capacity building and 3) ownership and accountability of child health services at the district level. The intervention activities under these components, along with the underlying assumptions, intermediate indicators and outcomes are detailed in table 1. Map of Ethiopia showing all regions (left) and the intervention and comparison districts within the four study regions (right). Optimising the Health Extension Programme intervention implemented in 26 districts of Ethiopia, the assumptions and the expected intermediate indicators and outcomes The protocol for the evaluation of the OHEP implementation has been published.20 This study was based on a plausibility design with 26 intervention and 26 comparison districts (woredas) in four regions of Ethiopia. The baseline survey was conducted from December 2016 to February 2017 and the endline survey from December 2018 to February 2019 (figure 1). The surveys were conducted by the London School of Hygiene and Tropical Medicine and Ethiopian Public Health Institute along with representatives from Gondar, Jimma, Mekelle and Hawassa Universities. Intervention districts had a low coverage of maternal, newborn, and child health indicators. Comparison districts were selected by the Regional Health Bureaus to be similar to the size of the population, the burden of diseases, number of primary healthcare units, health service coverage, length of iCCM and CBNC service delivery and absence of partners implementing other demand generation activities. We used a two-stage stratified cluster sampling to select a representative sample of households within intervention and comparison areas. In the first stage, a list of all enumeration areas of the study districts was obtained based on the 2007 Ethiopian Housing and Population Census. Two hundred enumeration areas (clusters) were selected with probability proportional to the size of the district. Each cluster served as the primary sampling unit. Within clusters, households were selected by systematic random sampling. The Women’s Development Army leaders, HEWs, health posts, health centres with staff and woreda health offices serving the selected clusters were also surveyed. The sample size was calculated to measure changes with adequate power in a fixed number of percentage points of key indicators between intervention and comparison areas at baseline and endline. Based on the Ethiopian DHS data, a cross-sectional survey of 3000 households in 100 intervention and 100 comparison clusters was expected to find 1747 children under the age of 5 years in each arm.21 A Tanzanian childhood study found that 50% of under-fives had an illness in the 2 weeks before the survey.22 The current research assumed more conservatively 30% of children 2–59 months to have had any illness during the 2 weeks before the interview. This sample size of 3000 households per group with 90% completeness and a design effect of 1.3 would give 80% power to detect differences of 10–20 percentage points across a range of child health indicators (5% significance level). The baseline survey found fewer than the expected number of sick children in the 2 weeks before the survey. As a result, the household sample size for the endline survey was doubled. We used the baseline list of households to select 66 households randomly in each enumeration area. For every selected household, we interviewed the household head, listed residents and collected sociodemographic characteristics. The interviews included caregivers of children aged 2–59 months to assess their knowledge of childhood illnesses and care-seeking for sick under-five children in the 2 weeks before the survey. Furthermore, women of reproductive age (13–49 years) were interviewed to identify births in the 12 months before the survey, with additional questions on care-seeking for illness in the neonatal period. Up to three visits were made to each participant to ensure the study reached its target sample size. The health facility questionnaires assessed the infrastructure, equipment, supplies and staff available on the day of the survey. Also, data were collected from facility registers on services provided to sick children at health posts and health centres in the 3 months preceding the survey. The health centre staff, HEWs and Women’s Development Army leader modules covered their background, knowledge, training in the last 12 months, supervision in the previous 6 months and the services they provided in the last 3 months. Interviewers collected information at the district health office on demography and characteristics that might affect services for under-five children. The questions and content of each survey module were based on existing large-scale survey tools and the authors’ previous evaluation of the iCCM and CBNC programmes.22 23 All questionnaires were translated into three local languages (Amharic, Oromifa and Tigrigna), pretested and revised. Data collectors and supervisors were trained over 10 days, including field training before the start of data collection. They were not provided information on whether a district was an intervention or comparison area. Data were collected on personal digital assistants (Companion Touch 8), and tablets (Toshiba and Hewlett Packard) programmed with CSPro 6.3 at baseline and CSPro 7.1 at the endline. Data collectors sent encrypted data from the field to the password-protected server at the Ethiopian Public Health Institute. Data managers conducted quality checks and feedback to field teams. Data were cleaned, checked for errors, including consistency and completeness. Since OHEP was a community and health system level intervention, a data monitoring committee was not deemed necessary. The primary outcome was the proportion of children aged 2–59 months who were reported to have had any illness in the 2 weeks before the survey for whom advice or treatment was sought from an appropriate provider (health post, health centre, hospital and private clinic). Secondary outcomes included: 1) the proportion of sick children aged 2–59 months who were reported to have received appropriate treatment for diarrhoea (oral rehydration solution (ORS) with or without zinc tablets) and possible pneumonia (antibiotics), 2) the proportion of infants born in the 12 months before the survey who were reported to have had any illness in the first 28 days of life for whom advice or treatment was sought from an appropriate provider and 3) the proportion of infants born in the 12 months before the survey who received adequate treatment for suspected neonatal sepsis (antibiotics). Since malaria was not a common illness in the study areas at the time of the surveys, the assessment of appropriate treatment for this illness was excluded. Additionally, the registers for children aged 0–59 days and 2–59 months were reviewed to assess the median number of children seeking care at health centres and health posts in the 3 months before the surveys. We also evaluated intermediate indicators that included, at the community level, the proportion of caregivers that knew signs of illness in children and the proportion that cited appropriate action to be taken for a sick child under 5 years of age. We also assessed the proportion of caregivers that reported receiving health messages on common childhood illnesses and those attending community meetings to discuss maternal, newborn and child health issues. At the health system level, we assessed the proportion of HEWs that had received training and the proportion that had attended performance review and clinical mentoring meetings. We also evaluated the proportion of health centres and health posts that had received supervision and the proportion that had the necessary equipment, supplies and drugs for the provision of child health services. District-level ownership and accountability for child health programmes were reflected in the proportion of districts that had iCCM and CBNC scorecards. These cards were programme management tools for setting targets and monitoring performance. Information was also collected on the average number of ambulances available in districts to transport sick under-five children and whether there were standardised hours of operation for health posts. Descriptive analysis of baseline and endline characteristics in intervention and comparison areas was conducted at the household, caregiver, child, health facility and district levels. Categorical variables were summarised using percentages with 95% CIs. We used means, with SEs, or medians, with IQRs, to summarise continuous variables. At the district level, the demographic and health system-level characteristics were examined. For households, the characteristics of mothers or caregivers of children aged 2–59 months, and women who had a delivery in the 12 months before the survey were assessed. Distribution of age, religion, education, self-reported distance to the nearest health post and socioeconomic status was analysed in intervention and comparison areas at baseline and endline surveys. Similar assessments were done for the distribution of age and sex among children 2–59 months of age and infants born during the 12 months before the survey. Household socioeconomic status was captured by asset ownership, access to utilities and household characteristics. These were aggregated into a single wealth index score using principal component analysis.24 The household aggregated scores were grouped into wealth quintiles, where quintile 1 represented the poorest fifth of the households, and quintile 5 represented the least poor fifth. A linear or logistic regression model was fitted, depending on the variable type, to assess if there were any differences between intervention and comparison areas that changed over time. A variable was considered a potential confounder if the differences between intervention and comparison areas showed a statistically significant (p<0.05) change over time. Caregivers’ knowledge, practice and community participation relating to child health were assessed. Furthermore, the health system level factors associated with child health services, including training and supervision of health workers providing under-five services, and the observed availability of infrastructure, equipment, supplies and drugs for the treatment of childhood illnesses at health posts and health centres were assessed. Using data from facility registers, we also compared the median number of young infants and children 2–59 months of age who received care in the 3 months before the survey in intervention and comparison areas at baseline and endline. We analysed differences between intervention and comparison areas over time using quantile regression analysis. Difference-in-differences analyses were used to estimate the effect of the OHEP intervention on care-seeking for sick under-five children. Binary outcome indicators were used to capture whether a sick child had sought care or received treatment. The key independent variable for the outcomes of this study was whether the child lived in the OHEP intervention or comparison area. A model was then created that included an interaction term for the timing of the survey (baseline or endline) and the survey area (OHEP intervention or comparison area). This model allowed for the calculation of the odds of care-seeking or treatment for under-five children in intervention areas as compared with comparison areas, accounting for any differences between baseline and endline survey areas, with adjustment for the cluster sampling and identified confounding factors. The Stata V.13 (StataCorp, College Station, Texas, USA) svy commands were used to adjust for clustering. The assessment used a blinded analysis. The code identifying the intervention and comparison areas were revealed after the analysis and interpretations were completed. Patient and/or the public were not involved in the design or conduct, or reporting or dissemination plans of this research.