The effect of supervision on community health workers’ effectiveness with households in rural South Africa: A cluster randomized controlled trial

Background Community health workers (CHWs) can supplement professional medical providers, especially in rural settings where resources are particularly scarce. Yet, outcomes of studies evaluating CHWs effectiveness have been highly variable and lack impact when scaled nationally. This study examines if child and maternal outcomes are better when existing government CHWs, who are perinatal home visitors, receive ongoing enhanced supervision and monitoring, compared to standard care. Methods and findings A cluster randomized controlled effectiveness trial was conducted comparing outcomes over 2 years when different supervision and support are provided. Primary health clinics were randomized by clinic to receive monitoring and supervision from either (1) existing supervisors (Standard Care (SC); n = 4 clinics, 23 CHWs, 392 mothers); or (2) supervisors from a nongovernmental organization that provided enhanced monitoring and supervision (Accountable Care [AC]; n = 4 clinic areas, 20 CHWs, 423 mothers). Assessments were conducted during pregnancy and at 3, 6, 15, and 24 months post-birth with high retention rates (76% to 86%). The primary outcome was the number of statistically significant intervention effects among 13 outcomes of interest; this approach allowed us to evaluate the intervention holistically while accounting for correlation among the 13 outcomes and considering multiple comparisons. The observed benefits were not statistically significant and did not show the AC’s efficacy over the SC. Only the antiretroviral (ARV) adherence effect met the significance threshold established a priori (SC mean 2.3, AC mean 2.9, p < 0.025; 95% CI = [0.157, 1.576]). However, for 11 of the 13 outcomes, we observed an improvement in the AC compared to the SC. While the observed outcomes were not statistically significant, benefits were observed for 4 outcomes: increasing breastfeeding for 6 months, reducing malnutrition, increasing ARV adherence, and improving developmental milestones. The major study limitation was utilizing existing CHWs and being limited to a sample of 8 clinics. There were no major study-related adverse events. Conclusions Supervision and monitoring were insufficient to improve CHWs’ impact on maternal and child outcomes. Alternative strategies for staff recruitment and narrowing the intervention outcomes to the specific local community problems are needed for consistently high impact. The study took place in the deeply rural King Sabata Dalindyebo Health Subdistrict of the OR Tambo District, Eastern Cape of South Africa [28]. The catchment area was served by 8 primary care clinics that refer patients to Zithulele District Hospital, a state-run facility that serves about 130,000 people [29]. The district is situated in the former Transkei homeland and is one of the poorest and most underdeveloped in the country, ranking below national standards in terms of access to water, sanitation, and healthcare [30]. Only 27% of households have access to communal taps, with 48% relying on unsafe river water, while 93% of households receive a government grant [30]. Prior to randomization, we documented the size of the clinics’ catchment areas, density of housing in the catchment area, the number of women initiating antenatal care at the clinic over the previous year, and the topography of the clinic catchment area (including the average slope). Due to substantial variations between some of the clinics’ catchment areas and size of their catchment populations, UCLA paired 2 sets of 4 clinics each to help create catchment areas that were largely similar for each pair and then used a random number generator to randomly assign one set of the matched pairs to either the Accountable Care (AC) or to the Standard Care (SC) conditions. Local women who lived in adjoining neighborhoods were recruited and trained as data collectors. These isiXhosa-speaking women also spoke English, typically had a 12th grade education, and were selected based on their good social skills, the ability to engage peers, and the responsibility and ability to complete interviews on a digital platform. Data collectors were assigned to clinics during the recruitment period and then conducted all assessments in the participants’ home or outdoors if there was no privacy in the home. Data collectors were blind to intervention condition but may have realized intervention status over time, based on mothers’ answers to assessment questions. All pregnant women over the age of 14 years, presenting to any of the 8 study clinics for antenatal care from June 2017 to September 2018, were invited to participate in the study and signed voluntary informed consent. Mothers’ consent was to participate in a study of maternal and child well-being, and mothers were blind to the supervision status of the CHWs routinely assigned to all pregnant and new mothers. Data collection at follow-up interviews continued until February 2021. COVID-19 led to a break in field operations with no in-home data collection from March to October 2020. During this period, telephone interviews were allowed from July to October 2020. Telephone interviews did not allow anthropomorphic measures to be gathered for 89/1,369 (6.5%) mothers or children on their follow-up interviews. Pregnant women under the age of 18 years provided both youth assent and parent/guardian consent. Women with intellectual impairment or who demonstrated significant psychiatric problems at initial contact were excluded based on the interviewer’s evaluation. Data collectors reviewed all clinic and hospital antenatal and birth records to identify pregnant women and ensure that a consecutive series of 900 pregnant women were recruited; 27/900 women were then found ineligible (underage, not pregnant, recruited twice), resulting in 873 pregnant women. Twins/triplets were excluded because the developmental trajectories of these children have been found to be quite different in previous studies [31]. Given multiple births, miscarriages, still births, and maternal deaths, 50 participants (22/450 in AC; 28/423 in SC) were also excluded prior to the child’s live births. Fig 1 outlines the flow of participants throughout the study. The baseline assessment was conducted during pregnancy, and mothers and their babies were then reassessed at 3, 6, 15, and 24 months post-birth with follow-up rates of 75.5% (605/802; 22 deaths), 84% (669/796; 6 deaths), 81% (640/790; 6 deaths), and 84.2% (666/790), respectively. Only 9/873 participants were not reassessed at least once after the baseline assessment. * Baby also died soon after birth, **Though mother died, follow up of child with caregiver was still performed, therefore denominator was not affected. All interviews were recorded, and a random sample of about 10% of the 3,178 recorded interviews were reviewed for quality by an isiXhosa-speaking researcher. Collected data were reviewed weekly so that errors were queried and corrected. Background characteristics collected included maternal age, whether in school currently, number of years of schooling completed, current employment, and the presence of a live-in partner or husband. Economic resources were identified as present (1) or absent (0) including food insecurity, monthly income (>2,000 South African Rand [ZAR]), access to electricity, and access to safe water. The number of adult and child household members were reported, food insecurity, receipt of the child support grant prior to childbirth for other household members), the number of previous births, HIV status, histories of lifetime suicide attempts, having a chronic illness, lifetime and recent number of sexual partners, and interpersonal violence. We also assessed alcohol use prior to recognizing one is pregnant and during pregnancy post-pregnancy discovery, and problematic alcohol use with the Alcohol Use Disorders Identification Test-C (AUDIT-C) [32] for the 2 time frames. The AUDIT-C is a 3-item scale indicating alcohol consumed in the last year; number of drinks daily when drinking; and number of times having more than 6 drinks in a day. A score >3 is considered problematic alcohol use. Reports of home visits. Mothers reported visits by CHWs at each assessment. Alcohol Use During Pregnancy. Mothers were asked if they ever used alcohol after discovering they were pregnant at the baseline assessment (1) or not (0). Depressive symptoms were measured at each assessment using the Edinburgh Postnatal Depression Scale (EPDS) [31], a measure often administered in South Africa [24,33,34]. The EPDS is a 10-item scale with 4 Likert-type responses for each item (maximum score of 30), with mothers’ self-reports indicating possible depressed mood with scores > = 13 (1) at every assessment (3, 6, 15, and 24 months) or not (0). Antenatal adherence to 4 healthcare visits (1) or not (0) was assessed based on answers at the baseline and the 3-month assessment. Adherence to tasks to Prevent Mother-to-Child Transmission (PMTCT) by MLH. HIV testing identified MLH during pregnancy, confirmed by self-report and by the government-issued Road to Health Card. All pregnant mothers were tested for HIV or were previously identified as HIV seropositive. We created a count of the following tasks the MLH completed: (a) exclusively breastfed for 6 months; (b) gave nevirapine at birth; (c) gave Bactrim for 6 weeks; (d) tested the child for HIV prior to 3 months of age; and (e) went to the clinic to receive the results of the baby’s HIV test. These were self-reported at each assessment and checked on the child’s Road to Health card to identify if each of the tasks were completed (1) or not (0). Adherence to ARV Medication for MLH. At every time point, MLH were asked to rate their adherence to ARV medication on a scale ranging from poor to excellent. The outcome was the number of assessments (that is, 3, 6, 15, 24 months, range = 0 to 4) the mother reported “Very Good” or “Excellent” ARV adherence. Breastfeeding for at least 6 months, which was self-reported by mothers and calculated as yes (0) or not (1) if breastfeeding at both the 3- and 6-month assessment or not (0). Having a low birthweight (LBW) infant (that is, less than 2,500 grams = 0) or not (1) as recorded on the clinic or hospital birth registers or the child’s Road to Health Card. Having a stunted or malnourished child over 24 months (2 outcomes as occurring (1) or not (0) at any assessment at 3, 6, 15, or 24 months). Data collectors carried scales and weighed the child in kilograms and measured their length (centimeters) using a measuring mat. Infant anthropometric data were converted to z-scores based on the World Health Organization’s (WHO) age-adjusted norms for gender [35]. A Z-score below −2 was considered a serious growth deficit: <−2 for height-for-age z-scores (HAZ) was considered stunted (1) or never stunted (0). A standard deviation (SD) of <−2 for weight-for-age z-scores (WAZ) was considered underweight (1) or of weight >−2 SD was considered nourished (0). Securing the child support grant by 6 months (0) or not (1). Immunizations were classed as up to date if those expected were completed at the 6-, 15-, and 24-month time points, based on the South African government directives [36]. Adherence was documented on the child’s Road to Health card. Child hospitalizations were recorded as occurring (1) if the child was ever hospitalized at the 3, 6-, 15-, and 24-month time points on the child’s Road to Health card or not (0). Developmental milestones were based on the WHO stated developmental milestones [26,36] at 6, 15, and 24 months. The outcome was the count of developmental milestones the child completed at all the time points. UCLA randomized matched clinic sites to the SC and the AC intervention conditions. There were 23 CHWs in the 4 SC clinics compared to 20 at the AC clinics; the number varied slightly due to resignations and hiring throughout the study. In particular, at one point in the study, a hiring freeze by the Provincial Dept. of Health resulted in a deficit of 6 CHWs over both conditions. At this point, the study agreed to pay the salaries of 6 new CHWs that the Dept. of Health hired, using their normal advertising, appointment, and training procedures. This was done in order to ensure that we were not evaluating the effect of a hiring freeze, but of CHWs’ implementation of home visiting in typical settings. These new CHWs received exactly the same 1-month training as the other CHWs who had been working from the start of the study. The South African Dept. of Health employs CHWs as salaried, monthly workers with a minimum wage of R21.91 per hour. Each government CHWs was assigned to a specific clinic and the pregnant women living within the specific geographic area served by the clinic. Prior to the study, CHWs had previously received 10 days of in-service orientation and training by government supervisors when reassigned from clinic to community service. An additional month-long training was conducted by the Philani Mentor Mother Program prior to the clinic randomization, with trainers shadowing the CHWs during a 2-week period immediately afterwards on their early home visits to ensure good practice. CHWs provided consent to collect data on their sociodemographic characteristics. All CHWs were trained to provide general information about the key perinatal health challenges: living with HIV, TB, reproductive health, danger signs during pregnancy, typical taught a process for engaging with the mothers—how to enter a house and bond with a mother, how to interview a mother, introduce oneself and one’s role, basic counseling skills, as well as how to monitor maternal and child status. This included growth monitoring, oral hydration, recognizing a child’s breathing problems, teaching the principles of good hygiene, and recording visits. After this training, the ongoing supervision and monitoring of home visits varied across conditions, as summarized in Table 1. AC, Accountable Care; CHWs, community health workers; SC, Standard Care. The CHWs were expected to be in the field conducting home visits for 4 days a week and to report to their clinic team leader 1 day a week. Paper records of the number of visits performed were to have been kept, but there were no supervisory visits in the field and no verification of whether visits took place or not. There was also no help with transport in medical emergencies. While we attempted to obtain records of visits and case assignments by CHWs, we were not successful. The CHWs implementation model in the AC was adapted from the Philani Maternal, Child Health, and Nutrition Programme [37], an evidence-based home visiting intervention model evaluated in 4 studies [24,33,38,39]. There were 3 differences between conditions: (1) CHWs in the AC received support materials to reinforce their prevention messages (backpacks with scales, thermometers, deworming medication, vitamin A, and condoms) and were required to document each home visit in paper logs and/or on mobile phones; (2) supervisors monitored daily implementation of both the paper/mobile reports and made home visits with the CHWs; and (3) supervisors had transport support for medical emergencies. Two experienced Philani supervisors were allocated to provide supportive supervision to the CHWs allocated to the AC, even though the official supervisor remained the government clinic supervisor. Supportive supervision requires the following: the psychoeducational and interpersonal skills to both motivate and support CHWs to conduct effective home visits; the ability to identify and act effectively when either mothers or children are failing to maintain healthy routines; and the ability to hold CHWs accountable for not visiting mothers’ homes nor sharing key information about risk and protective factors for mothers and children. The supervisor randomly dropped in on the typical home visits on an ongoing basis (once every 2 weeks), covering 4 to 6 households a day. CHWs and supervisors then jointly identified at-risk children. These visits allowed them to support and validate the CHWs, as well as provide a role model for coping well with the mother on the home visit. CHWs in the AC group were expected to complete 6 home visits a day and were required to briefly log each visit onsite on a mobile phone with global positioning coordinates. Supervisors regularly checked the logging of home visits through an online portal and were able to quickly identify when CHWs were not out in the field, which was then addressed by a phone call or visit [37]. Supervisors had no power to fire or discipline CHWs; they reported personnel problems to government supervisors. Our primary research question was whether the intervention improved health outcomes. Because there were multiple outcomes of interest, we evaluated all 13 of them independently and created a primary outcome from those analyses. The primary outcome was the number of statistically significant intervention effects (defined as p < 0.025 using a one-sided test favoring the intervention) among the 13 outcomes; this strategy for assessing multiple outcomes in one test was developed by Harwood and colleagues [27]. Because these outcomes are likely correlated, we estimated a correlation of 0.10 as our average correlation between outcomes. Based on the Monte Carlo simulations run by Harwood and colleagues [27], 3 significant outcomes are needed to claim overall significance comparing the SC and the AC. To evaluate the intervention’s effect on each of the 13 outcomes, we fit linear mixed effects model for the continuous outcomes and a logistic mixed effects model for each the binary outcomes. Because the intervention was assigned at the clinic level, we account for that with a random clinic effect; this is standard practice for cluster randomized trials. So, each mixed-effects model included the baseline clinic as a random effect and the intervention as a fixed effect. Because of computational difficulties and the prior assumption that there is clinic-to-clinic variation, we used a penalized likelihood approach for the clinic variance; this is equivalent to estimating via posterior mode with a weakly informative prior [40]. All models were fit with the blme package in R [40] using the default prior distributions on clinic variance. Using these models, we tested whether the outcome was improved in the intervention arm; if the p-value for that one-sided test is 0.025 or lower, it counts as a statistically significant result. We estimated we had sufficient power of 0.80 to detect a small effect size of 0.21 overall omnibus test between the AC and SC conditions by 24 months. We assumed an intraclass area correlation of 0.01 and assumed 80% retention for the power analysis. The study [28] was approved by the Stellenbosch Health Research Ethics Board (N16/05/064) by the UCLA Institutional Review Board (IRB#16–001362), and permission to recruit mothers to the study at primary care clinics was provided by the Eastern Cape Department of Health, South Africa. This study is reported as per the CONSORT extension for cluster trials.