Evaluating the effect of a community score card among pregnant and breastfeeding women living with HIV in two districts in Malawi

Malawi faces challenges with retaining women in prevention of mother-to-child HIV transmission (PMTCT) services. We evaluated Cooperative for Assistance and Relief Everywhere, Inc. (CARE’s) community score card (CSC) in 11 purposively selected health facilities, assessing the effect on: (1) retention in PMTCT services, (2) uptake of early infant diagnosis (EID), (3) collective efficacy among clients, and (4) self-efficacy among health care workers (HCWs) in delivering quality services. The CSC is a participatory community approach. In this study, HCWs and PMTCT clients identified issues impacting PMTCT service quality and uptake and implemented actions for improvement. A mixed-methods, preand post-intervention design was used to evaluate the intervention. We abstracted routine clinical data on retention in PMTCT services for HIV-positive clients attending their first antenatal care visit and EID uptake for their infants for 8-month periods before and after implementation. To assess collective efficacy and self-efficacy, we administered questionnaires and conducted focus group discussions (FGDs) pre- and post-intervention with PMTCT clients recruited from CSC participants, and HCWs providing HIV care from facilities. Retention of HIV-positive women in PMTCT services at three and six months and EID uptake was not significantly different pre- and post-implementation. For the clients, the collective efficacy scale average improved significantly post-intervention, (p = 0.003). HCW self-efficacy scale average did not improve. Results from the FGDs highlighted a strengthened relationship between HCWs and PMTCT clients, with clients reporting increased satisfaction with services. However, the data indicated continued challenges with stigma and fear of disclosure. While CSC may foster mutual trust and respect between HCWs and PMTCT clients, we did not find it improved PMTCT retention or EID uptake within the short duration of the study period. More research is needed on ways to improve service quality and decrease stigmatized behaviors, such as HIV testing and treatment services, as well as the longerterm impacts of interventions like the CSC on clinical outcomes. The CSC approach is a participatory forum that consists of five core phases (illustrated in Fig 1 and described in detail elsewhere [16,17]), four of which are repeated on a regular basis (called “rounds”), during which service users and service providers separately identify issues and then jointly propose solutions and monitor implementation of the solutions. For this project, CARE-Malawi staff implemented three rounds of issue identification and implementation periods of four months duration from September 2017-September 2018. This project adapted elements of the CSC approach originally aimed at broad-based community engagement to focus on engaging HCWs and clients of PMTCT clinical services to identify and solve PMTCT-related issues. The adaptation process and lessons learned are described elsewhere [19]. HCWs were recruited from the intervention clinics and PMTCT clients were identified through the intervention clinics and associations of people living with HIV within the catchment areas surrounding the intervention clinics and invited to participate in the CSC intervention. For the first round, HCWs and PMTCT clients separately identified the issues and barriers most impacting their ability to successfully deliver and access quality PMTCT services, then grouped these issues into indicators. These indicators were then scored by both HCWs and PMTCT clients and used to develop the individual score cards. HCWs and PMTCT clients then came together with other community and government leaders to present their scores and develop, implement, and monitor specific actions for improving PMTCT service delivery and access. Examples of specific actions identified were increasing the availability of trained HCWs providing PMTCT services and adequate infrastructure, supplies, and equipment. In subsequent rounds, previous issues were revisited in order to assess progress made as a result of the action period and re-scored, and action plans were updated. In the CSC approach, participants’ perception of the quality of services meaningfully affects health-seeking behavior including decisions to remain in care [20,21]. A total of 822 PMTCT clients and 64 HCWs participated in the intervention, with participation varying by round. The 11 health facilities in Ntcheu and Dedza districts (9 health centers and 2 district hospitals), were selected based on several criteria, including: 1) at least 25 HIV-positive women receiving HIV services annually; 2) less than 85% retention of HIV-positive pregnant and breastfeeding women in PMTCT services at six months after ART initiation; and 3) lower than national average uptake of EID. Ntcheu and Dedza district hospitals were included to test the CSC intervention in a setting with high patient volume. We used a mixed methods, pre- and post-intervention cross-sectional design to assess four key outcomes: (1) retention in PMTCT services, (2) uptake of EID, (3) collective efficacy among clients, and (4) self-efficacy among HCWs in delivering quality services. To assess the effect of the CSC intervention on clinical outcomes (early retention in PMTCT for newly and previously diagnosed HIV-positive pregnant women and EID uptake), we abstracted data from routine health records maintained by the clinics in the study. The study used clinical records to determine retention in PMTCT services for HIV-positive pregnant and breastfeeding women and EID uptake for their HEIs in the periods before and after implementation of the intervention. Women did not have to participate in the CSC intervention to be included in the abstraction, as it was hypothesized that CSC-related clinical improvements may affect the quality of care, PMTCT retention, and EID uptake for all PMTCT clients. To assess changes in collective efficacy and self-efficacy among the women and HCWs, we administered quantitative questionnaires and conducted pre- and post-intervention focus group discussions (FGDs) with PMTCT clients and HCWs engaged with the CSC intervention. We selected this mixed methods approach in recognition of the complex theories of change behind social accountability interventions [22]. Each component was conducted to provide unique information on the effectiveness of the intervention. By examining these findings together in one paper, we attempt a complete examination and discussion of the entirety of the findings to better understand the complex model of change. In accordance with the MOH guidelines at the time of the study, pregnant women underwent “opt-out” HIV testing (default testing unless the woman refused testing) at first antenatal care (ANC) visit or delivery, and were immediately offered ART if found to be HIV-positive [23]. HIV-positive pregnant or breastfeeding women obtained monthly ART refills from maternal and child health or HIV clinics. HEIs were offered DNA PCR testing at 6–8 weeks of age. For the abstraction of data from existing clinical records, women were eligible for inclusion if they were HIV-positive (newly or previously diagnosed), aged 15 years or older, and attended their first ANC appointment and received ART between August 2016-March 2017 (pre-intervention) and between August 2017-March 2018 (post-intervention) in one of the 11 selected facilities. All eligible women were identified using the clinic’s ANC register and assigned a unique study identifier. All documented live births of the identified women were eligible for inclusion and listed with a linking identifier. We targeted a sample size of women and infants to detect a 10% change in six-month retention in PMTCT services and uptake of EID at 6–8 weeks, from baseline rates of 74% and 56% respectively. Given the hypothesized effect size, our power analysis determined a sample size of 481 women and 481 infants at pre- and post-intervention (power = 0.80, 2-tailed, α = 0.05, potential attrition = 20%) across the 11 sites was needed. Surveys and FGDs were administered to CSC participants. PMTCT clients were eligible for inclusion if they were HIV-positive (newly or previously diagnosed), aged 15 years or older, and were currently pregnant or breastfeeding. All eligible PMTCT clients were approached and briefed about the survey at the first and last CSC meetings. Those who expressed interest were administered a consent script in a private place. Women who consented were invited to participate in the survey on the same day as the CSC meeting. Participants for the FGDs were selected from CSC meetings using a lottery method. Attendees of the CSC meetings were provided with a number, and those who received numbers 2–6 were invited to participate in an FGD. Three FGDs with HCWs and three FGDs with the women were conducted in each of the two districts for the pre and post period. Those interested were given further details of the FGD. On the day of the FGD, eligibility criteria were reviewed for the women who came. Eligible women provided written consent. HCWs were eligible for inclusion in the surveys and FGDs if they were 18 years and older and were working in HIV care in one of the selected facilities for at least one month. Eligible HCWs were recruited through the facilities. All eligible HCWs were invited to participate in the collective action survey and the FGDs. Booking for interviews and FGDs was conducted simultaneously, and consent was administered prior to each activity. With expected attendance of 300 PMTCT clients at CSC sessions and 65 eligible HCWs, we determined we would have at least 90% power to detect a 5% change in empowerment indicators in the survey. The subsets of these groups who participated in the FGDs were deemed sufficient in size to reach thematic saturation. Trained research assistants abstracted clinical data from the ANC, ART, and maternity registers, as well as from the facility-maintained patient files documenting ART treatment for all eligible PMTCT clients. Information abstracted included sociodemographic characteristics, pregnancy information, birth outcomes, and visit information from up to seven ART visits recorded in the ART register. All HEIs of eligible PMTCT clients had their clinical information abstracted from the EID DNA PCR logbook and infant patient files. Information abstracted included gender, birth weight, up to six months of visit information, and HIV testing and results. Abstraction for both populations was conducted from November 2017-January 2018 (pre-intervention) and October 2018-December 2019 (post-intervention). Research assistants recorded abstracted data on a paper form, which was then entered into an Epi Info v7.2 database. Research assistants orally administered the surveys in Chichewa or English. The PMTCT client survey included questions on sociodemographic variables, collective efficacy, and at post-intervention only, participation in the CSC intervention. The HCW survey included questions on sociodemographic variables, work responsibilities, self-efficacy for collective action, social cohesion (mutual aid, trust, connectedness, and social support), work attachment, social participation, collective efficacy, and mutual responsibility. Seven research assistants were trained and responsible for conducting the FGDs in the role of moderator or notetaker. Research assistants recorded the responses on paper forms, which were then entered into an Epi Info database. Research assistants facilitated the FGDs in Chichewa for PMTCT clients and English for HCWs. PMTCT client FGDs explored satisfaction with and perceived quality of health services, and relationship and trust with HCWs. HCW FGDs explored service challenges and accountability, and relationship and trust with clients. To reduce social desirability bias, the FGDs used anecdotal scenarios called “vignettes” to talk about the experience of new PMTCT clients and new HCWs. The vignettes focus on a young woman pregnant for the first time, attending her first ANC visit, and her experiences at the health facility, from her reception to her HIV-positive test, counseling, and drug pick-up. The scenario includes her perception of the services, and treatment by HCWs. At post-intervention, both populations were asked about the effect of the CSC on services. Prior to each discussion, FGD facilitators presented discussion guidelines, according to the standard operating procedures. In addition to assurances of the confidentiality of the discussions, the guidelines included the importance of confidentiality and the discouragement of sharing of personal information, especially for PMTCT clients. FGDs were audio recorded, transcribed, and for PMTCT client FGDs, translated to English. Ethical approval was received from the Malawi National Health Sciences Research Committee, Advarra Institutional Review Board (IRB), and the CDC IRB. We obtained a waiver of informed consent for clinical data abstraction, as data was collected as part of routine care and it was not feasible to seek client permission. Informed consent was obtained from the clients and HCWs who participated in the questionnaire surveys and FGDs. In Malawi, pregnant women aged 15–17 years are emancipated minors and thus can provide informed consent for themselves and their infants. Clients provided written consent or a thumbprint if they were unable to sign their names, while HCWs provided written consent. We adapted validated measures from the “Women’s and Health Worker Voices in Open, Inclusive Communities and Effective Spaces (VOICES): Measuring Governance Outcomes in Reproductive and Maternal Health Programmes” [24] to make them more explicitly related to PMTCT services. We adapted measures of collective efficacy for PMTCT clients, and self-efficacy, social cohesion, work attachment, collective efficacy, and mutual responsibility for and support of services for HCWs [25,26]. The modified collective efficacy scale consisted of four items: How sure are you that the community can work together to (1) obtain government services and entitlements; (2) improve PMTCT services in this community; (3) improve how HIV -positive pregnant and postpartum women are treated at the health facility; and (4) improve the health and well-being of HIV-positive pregnant and postpartum women and their children. Collective efficacy was measured on a 5-point unipolar Likert scale from not at all sure to completely sure. The Cronbach alpha for the modified measure was 0.76, demonstrating sufficient internal consistency and scale reliability [27]. Self-efficacy for delivering quality health/PMTCT services was measured with two (modified) items: (1) how sure are you that you personally can do things to improve your own performance at work; and (2) how sure are you that you personally can do things to improve the quality of PMTCT services in the health facility and the catchment area. Self-efficacy for participation was measured with three items: (1) how sure are you that you can speak up in the community or health facility meetings about things that need improvement; (2) how sure are you that you can ask people in the community what HIV/PMTCT services their community needs; and (3) how sure are you that you can answer questions and share information with the community about PMTCT services available. All were measured on a 5-point unipolar Likert scale from not at all sure to completely sure. The Cronbach alpha for self-efficacy for delivering quality PMTCT services was 0.78 and 0.81 for participation. Social participation was essentially an indicator of participation in the CSC process with questions on whether, in the past six months, they had met with the Village Health Committee, Health Center Committee, or community members to discuss and work on PMTCT issues. Participants were also asked if, in the past six months, they had meetings between the community, HCWs, and district government authorities where problems or other issues with health services were discussed, and plans for improving health services had been made to assess whether the collective action process had been implemented in full. Abstracted clinical data were analyzed by calculating the percent of women retained in PMTCT services by timing of diagnosis (newly or previously diagnosed) and duration of time on ART (three or six months). Retention for this study was defined as having attended the most recent scheduled ART visit before the specified time (three or six months from first ANC), with a seven-day grace period. Chi square tests and logistic regression modeling were performed to examine the relationship between timing of diagnosis at both pre- and post-intervention and to compare PMTCT retention before and after the CSC intervention. We adjusted for type of health facility (i.e., hospitals or health centers) in the analysis. Similarly, the percent of HEIs tested with DNA PCR by eight weeks of age was calculated for pre- and post-intervention and compared. EID uptake was defined as having a recorded DNA PCR test in the infant’s patient file or the EID DNA PCR logbook. We used generalized estimating equation (GEE) modeling to examine potential clustering. To account for the clustering of individuals within sites, we considered a compound symmetry working correlation structure in the GEE model. The correlation coefficient was negligible (close to 0), so we decided to proceed with logistic regression modeling and chi square tests. All statistical tests were conducted on the combined data for all sites and used p < .05 as the significance level. The collective and self-efficacy measures were composed of items presented as questions that employed Likert scales with positively and negatively worded statements. These data were analyzed by summing the values for each item (a measure was composed of one or more items) and dividing by the number of items. For each measure, at both pre- and post-intervention, a mean and standard deviation (SD), and median and interquartile range (IQR), were determined. We assessed pre-/post-intervention differences using the Wilcoxon-Rank Sum test, as we found that the data were not normally distributed. Analyses were conducted using STATA v12 and SAS v9.4. FGDs were transcribed and translated into Word documents and uploaded into the qualitative analysis software program MaxQDA v12. A detailed code list was developed by the study team and used in the analysis of the pre- and post-intervention data. The pre-intervention data was analyzed after the first round of data collection, the post-intervention data was analyzed after the post-intervention data were collected, and then a comparison analysis was done to compare the results of the pre-/post-intervention data. This was done by comparing the data reduction and summary tables (matrices). The matrices compared the perspectives of the two populations at the two different time points. Data were compared pre- and post-intervention and changes in service quality, client satisfaction, and HCW-PMTCT client relationships were summarized, including specific changes attributed to the CSC intervention and process. Data were coded by two research assistants for the pre-post analysis. For the comparison analysis, the study team reviewed the matrices and identified the changes pre-/post-intervention together. Reviews of coded transcripts were conducted throughout the coding process to ensure inter-coder reliability. This was done by comparing the coding of the same transcripts by the two research assistants.