Theory-driven process evaluation of the SHINE trial using a program impact pathway approach

listen audio

Study Justification:
– The lack of success of programs or interventions can be attributed to poor alignment of interventions with the causes of the targeted problem and failure to implement interventions as designed.
– The Sanitation Hygiene and Infant Nutrition Efficacy (SHINE) Trial aims to address these failures by utilizing the program impact pathway (PIP) approach to track intervention implementation and behavior uptake.
– The PIP approach allows for monitoring and strengthening intervention delivery, facilitating course-correction at various stages of implementation.
– The study will provide a richer understanding of the mediating and modifying determinants of intervention impact.
Highlights:
– The study collects process evaluation data through record reviews, structured observations, and interviews with Village Health Workers (VHWs) and study participants.
– The data collected will characterize supportive supervision, VHW performance, maternal behavioral determinants, and maternal behavior/performance.
– The study will conduct both intention-to-treat (ITT) and per-protocol analyses to examine the impact of the interventions on intermediate outcomes and final outcomes.
– The per-protocol analyses will explore the linkages between earlier and later steps in the program impact pathway.
– The study will identify potential modifiers and drivers of effect heterogeneity to understand the circumstances in which the interventions are most effective.
Recommendations:
– Strengthen intervention delivery by monitoring and providing support to VHWs.
– Enhance VHW performance by providing training and standardized protocols.
– Improve maternal behavioral determinants and capacity through targeted interventions.
– Promote and maintain desired maternal behaviors through continuous support and reinforcement.
– Increase access to water to encourage hand washing with soap.
Key Role Players:
– Village Health Workers (VHWs): Responsible for delivering the interventions and maintaining records.
– Supervisors: Conduct routine inspections and collect data from VHWs.
– Enumerators: Administer questionnaires to VHWs and study participants.
– Research Nurses: Administer questionnaires to participating women during antenatal and postnatal visits.
Cost Items:
– Training and standardization of VHWs, enumerators, and research nurses.
– Data collection tools and materials.
– Supervisory visits and inspections.
– Questionnaire administration.
– Data analysis and interpretation.
Please note that the cost items provided are general categories and not actual cost estimates.

The strength of evidence for this abstract is 7 out of 10.
The evidence in the abstract is moderately strong. The abstract provides a detailed description of the program impact pathway (PIP) approach used in the SHINE trial and outlines the data collection methods and analysis plan. However, it does not provide specific results or findings from the trial. To improve the strength of the evidence, the abstract could include a summary of the main outcomes or preliminary findings from the trial, providing more concrete evidence of the effectiveness of the interventions.

Two reasons for the lack of success of programs or interventions are poor alignment of interventions with the causes of the problem targeted by the intervention, leading to poor efficacy (theory failure), and failure to implement interventions as designed (program failure). These failures are important for both public health programs and randomized trials. In the Sanitation Hygiene and Infant Nutrition Efficacy (SHINE) Trial, we utilize the program impact pathway (PIP) approach to track intervention implementation and behavior uptake. In this article, we present the SHINE PIP including definitions and measurements of key mediating domains, and discuss the implications of this approach for randomized trials. Operationally, the PIP can be used for monitoring and strengthening intervention delivery, facilitating course-correction at various stages of implementation. Analytically, the PIP can facilitate a richer understanding of the mediating and modifying determinants of intervention impact than would be possible from an intention-to-treat analysis alone.

We collect process evaluation data through record reviews, structured observations, interviews with VHWs and interviews with study participants at outcome measurement time points [31]. All enumerators, research nurses, and supervisors were trained and standardized on the various methods. VHWs maintain a module-delivery schedule for each mother they recruit into SHINE. These schedules specify which module should be delivered when, as well as allowable and acceptable windows around the target date. VHWs record when the module was delivered. VHWs also maintain registers in which they record their activities, such as prospective pregnancy surveillance. Supervisors routinely inspect this documentation and collect these data from each VHW during their scheduled monthly supervisory contacts. These data will be used to characterize supportive supervision (frequency of VHW-supervisor contacts) and VHW performance. VHW supervisors conduct structured observations of all VHWs to assess and document VHW interactions with study participants and adherence to behavior change intervention protocols. For each VHW, these observations are conducted during the first delivery of each new behavior change intervention module and quarterly thereafter. The assessment tools consist of Likert-type, multiple-choice, dichotomous, and subjective qualitative items that are used to assess specific behaviors of VHWs. Measures of VHW performance, such as lesson delivery scores, will be derived from these data. Research staff (part-time enumerators) administer a questionnaire to each VHW (following their informed consent as a research subject) 3 times, at baseline, midline, and endline. Data on sociodemographic, supervisory and motivational characteristics [14], curriculum knowledge, [18] and goal-setting capacity are collected. Research nurses administer questionnaires to participating women during 2 antenatal and 5 postnatal visits between recruitment at approximately 14 weeks of gestation and 18 months postpartum. Data collected include sociodemographic information, exposure to behavior change interventions, curriculum knowledge, maternal capabilities for caregiving [38], and WASH and infant feeding behaviors. A questionnaire module ascertains different indicators of household water access: source, type, walking time [44], distance of water for drinking and water for uses other than drinking, and 24-hour recall of household water collection. A composite measure of knowledge-sharing efficacy [18] will be derived from combining data on the curriculum knowledge of participating women with curriculum knowledge of VHWs, to assess VHW performance in knowledge sharing. Also, we will explore the computation of separate WASH and infant feeding behavior scores incorporating the behaviors promoted by the SHINE interventions. Relative socioeconomic (wealth) status will be derived using a principal components analysis that includes data on household assets, income, expenditures, and access to agricultural land at the time of the baseline household visit. A summary of the data collected, data sources, the indicators derived, and timing of data collection is presented in Supplementary Appendix Table 1. The full PIP, from randomized treatment allocation to reduced childhood stunting and anemia, elucidates several intermediate steps, a number of potential modifiers at each step, and different potential measures to characterize each step (including of FOI at delivery/receipt steps such as between the VHW and caregiver or between the caregiver and infant). Above and elsewhere [31], we describe our efforts to collect data that characterize this complex system. However, without making a large number of assumptions, it is infeasible to model this full PIP in a single statistical analysis. Instead, we will carry out a series of separate “partial” analyses that, when taken as a whole, test the theorized links in the PIP [7]. The statistical approaches we use complement the analysis plan for the primary outcomes of the trial [31], applied to the intermediate outcomes in the PIP. More specifically, we will conduct analyses of intermediate outcomes at each step along the PIP: (1) VHW performance capacity; (2) VHW performance; (3) maternal behavioral determinants/capacity; and (4) maternal behavior/performance. We will employ 2 analytical approaches in these analyses: (1) ITT based on the original randomized design and examining each intermediate outcome separately as an endpoint; and (2) per-protocol analyses linking together intermediate steps and conditional on specific prior outcomes or achievements in an earlier step, such as high FOI. For both approaches we will, via interactions, explore the role of pre-specified modifiers. Examples and potential hypotheses to be explored are presented in Table ​Table11. Potential Program Impact Pathway Hypotheses and Their Estimation Strategies Abbreviations: FOI, fidelity of implementation; IYCF, infant and young child feeding; PIP, Program Impact Pathway; VHW, village health worker; WASH, water, sanitation, and hygiene. The ITT analyses will examine the impact of the randomized interventions on an intermediate outcome, one at a time, treating that outcome as an endpoint [15, 20, 22], as well as assessing the role of modifying effects on it. For example, the second intermediate domain of the PIP is VHW performance. We hypothesize that a VHW’s performance of SHINE tasks (completion of module delivery visits, knowledge transfer) will differ according to their treatment assignment, and that the performance of VHWs assigned to implement both the WASH and IYCF interventions will be lower than that of VHWs assigned to implement only the WASH or IYCF interventions. Further downstream, we hypothesize that for the WASH intervention, mothers in households with greater access to water will practice hand washing with soap to a greater extent than households with less access to water. These analyses exploit the randomized design, and ITT estimates will be estimated as described [31] and will provide estimates of the average effect of the interventions (ie, the ITT effects) on the intermediate outcomes according to our program theory (PIP). Collectively, these analyses will address (1) the extent to which each of the 4 intermediate sequential processes were achieved; and (2) what the modifiers of those processes were, including whether the effects were modified by predetermined characteristics. The per-protocol analyses go beyond these intermediate outcome ITT estimates to examine movement along the PIP—that is, the linkages from earlier to later steps in the chain including, in particular, the final outcomes. For example, linking VHW performance capacity to actual VHW performance. Per-protocol analyses will also explore the linkages from earlier steps in the chain to the final outcomes, such as linking FOI of VHW delivery and stunting and anemia (to ascertain the effects among those who received the treatment as intended), and linking FOI of maternal/caregiver delivery to stunting and anemia (effects among those who tried and maintained the treatment behaviors). Conditional on having delivered/received the intervention relevant to the participant’s treatment arm (as defined by indicators for FOI), we will examine the association between the intervention and the outcome in a later step of the PIP, as well as with the final outcomes of the trial. As with the first set of ITT analyses, potential modifiers at each stage can be assessed using interactions. Depending on the starting point, the per-protocol analyses will be based on our categorization of FOI into 2 types—VHW and caregiver. In the first of these, FOI of VHW delivery, we classify participants who had at least 10 of the 15 VHW SHINE scheduled visits, starting at 24 weeks of gestation as having high/adequate fidelity. We standardized the number of VHW visits (15 module delivery contacts) across treatments to ensure that the content, rather than the number of contacts, is what differentiates the treatment groups. A visit is therefore defined by having contact at a scheduled behavior change intervention delivery visit. For FOI of caregiver delivery, we will develop separate and combined compliance indices for the WASH and infant feeding behaviors and apply a similar condition of at least two-thirds of the behaviors implemented. In contrast to the ITT, for these analyses the estimation sample is limited to those following protocol, and for whom effects are hypothesized to be larger. A limitation to this approach is that it no longer fully exploits the randomized design and therefore weakens causal inference. A benefit to this approach, however, is that it allows us to explore more directly the links between improved WASH and infant feeding practices themselves and the final outcomes. In particular, evidence on the linkages along the intermediate stages of the PIP, as well as any dose-response associations in the relationships between VHW delivery of interventions and the final outcomes, can provide additional plausibility to any observed ITT effects. Furthermore, identifying the drivers of effect heterogeneity can elucidate the circumstances, persons, and contexts in which any such effects are likely to be greatest.

N/A

Based on the provided description, it is difficult to identify specific innovations for improving access to maternal health. However, some potential recommendations based on the information provided could include:

1. Implementing a comprehensive training program for Village Health Workers (VHWs) to enhance their performance capacity and knowledge transfer abilities.
2. Developing a system for supportive supervision of VHWs to ensure adherence to behavior change intervention protocols.
3. Utilizing structured observations to assess and document VHW interactions with study participants and monitor their performance.
4. Collecting data on sociodemographic, supervisory, and motivational characteristics of VHWs to better understand their performance and identify areas for improvement.
5. Conducting regular assessments of maternal behavioral determinants and capacity to identify barriers and develop targeted interventions.
6. Implementing behavior change interventions that focus on improving water, sanitation, and hygiene (WASH) practices among households with pregnant women.
7. Promoting and facilitating access to clean water for drinking and other household uses to encourage handwashing with soap and improve overall hygiene practices.
8. Monitoring and evaluating the implementation of interventions using a program impact pathway (PIP) approach to track progress and identify areas for course correction.

These recommendations are based on the information provided and may need to be further tailored to the specific context and needs of the maternal health program.
AI Innovations Description
Based on the provided description, the recommendation to improve access to maternal health would be to implement a theory-driven process evaluation using a program impact pathway (PIP) approach. This approach involves tracking intervention implementation and behavior uptake to ensure alignment with the causes of the targeted problem and to identify potential areas for improvement.

To implement this recommendation, the following steps can be taken:

1. Define and measure key mediating domains: Identify the key factors that mediate the impact of interventions on maternal health. This may include factors such as VHW performance capacity, VHW performance, maternal behavioral determinants/capacity, and maternal behavior/performance.

2. Collect process evaluation data: Use various methods such as record reviews, structured observations, and interviews with VHWs and study participants to collect data on intervention delivery, behavior change, and program implementation. This data can help monitor and strengthen intervention delivery and identify areas for improvement.

3. Analyze intermediate outcomes: Conduct separate analyses of intermediate outcomes at each step along the program impact pathway. This can involve using intention-to-treat (ITT) analyses to examine the impact of interventions on each intermediate outcome separately, as well as per-protocol analyses to examine movement along the pathway and linkages between earlier and later steps.

4. Explore modifiers and potential hypotheses: Assess the role of pre-specified modifiers and explore potential hypotheses related to the impact of interventions on intermediate outcomes and final outcomes. This can involve examining interactions and potential dose-response associations.

5. Use findings to inform program improvements: Use the findings from the process evaluation and analysis of intermediate outcomes to inform program improvements and facilitate course-correction at various stages of implementation. This can help ensure that interventions are effectively addressing the causes of the problem and improving access to maternal health.

By implementing this recommendation, it will be possible to gain a richer understanding of the mediating and modifying determinants of intervention impact and make informed decisions to improve access to maternal health.
AI Innovations Methodology
The article describes a methodology called the Program Impact Pathway (PIP) approach, which is used to evaluate the implementation and impact of interventions in the Sanitation Hygiene and Infant Nutrition Efficacy (SHINE) Trial. The PIP approach involves collecting process evaluation data through record reviews, structured observations, and interviews with Village Health Workers (VHWs) and study participants. These data are used to track intervention implementation and behavior uptake.

The methodology includes several steps:

1. Data collection: Process evaluation data is collected through record reviews, structured observations, and interviews with VHWs and study participants. This includes data on intervention delivery, VHW performance, maternal behavioral determinants, and maternal behavior/performance.

2. Data analysis: The collected data is analyzed using two approaches: intention-to-treat (ITT) analysis and per-protocol analysis. ITT analysis examines the impact of the randomized interventions on intermediate outcomes, treating each outcome as an endpoint. Per-protocol analysis goes beyond intermediate outcomes to examine movement along the Program Impact Pathway, including linkages from earlier to later steps in the chain and the final outcomes.

3. Hypothesis testing: The analysis aims to test the theorized links in the Program Impact Pathway. Hypotheses are developed based on the intermediate outcomes and potential modifiers at each step. The analysis explores the role of pre-specified modifiers and assesses the impact of the interventions on each intermediate outcome.

4. Estimation strategies: Estimation strategies are used to estimate the effects of the interventions on the intermediate outcomes. ITT estimates provide estimates of the average effect of the interventions on the intermediate outcomes. Per-protocol analyses examine the association between the interventions and the outcomes in later steps of the Program Impact Pathway, as well as the final outcomes of the trial.

5. Interpretation of results: The results of the analysis provide insights into the extent to which each intermediate process was achieved and the modifiers of those processes. The analysis also explores the linkages between earlier steps in the chain and the final outcomes, providing additional plausibility to the observed effects.

Overall, the methodology aims to provide a comprehensive understanding of the implementation and impact of interventions in improving access to maternal health. By tracking intervention implementation and behavior uptake, the Program Impact Pathway approach helps identify areas for improvement and informs future interventions and programs.

Share this:
Facebook
Twitter
LinkedIn
WhatsApp
Email