Mlearning in the Democratic Republic of the Congo: A mixed-methods feasibility and pilot cluster randomized trial using the safe delivery app

Methods: Eight BEmONC facilities in central DRC were randomized to either an mLearning intervention or to standard practice (control). Maternal and newborn health workers in intervention facilities (n=64) were trained on the use of smartphones and the French version of the SDA. The SDA is an evidence-based BEmONC training resource with visual guidance using animated videos and clinical management instructions developed by the Maternity Foundation and the Universities of Copenhagen and Southern Denmark. Knowledge on postpartum hemorrhage (PPH) and neonatal resuscitation (NR) and self-confidence in performing 12 BEmONC procedures were assessed at baseline and at 3 months post-intervention. Eighteen qualitative interviews were conducted with app users and key stakeholders to assess feasibility and acceptability of mLearning and the use of the SDA. Maternal mortality was compared in intervention and control facilities using a smartphone-based Open Data Kit (ODK) data application. One smartphone with SDA and ODK was entrusted to intervention facilities for the study period, whereas control facilities received smartphones with ODK only. Results: The analysis included 62 heath workers. Knowledge scores on postpartum hemorrhage and neonatal resuscitation increased significantly from baseline among intervention participants compared with controls at 3 months post-intervention (mean difference for PPH knowledge, 17.4 out of 100; 95% confidence interval [CI]=10.7 to 24.0 and 19.4 for NR knowledge; 95% CI=11.4 to 27.4), as did self-confidence scores on 12 essential BEmONC procedures (mean difference, 4.2 out of 48; CI=0.7 to 7.7). Increases were unaffected by health worker cadre and previous smartphone use. Qualitative interviews supported the feasibility and acceptability of the SDA and mLearning, and the potential for it to impact maternal and neonatal mortality in the DRC. Conclusion: Use of the Safe Delivery App supported increased health worker knowledge and self-confidence in the management of obstetric and newborn emergencies after 3 months. SDA and mLearning were found to be feasible and acceptable to health workers and key stakeholders in the DRC. This feasibility pilot study was a cluster RCT with the health care facility as the unit of randomization. The study followed the Consolidated Standards of Reporting Trials (CONSORT) guidelines for reporting pilot and feasibility trials.25 Using mixed-methods convergent parallel design,26 the principal investigator (PI) conducted qualitative semistructured interviews with app users and key stakeholders.27 Additionally, selected patient outcomes were compared pre- and post-intervention. The DRC Institutional Review Board (IRB) housed at the Protestant University of Congo (UPC) provided ethical clearance for the study in April 2017, as did the Medical University of South Carolina (USA) IRB. The study took place over 3 months (April–July 2017) in 2 health zones (Alunguli and Kindu) in the province of Maniema, an under-resourced area in central-eastern DRC with weak infrastructure and some of the poorest maternal and newborn health outcomes in the country.6 Ten health care facilities constituted sites eligible for cluster randomization owing to their being accessible by vehicle and being designated as EmONC centers supported by the Access to Primary Health Care Project (ASSP). ASSP, led by IMA World Health (IMA), an international NGO, is a health systems strengthening and primary care redevelopment project funded by the UK government. The project is carried out in collaboration with the Congolese government and an array of local and international partners to revitalize the country’s health system in targeted health zones, fight disease, and improve key health indicators, particularly related to maternal and child mortality.28 As designated EmONC centers, the 10 facilities (1 hospital and 4 health centers per zone) have received EmONC commodities and equipment, and personnel have participated in EmONC trainings. Identified facilities were stratified by type into hospital or health center categories (Figure). In the hospital category, 1 facility was selected randomly for intervention, using an urn filled with labeled papers, from the matched group of 2 facilities. One health center was excluded due to being non-functional. In the health center category, 3 centers were chosen randomly from among the 7 matched health centers for intervention and 3 for control, giving a total of 4 intervention and 4 control facilities (N=8). CONSORT Flow Diagram Abbreviations: CONSORT, Consolidated Standards of Reporting Trials; MD, medical doctor. Medical doctors (MDs), nurses, and midwives working in the selected facilities who manage deliveries and newborn care were invited to participate in the study. The study population included 64 health care workers at the 8 selected health care facilities (Figure). Attrition included 2 individuals (MDs) in the intervention group who completed the pretest but were unable to participate in the posttest due to ill health. For this mixed-methods study, the PI conducted qualitative semistructured interviews with 2 categories of professionals for a cumulative total of 18 interviews. The first category of professionals consisted of 10 key stakeholders in Kinshasa (national capitol) and Kindu (capitol of Maniema Province), and the second category comprised 8 app users. For key stakeholders, the researcher used “snowball sampling,” which relies on the personal networks of the persons the researcher taps into for referrals to other key persons.29 Key stakeholders included educational, policy, program and health service leaders in the field. App users were identified as a convenience sample from among users trained on the app from the study facilities in Kindu.26 All study participants provided verbal consent after they were informed about the purposes of the study during a site visit by the research team and were given IRB-approved written information (“Information for Participants Sheet”), assuring the confidentiality of all information obtained during the study and informing them of their right to withdraw from the study at any time without any effect on their employment status. The study participants and the clinic staff were not masked because the intervention required overt participation. Facilities were randomized rather than individuals to avoid contamination among health care workers in the same facility.14 The study included medical doctors, nurses, and midwives working in the DRC who manage deliveries and newborn care. The SDA is a training tool and job aid developed by the Maternity Foundation, University of Copenhagen, and the University of Southern Denmark. It was designed to reinforce the capability and confidence of health care workers in low-income countries on how to manage basic obstetric and neonatal emergencies. The content of the app is based on global clinical BEmONC guidelines and has been validated by an international group of global health experts.15 The SDA can be downloaded free of charge for iPhone at https://itunes.apple.com/dk/app/safe-delivery/id985603707?mt=8 and for Android at https://play.google.com/store/apps/details?id=dk.maternity.safedelivery. The SDA conveys knowledge and skills via animated videos and instructions on key procedures. It also contains information on essential drugs for BEmONC. All features and functions are designed for low-literacy, low-income settings and work completely offline once downloaded. The 10 instruction films include the 7 signal functions of BEmONC as well as 3 additional essential procedures (infection prevention, management of infection in newborns, and active management of the third stage of labor). In this study, the French version of the SDA was pre-downloaded to Android smartphones in the DRC capital, Kinshasa, due to poor Internet connectivity in the pilot region (Kindu, Maniema), and 1 smartphone was allocated per facility. An Open Data Kit (ODK) data collection instrument, purposefully designed by the PI and study authors for this study, was also loaded onto the smartphones to collect information on BEmONC vital statistics and signal function execution, beyond what was normally captured in the District Health Information System 2 (DHIS 2) (to be referred to as the health information system, or HIS), and was to be entered manually by facility staff daily. The Safe Delivery App conveys knowledge and skills via animated videos and instructions on key BEmONC procedures. Staff in participating facilities received explanation of the nature and purpose of the trial. Intervention health care workers received a half-day training session on the use of the smartphone, SDA, and ODK, with joint app video viewing and discussion. At the non-intervention health care facilities, the health care workers provided standard care without the assistance of the SDA. However, training was conducted for the smartphone-based ODK data collection, as data were collected at control facilities in the same manner as at intervention facilities during the study period. To ensure equal possibilities to provide standard care, the availability of a minimum package of drugs and equipment was ensured by ASSP in both groups of facilities. For the 3-month study period, the smartphone with SDA was available to all maternity providers at the intervention facilities. Solar panel battery chargers were given to all 8 facilities with the smartphones to ensure consistent ability to charge. Providers were instructed to use the SDA as often as they wished and that the phone should be made available to the team on duty at all times. Ministry of Health supervisors were tasked with visiting intervention and control facilities weekly to remind providers to use the app and/or the ODK. The Theoretical Domains Framework, which positions health professional behavior change as key to increasing the uptake of evidence into health care practice, was used to guide this research.24 The initial aim of this framework was to simplify and integrate a number of behavior change theories to provide a theoretical lens through which to view the cognitive, affective, social, and environmental influences on provider behavior.30–31 Explanatory constructs from 33 theories of behavior change were reduced and grouped into 14 theoretical construct domains, each of which consists of a grouping of theoretical constructs, which are proposed as potential mediators of behavior change (Table 1).30–31 The Theoretical Domains Framework provides a useful conceptual basis for assessing implementation problems of evidence-based care and understanding provider behavior-change processes.24 In this research, the Theoretical Domains Framework influenced the design of interview questions to explore the specific content of these domains in relation to barriers and facilitators to the use of the SDA, mLearning, and CE implementation in the DRC. It was also used as the coding framework for analysis. The Theoretical Domains Framework With Definitions and Component Constructs30 The Theoretical Domains Framework reduces and groups explanatory constructs from 33 theories of behavior change into 14 theoretical construct domains. The primary outcomes of the pilot SDA trial were self-confidence and knowledge scores by the health care workers. Self-confidence and knowledge data collection instruments were developed, tested (in English), and translated into French by the Maternity Foundation in Copenhagen. Reliability and validity measures have not yet been published for these measures; this study will contribute to the assessment of the measures. Self-confidence scores were assessed for 12 essential BEmONC services. Knowledge scores were assessed for 2 key BEmONC services, management of PPH and neonatal resuscitation (NR) at baseline and at 3 months post-intervention. Additionally, baseline demographic characteristics were collected for the health workers in intervention and control groups. Births, maternal deaths, obstetric complications, and execution of BEmONC signal functions were assessed in intervention and control clusters post-intervention using a smartphone-based ODK application designed for this study by the researchers and piloted with the SDA, as part of an examination of study procedures for a future adequately powered RCT. ODK-generated data were compared with hand-collected statistical data from health facility registers and with HIS data. Feasibility and acceptability of the SDA were assessed through qualitative semistructured interviews with app users. Key stakeholder perspectives on the use of mLearning more broadly in the DRC were also assessed. Data collection was conducted in parallel in the intervention and control facilities using the same methods at baseline and 3 months after the training intervention. Measures were taken prior to the training of facility-based providers on the SDA and included: All data were collected on paper in a classroom setting; knowledge scoring was provided by the SDA. Results were entered in Microsoft Excel and subsequently transferred and analyzed in SPSS (version 23). Three months after the SDA introduction, self-confidence and knowledge were measured a second time in the classroom using the same data collection instruments. The PI developed 2 qualitative interview guides for the 2 qualitative target groups (SDA users and key stakeholders) using the theoretical construct domains of the Theoretical Domains Framework31 to guide the questions (Table 1). Semistructured interviews were audio recorded by the PI with 8 SDA users and 10 key stakeholders after the 3-month study period. SDA users were asked about the feasibility and acceptability of using the SDA and barriers and facilitators to its use. Key stakeholders were asked about the feasibility and acceptability of the use of mLearning and CE in the DRC more broadly, as well as barriers and facilitators to the implementation of CE. Facility-based reporting of selected health outcomes collected with the use of the ODK was compared with data reported in the HIS and with data collected by hand-review of health facility registers by the PI. Data were collected by hand at baseline for the 3 months prior to the intervention and then at 3 months post-intervention for comparison. The ODK app was developed for this research study and piloted during the study period in the 8 intervention and control facilities. The ODK data were entered by the health workers into mobile phones provided by the project immediately post-delivery/event and were available online (after uploading) for consultation from any location. Descriptive summary statistics were analyzed on demographic data including age, gender, profession, educational level, years of experience, number of deliveries performed in the past month, and previous use of smartphone. Given that this was a feasibility study, power calculations were not made in choosing the sample size for the pilot trial. However, the study team did gear the sampling strategy to achieve a minimum sample size of 30 for both intervention and control groups to support the use of parametric statistical tests. T-tests examined within-subject differences on test scores pre- and post-intervention, (where the dependent variable was the score on self-confidence and knowledge tests within the intervention and control groups) and between-group differences in change in self-confidence and knowledge (where the dependent variable was the mean difference in change on scores for the 2 groups). Confidence intervals (CIs) and effect size were calculated. To test for potential confounding, between-group differences were calculated to examine the role of gender on test scores and the role of previous smartphone use. One-way analysis of variance (ANOVA) was used to examine test scores analyzed by the 3 health professional cadres (nurses, midwives, and MDs) across both intervention and control groups. The criterion for significance for all analyses was set at P<.05. All data were entered into Microsoft Excel and analysis was performed using SPSS (version 23). Data coding for both target groups was carried out deductively by the PI, using the 14 domains from the Theoretical Domains Framework (Table 1) as the coding framework for content analysis, in order to interpret meaning from the content of the qualitative data.30–31 Quantitative and qualitative data were interpreted and merged together, noting both the quantitative statistical results and qualitative quotes or themes that supported or refuted the quantitative results.27