Acceptability and Potential Effectiveness of eHealth Tools for Training Primary Health Workers from Nigeria at Scale: Mixed Methods, Uncontrolled Before-and-After Study

Background: The in-service training of frontline health workers (FHWs) in primary health care facilities plays an important role in improving the standard of health care delivery. However, it is often expensive and requires FHWs to leave their posts in rural areas to attend courses in urban centers. This study reports the implementation of a digital health tool for providing video training (VTR) on maternal, newborn, and child health (MNCH) care to provide in-service training at scale without interrupting health services. The VTR intervention was supported by satellite communications technology and existing 3G mobile networks. Objective: This study aims to determine the feasibility and acceptability of these digital health tools and their potential effectiveness in improving clinical knowledge, attitudes, and practices related to MNCH care. Methods: A mixed methods design, including an uncontrolled pre- and postquantitative evaluation, was adopted. From October 2017 to May 2018, a VTR mobile intervention was delivered to FHWs in 3 states of Nigeria. We examined changes in workers’ knowledge and confidence in delivering MNCH services through a pre- and posttest survey. Stakeholders’ experiences with the intervention were explored through semistructured interviews that drew on the technology acceptance model to frame contextual factors that shaped the intervention’s acceptability and usability in the work environment. Results: In total, 328 FHWs completed both pre- and posttests. FHWs achieved a mean pretest score of 51% (95% CI 48%-54%) and mean posttest score of 69% (95% CI 66%-72%), reflecting, after adjusting for key covariates, a mean increase between the pre- and posttest of 17 percentage points (95% CI 15-19; P<.001). Variation was identified in pre- and posttest scores by the sex and location of participants alongside topic-specific areas where scores were lowest. Stakeholder interviews suggested a wide acceptance of VTR Mobile (delivered via digital technology) as an important tool for enhancing the quality of training, reinforcing knowledge, and improving health outcomes. Conclusions: This study found that VTR supported through a digital technology approach is a feasible and acceptable approach for supporting improvements in clinical knowledge, attitudes, and reported practices in MNCH. The determinants of technology acceptance included ease of use, perceived usefulness, access to technology and training contents, and the cost-effectiveness of VTR, whereas barriers to the adoption of VTR were poor electricity supply, poor internet connection, and FHWs'workload. The evaluation also identified the mechanisms of the impact of delivering VTR Mobile at scale on the micro (individual), meso (organizational), and macro (policy) levels of the health system. Future research is required to explore the translation of this digital health approach for the VTR of FHWs and its impact across low-resource settings to ameliorate the financial and time costs of training and support high-quality MNCH care delivery. The e-learning study reported here was embedded within a larger project that combined the video training (VTR) and digitization of health data interventions [13]. Only VTR interventions are reported here. The e-learning component of the larger project involved supplying a computer tablet–based VTR app to 126 rural PHC facilities across three Nigerian states: Federal Capital Territory (FCT), Kano state, and Ondo state [13]. The system enabled the transmission of prerecorded, high-quality training videos and other learning content from a remote server to facilitate the training of rural FHWs on MNCH care, further reducing the need for FHWs to travel to metropolitan cities for training. This larger project included a nonrandomized cluster trial examining the impacts of providing eHealth tools and facilitating infrastructure, specifically satellite communication (SatCom) equipment, to enable remote rural PHC facilities for accessing the internet. This was compared with not providing any eHealth intervention, facilitating infrastructure or any internet access, on routine health service data quality and service provision and use. The data reported in this study relate only to intervention sites that had internet access either via existing 3G mobile networks or through SatCom in facilities without 3G connectivity. For the quantitative component of this study, we used an uncontrolled before-and-after (or pre-post) design to compare whether rural FHWs’ knowledge, attitudes, and reported practices across a range of key MNCH topics changed before and after receiving access to the VTR intervention tools and associated training content. For the qualitative component of the study, we used face-to-face, semistructured in-depth interviews (IDIs) with purposefully selected stakeholders, including FHWs, heads of health facilities, and policy makers, to understand the acceptability, feasibility, and use of computer-enabled VTR to improve health care provision in participating states of Nigeria. IDIs were conducted from February 19 to March 9, 2018 (ie, 12-14 weeks into the implementation of VTR). The study was conducted across three states in Nigeria (Kano, Ondo, and FCT), as outlined in Table 1. Within each state, health facilities were selected purposively (for the wider project) across local government areas, which were assigned as intervention local government areas. We included a total of 126 health facilities in this study, which were subcategorized according to the National PHC Development Agency criteria as PHC facilities, comprehensive health centers, health posts, and basic health centers, and were unequally distributed in number and type across the three study areas (Table 1). Distribution of participating health facilities by their locations in Nigeria (N=126). aFCT: Federal Capital Territory. PHC facilities, health posts, and basic health centers all provide primary-level care, whereas comprehensive health centers provide secondary-level care. Primary-level facilities often serve as the first point of contact for patients and are mainly staffed by community health extension workers (CHEWs) but have no medical doctors or midwives, whereas secondary-level facilities serve as referral centers and are staffed by CHEWs, medical doctors, nurses, and midwives. In Nigeria, CHEWs are FHWs trained for 2 to 3 years in the schools of health technology to provide basic public health services at primary-level facilities and mainly to assist nurses and midwives in their duties [16]. Our study involved two types of FHWs who were targeted by the intervention: CHEWs, who are present in all four types of health facilities in the study, plus nurses and midwives, who are only present in PHC facilities and comprehensive health centers. In basic health centers and health posts, there was typically just one FHW available for the study (often the facility manager), whereas there were typically at least two FHWs available in PHC facilities and comprehensive health centers, as they usually have a mix of cadres present. Members of the research team recruited all selected FHWs after obtaining permission from their facility managers, explaining the study’s objectives to the FHWs and obtaining their consent to participate. This was followed by an orientation on how to use eHealth interventions. The intervention involved providing all recruited facilities with a tablet computer containing a VTR app (VTR Mobile). VTR Mobile allows users to access video, audio, and text-based learning materials through the internet. The educational videos used for this study were developed by Medical Aid Films [17] and Global Health Media Project [18] and accessed via the ORB platform [19] developed by the mPowering FHWs Partnership [20]. The ORB platform hosts high-quality medical content that can be used under a Creative Commons License to train frontline workers via the internet or via downloads to mobile devices. The educational videos provided clear educational content and engaged clinical scenarios focused on MNCH care, specifically antenatal care, basic obstetric care, perinatal care, and postnatal care. We selected the content of the videos in consultation with the relevant state ministries of health. The videos were delivered to the users via a structured VTR mobile program. User log-ins were created and provided by the staff of the eHealth intervention provider, InStrat Global Health Solutions [21], to the study participants to enable them to log in and work their way through the program, which also tracked their progress. We collected all data via the tablet computers used by the FHWs, with the data automatically uploaded onto remote servers before being accessed by the research team. To assess whether there were any changes in FHWs’ knowledge, attitudes, and reported practices related to MNCH care, following access to the eHealth intervention tools and information, all FHWs accessing the VTR mobile system first took a multiple-choice (48 questions) preintervention test (pretest) that assessed their reported MNCH knowledge, attitudes, and reported practices on the following 9 topics: (1) focused antenatal care (5 questions), (2) respectful maternity care (2 questions), (3) warning signs in pregnancy (6 questions), (4) how to use a partograph (5 questions), (5) the prevention of postpartum hemorrhage (5 questions), (6) the management of postpartum hemorrhage in a low-resource setting (8 questions), (7) the manual removal of placenta (5 questions), (8) neonatal resuscitation (5 questions), and (9) how to care for a newborn (7 questions). The postintervention test (posttest) questions were the same as the pretest questions. Questions were aligned to the content included in the e-learning program and the curriculum of the included educational videos. The questions were developed by the research team, which included specialists in practice and training in obstetrics and gynecology in Nigeria. Furthermore, consultation with state governments and policy makers occurred during the study planning to ensure that the curriculum of the e-learning program as well as the pre- and posttest questions aligned with the federal government and WHO guidelines for maternal and child health (eg, staff attitudes and provision of respectful maternity care) and that the pre- and posttest questions were clear and easy to understand. Participants who completed the nine VTR modules were automatically prompted via the tablet to take the posttest. Those who had not completed the posttest after 4 weeks of registering for and starting the intervention received fortnightly mobile telephone reminders from the intervention support staff of InStrat to complete the posttest. Multimedia Appendix 1 outlines the questions asked for each topic. Those who had not completed the posttest by the 18th week received weekly text messages from the intervention support staff. No other tests were conducted outside the pre- and postintervention tests in this study. For each user, the system collected data on whether each question was correctly answered. We then calculated our primary outcome as the overall percentage of questions correctly answered in the pre- and posttests. We also created several secondary outcomes based on the percentage of questions correctly answered in both the pre- and posttests, but within each test topic separately. In addition to the pre- and posttest outcome data, we also collected data on FHWs’ gender, staff type (CHEW or nurses and midwives), facility type (PHC, comprehensive health centers, health post, or basic health centers), SatCom availability at their facility, facility location (Ondo, Kano, or FCT), and the date of their pre- and posttests, which we used to create a variable measuring the number of days between FHWs’ pre- and posttests. We calculated that a sample size of 324 would provide >80% power to detect an overall increase of 20 percentage points between the pre- and posttest scores, assuming the most conservative overall prescore of 50%, using a two-sided hypothesis test with a significance level of 0.05 and assuming a modest, typical design effect of 1.5, in the absence of any comparable or pilot data, and 10% loss to follow-up. To describe the characteristics of FHWs and health facilities in our study sample, we produced relevant descriptive statistics. To estimate the change in the overall FHW test score results between the pre- and posttests, we first fitted a multilevel linear regression model with the outcome of test score (including both pre- and posttest scores for every FHW with complete data) and fixed effects for the test period (pre- or posttest), gender (male or female), staff type (CHEW or nurses and midwives [merged due to low sample sizes]), facility type (PHC or comprehensive health centers, or health post and basic health centers [merged due to low sample sizes]), facility SatCom status (yes or no), state (Ondo, FCT, or Kano), and the number of days between FHWs’ pre- and posttests. The model also included a random intercept for individual FHWs to account for the repeated outcomes within individuals (ie, the pre- and posttest scores) and a separate random intercept for health facilities to account for any clustering effects at the health facility level. Using the fitted model, we then estimated the overall mean pre- and posttest scores, and the overall posttest minus pretest change in test scores (ie, the estimated change from before to after the intervention), along with the associated 95% CI and P values of these means. The means were based on estimated marginal means, also known as least-squares means or adjusted means, as calculated from the fitted model. The estimated marginal means assume a balanced population across all covariates, and when estimating them, we set the only numerical variable in the model (days between pre- and posttest) to its mean value across the sample. We then estimated test score results within the following mutually exclusive sets of subgroups: (1) male or female FHWs; (2) CHEWs, or nurses and midwives; (3) FHWs in PHCs, or FHWs in comprehensive health centers or FHWs in health posts and basic health centers; (4) FHWs in facilities with SatCom available or FHWs in facilities without the availability of SatCom; and (5) FHWs based in facilities located in Ondo, FCT, or Kano states. To calculate these results, for each set of subgroups, we fitted the same multilevel linear regression model described above, but with an additional term for the interaction between the test period (pre or post) and the relevant categorical variable defining the relevant set of subgroups (eg, gender for male or female FHWs). Using each of these models, we then estimated the pre- and posttest scores and the posttest minus pretest change in the test score for each subgroup, along with the associated 95% CI and P values. Within each set of subgroups, we then explored whether the observed changes in test scores between the pre- and posttest periods differed between each subgroup (eg, male vs female FHWs). To do this, we used the same models with interaction terms described above to calculate the differences in estimated test score changes (from pre to post) between the relevant subgroups, taking one subgroup within each set as the reference or comparison group, along with their associated 95% CI and P values (again based on estimated marginal means). Finally, we also calculated adjusted overall posttest scores and their 95% CI for each separate topic covered by the test, by repeatedly fitting multilevel linear regression models with outcomes of each topic-specific posttest score, in turn, and independent variables and random intercepts that were the same as described above for the overall primary outcome analysis, excluding a variable for the test period. For all analyses, we excluded observations (FHWs) if they were missing any outcome or required covariate data (ie, complete case analyses). We calculated CI and P values based on t statistics using the Kenward–Roger degrees of freedom approximation. We checked for adherence or violation of model assumptions using the standard range of residual and influence plots for multilevel linear regression models, but found no issues. All results were calculated using R version 3.5.2 statistical software (R Foundation for Statistical Computing) [22], with all models fitted using the lme4 package [23], and all estimated marginal means calculated using the emmeans package [24]. To assess the acceptability and feasibility of using the VTR mobile education intervention, we conducted face-to-face semistructured qualitative interviews with 34 participants in 3 states—12 FHWs, 12 facility managers, and 10 policy makers. Participants were recruited between February 19 and March 9, 2018. Interviews were conducted by 4 medical doctors (KO, AA, OD, and RMY) and a sociologist (DA), who were trained in qualitative interviewing techniques. Only 1 of the 5 data collectors was a female (OD). The research staff provided study information sheets to potential participants to help them understand the objectives of and decide whether to participate in the study. Participants were given at least 24 hours to express interest in participating in the study. Interview guides (Multimedia Appendix 2) were pretested before they were administered to the field. Interviews, which lasted about 30 minutes each, were conducted in a private setting in the workplace of respondents, audio-recorded, transcribed verbatim, and where appropriate, translated into English for analysis. The framework approach was used for analysis, while allowing for the emergence of new themes. The framework analysis involves the stages of familiarization with data, coding (done by the 5 interviewers above), indexing and charting, mapping, and interpretation [25]. The analysis was performed manually. We drew on the technology acceptance model (TAM) to help explain stakeholders’ acceptance and use of VTR Mobile intervention in the workplace environment [26]. The TAM proposes that an individual’s acceptability of (ie, intent to use) and use behavior (ie, actual use) of a technology is determined by two variables. These are the perceived usefulness of the technology to enhance job performance, and the perceived ease of use of the technology, that is, the effort needed to learn and use a given technology. An individual’s motivation to use an emerging technology is higher if the technology is easy to use. The TAM also proposes that factors such as an individual’s understanding of a technology and organizational support measures have positive effects on the perception of usefulness and adoption of technology. Approval for the study was granted by the University of Leeds School of Medicine Research Ethics Committee (MREC16-178) and the Ondo State Government Ministry of Health (AD.4693 Vol. II/109), the Kano State Ministry of Health (MOH/Off/797/T1/350), and the Federal Capital Health Research Ethics Committee (FHREC/2017/01/42/12-05-17).