Menu
Menu
Menu
Menu
Introduction eHealth solutions that use internet and related technologies to deliver and enhance health services and information are emerging as novel approaches to support healthcare delivery in sub-Saharan Africa. Using digital technology in this way can support cost-effectiveness of care delivery and extend the reach of services to remote locations. Despite the burgeoning literature on eHealth approaches, little is known about the effectiveness of eHealth tools for improving the quality and efficiency of health systems functions or client outcomes in resource-limited countries. eHealth tools including satellite communications are currently being implemented at scale, to extend health services to rural areas of Nigeria, in Ondo and Kano States and the Federal Capital Territory. This paper shares the protocol for a 2-year project (€EXTEND’) that aims to evaluate the impact of eHealth tools on health system functions and health outcomes. Methodology and analysis This multisite, mixed-method evaluation includes a non-randomised, cluster trial design. The study comprises three phases – baseline, midline and endline evaluations – that involve: (1) process evaluation of video training and digitisation of health data interventions; (2) evaluation of contextual influences on the implementation of interventions; and (3) impact evaluation of results of the project. A convergent mixed-method model will be adopted to allow integration of quantitative and qualitative findings to achieve study objectives. Multiple quantitative and qualitative datasets will be repeatedly analysed and triangulated to facilitate better understanding of impact of eHealth tools on health worker knowledge, quality and efficiency of health systems and client outcomes. Ethics and dissemination Ethics approvals were obtained from the University of Leeds and three States’ Ministries of Health in Nigeria. All data collected for this study will be anonymised and reports will not contain information that could identify respondents. Study findings will be presented to Ministries of Health at scientific conferences and published in peer-reviewed journals. Trial registration number ISRCTN32105372; Pre-results.
The eHealth interventions will be implemented by ‘InStrat’ from March 2017 to March 2019, in collaboration with the State Ministries of Health in Ondo and Kano States and the DOH in the FCT. A successful pilot testing of VTR and CliniPAK apps in Ondo State in 2016 led to scaling up of eHealth interventions to Kano state and the FCT in 2017. In this evaluation study, we have selected two clusters in each state corresponding to LGAs: one LGA with facilities implementing VTR and CliniPAK tools, and the other LGA with facilities not implementing any eHealth intervention. The ‘intervention’ LGAs will be assessed against non-intervention LGAs. Intervention LGAs (see table 1) were selected because they had many primary healthcare facilities situated in areas without access to regular mobile network service. Intervention and control local government areas (LGAs) selected by state SatCom, satellite communication. A total of 126 PHC facilities in intervention LGAs across the three states have, since April 2017, been incrementally supplied with tablet computers loaded with data plans to enable the VTR and CliniPAK interventions. Health workers in these PHC facilities were then trained by InStrat staff to use the tablets. See table 2 for a description of VTR and CliniPAK interventions. Moreover, 75 SatCom facilities in intervention LGAs will be supplied with a broadband global area network link-based SatCom hardware, to enable internet connectivity in the PHC facilities. The remaining 51 non-SatCom facilities in intervention LGAs are already connected via regular terrestrial mobile network operators and so do not require linking via broadband global area network link-based SatCom hardware. Beyond the training to enable staff to use the tablets, InStrat staff will provide ongoing technical support to ensure that SatCom and tablets continue to function and that FHWs capacity is maintained despite attrition. Overview of eHealth tools FHW, frontline health worker; PHC, primary healthcare. The target population for this evaluation study comprise three groups: (1) FHWs and facility heads at intervention PHC facilities; (2) pregnant women at participating PHC facilities and (3) policy-makers. The FHWs will include nurses, midwives, laboratory technicians and community health workers. The study will use a mixed-method design to evaluate the acceptability and effects of novel eHealth tools implemented across three states of Nigeria. The quantitative part of the study will use a non-randomised cluster trial design, collecting longitudinal data before and after the implementation of eHealth tools in intervention facilities to compare with longitudinal National Health Management Information System (NHMIS) data in non-intervention facilities to understand the impact of the eHealth tools on health systems functions and health outcomes. The quantitative arm will also assess the impact of eHealth tools on FHWs’ knowledge in intervention sites only. The qualitative part of the study will enable: (1) process and impact evaluations of satellite connectivity and the scaled-up VTR and CliniPAK interventions in the three states; (2) evaluation of the influence of contextual factors on the implementation of the interventions. Implementation in states from different regions of Nigeria (Ondo in west, FCT in middle belt and Kano in the north) facilitates the examination of different contextual factors that may affect implementation and project outcomes. To assess the relationship between project inputs, processes of implementation and outcomes, we will use the framework in figure 1 to conceptually explore how inputs lead to processes, how processes lead to outputs and how outputs contribute to outcomes and impact.20 Given the significance of context to attaining project results, we will examine the roles of SatCom, VTR and CliniPAK interventions in achieving project effects within a wider context, rather than ascribing changes in results and outcomes to our project alone. To achieve this, we will use insight from analysis of documents review and qualitative interviews (see the ‘Methods of data collection’ section) to assess whether/how the ‘context of implementation’ of the project affects project results. For example, though figure 1 depicts linear and simplified relationships between inputs, activities, outputs and outcomes of the project, we acknowledge that the study findings can be influenced by competing/concurrent maternal and child health interventions in either the intervention or control arm of the study (or both) that were unknown or unanticipated at the time of developing the protocol. We also acknowledge that there can be unintended positive or negative consequences of our interventions that are not currently mentioned in this protocol. Conceptual framework for EXTEND project, Nigeria. CliniPAK, Clinical Patient Administration Kit; HWs, health workers; PHC, primary healthcare; VTR, video training. Three hundred FHWs in 126 intervention sites will be selected to participate in CliniPAK and VTR interventions based on lessons from pilot testing in 2016, alongside resource and logistical feasibility considerations. This will comprise 200 FHWs in 75 SatCom facilities and 100 FHWs in 51 non-SatCom facilities (ie, three from each SatCom facility and two from each mobile network-enabled facility). As part of their orientation, the objectives of the EXTEND project will be explained to FHWs in intervention sites. To minimise possibility of coercion, FHWs at intervention sites will then be approached by a member of the research team a week after their orientation and invited to participate in the project. FHWs who agree to participate in the project will be trained to use the CliniPAK app for the daily documentation of MNCH care. For VTR intervention, participants will be provided with login instructions for completing a pretutorial survey, reviewing an electronic tutorial (see table 2), and completing a post-tutorial survey. Participants will complete a consent form prior to participation. The project plans to provide 4–6 monthly refresher of VTR modules to encourage FHW retention in the study. The evaluation will comprise three phases: baseline assessment within 3 months of start of project, midline assessment at 12 months (March 2018) and endline evaluation at 24 months (March 2019). Baseline assessment was conducted from 23 May to 30 June 2017 in intervention and control sites to ascertain the status of target key performance indicators (KPIs), before full implementation of the project, and involved assessment of three types of data: These phases will compare results with the baseline assessment. During each of these phases, a multidimensional approach will be adopted comprising: A project plan is shown in figure 2. All data collection and analysis during baseline, midline and endline assessments will be done by in-country university partners. Following baseline assessment, the three Nigerian universities produced state-level reports integrating quantitative and qualitative findings for the three states and these formed the dataset for a country baseline report. This approach to reporting will be repeated for midline and endline evaluations, respectively, to make sense of the effects and impacts of eHealth interventions. Project work plan. CliniPAK, Clinical Patient Administration Kit; FHWs, frontline health workers; KPIs, key performance indicators; MNCH, maternal, newborn and child health; PHC, primary healthcare; SatCom, satellite communication; VTR, video training. Additionally, a routine monitoring exercise, led by the University of Leeds, will run alongside the three phases of the study to provide quality assurance for the study. Interim evaluation of data quality (completeness and accuracy) will occur following monthly collation of routine NHMIS data alongside data from CliniPAK and VTR, collected by in-country teams. Monthly data will be collated into quarterly reports and submitted in aggregate form for audit by the University of Leeds monitoring team. This will support identification of discrepancies or irregularities in reporting and facilitate periodic performance reviews against KPIs. While monitoring will occur independently, findings from interim evaluations will be shared with study sponsors to inform project monitoring. To facilitate monitoring and ensure standardisation and consistency of reporting across the three states, a project logical framework (logframe) has been developed, outlining the project’s targets, indicators and means of verification of data collected by PHC facilities in the three states, to track progress towards meeting outputs, outcomes and potential impacts of the projects (see online supplementary appendix 1). In addition to the logframe, we have developed a defined set of KPIs to measure performance of against operational criteria (see online supplementary appendix 2). The KPIs will be monitored through periodic performance reviews and within baseline, midline and endline evaluation. bmjopen-2018-022174supp001.pdf bmjopen-2018-022174supp002.pdf The primary outcome for the trial is a binary facility-level indicator measuring whether the monthly NHMIS indicator ‘total number of ANC visits’ is complete (ie, available through the NHMIS) for every month of the 6-month postintervention period. The secondary outcomes are: (1) binary facility-level indicators of whether the monthly NHMIS indicators ‘total PNC visits’ and ‘percentage skilled birth attendance’ are complete or not for every month of the 6-month postintervention period; and (2) the NHMIS indicators ‘total number of ANC visits’, ‘total number of PNC visits’ and ‘percentage of skilled birth attendance’. For the non-randomised trial based on available resources, we will have 6 clusters in the intervention arm and 5 in the control arm, having a mean cluster size (number of facilities) of 25 and a cluster-size variance of 23. Based on preintervention data, for the primary outcome, we assumed an existing proportion in both arms of 0.18 and an intracluster correlation coefficient of 0.025. Using two-tailed testing at the 5% significance level, this allows us to detect an absolute reduction in the intervention arm to ≤0.01 with >80% power.21 We will analyse the primary outcome, adjusted for covariates, using a two-stage method that accounts for between-cluster variation and is appropriate for cluster trials with relatively few clusters per arm.21 First, we will use a logistic regression model of the primary outcome including our covariates of interest, but excluding the treatment effect, to compute a difference residual for each cluster. Second, we will estimate the intervention effect as the absolute difference in the primary outcome (intervention minus control), and base our inference on the associated (t-statistic based) 95% CIs and p value (two-sided, 5% level of significance). We will analyse all secondary outcomes related to NHMIS indicator data completeness using the same methods. All results will be adjusted for the baseline level of the relevant outcome, calculated as the facility-level proportion of data completeness for the monthly relevant outcome as collected over the 12 months prior to the implementation of the intervention, and for LGA. We will use controlled interrupted time-series analysis to analyse whether there have been any changes in the levels and/or trends of all NHMIS indicators following implementation of the intervention. For all these monthly indicators, we will have 12 months’ worth of preintervention data and 6 months’ worth of postintervention data for both intervention and control clusters. We will analyse all NHMIS indicators, aggregated at the LGA level, using a linear regression model including a time × treatment × period (preintervention vs postintervention) interaction to provide estimates of the changes in level and trend of outcomes before and after the intervention period. If model errors display non-negligible autocorrelation, this will be accounted for using by fitting a generalised least squares model adjusting for AR(1) errors. The models will include a random intercept for individual, potentially nested within a random intercept for facility if necessary. We will estimate the mean change in knowledge score percentage points based on the coefficient for a fixed effect of test time (post-test vs pretest). We will also control for a range of likely influential and potentially confounding covariates: age, sex, staff level (community health extension worker or nurse/midwife), facility type (basic or comprehensive) and state (FCT, Kano or Ondo). We will also explore whether any changes in knowledge scores differ between the following subgroups: (1) FHWs at SatCom vs non-SatCom sites, (2) FHWs at basic versus comprehensive facilities, (3) community health extension workers versus nurses/midwives, and (4) male versus female FHWs. We will again use linear mixed models (including the above covariates) to analyse changes in knowledge scores for each subgroup, and separate linear mixed models (including the above covariates) with an interaction between test time and the relevant subgroup indicator variable to provide estimates of any differences in change in knowledge scores between the subgroup comparisons listed. All inferences will be based on the associated (t-statistic based) 95% CI and two-sided p value (5% significance level) for the relevant coefficients. During each phase of the project, IDIs with policy-makers, facility heads, FHWs and service users will be audio recorded (subject to informed consent), transcribed and where appropriate translated into English for manual data analysis. Framework approach will be used for understanding the impact of eHealth interventions on health system functions, while allowing for emergence of new themes. The framework approach includes the stages of familiarisation with data, coding, indexing and charting, mapping and interpretation of data.22 Quantitative and qualitative findings will be integrated and triangulated to answer the research questions. Furthermore, we will conduct a comparative analysis of variations in adoption and effectiveness of eHealth innovations in the three states to ascertain the influence of contextual factors on processes of implementation and project outcomes. The two datasets will be repeatedly triangulated especially during the midline and endline evaluations to understand the impact of interventions on health systems functions and health outcomes. The project will be conducted with full respect for relevant legislations (eg, the Charter of Fundamental Rights of the European Union) and international conventions (eg, Helsinki Declaration). Data collection and analysis will take account of four key issues: The University of Leeds team will compile and analyse data collected by university teams in Nigeria and support their training, including providing information on protocols for anonymising and securely sharing study data. Data will be shared using online secure portals and will be stored with passwords and access only made available to data for those directly involved in data analysis. All transcripts from the study will be anonymised prior to sharing with the Leeds team. We will preserve the anonymity of study participants at all times. Unnecessary collection of personal data will be avoided, and respondents will have the right to review outputs and withdraw consent. Where personal data are collected (eg, age, sex, level of education), it will be coded, removed from the data for analysis and stored separately. Only designated project staff will have access to the keys linking the data with the personal information. We will routinely review the independence of the research team when undertaking monitoring work. While working closely with partners in the consortium, we will ensure that we are free of influence over the judgements relating to the evaluation. A comprehensive evaluation framework has been developed to direct data collection in the study. The project will also develop a shared online platform to facilitate data capture and reporting of variables for monitoring KPIs across project sites. The framework is impartial to any group and inclusive of all groups. The project will be implemented according to standard governance practice at the University of Leeds for the implementation of collaborative projects. This includes ensuring regular communication between partners and engagement with policy-makers and practitioners; quality assurance through regular peer-review both within and between teams; appropriate mentoring and coaching support of junior researchers. Improving MNCH knowledge and practice is a national and international priority. This initial scale-up of eHealth interventions to the FCT, Ondo and Kano states will be further expanded to other states of Nigeria and to non-health sectors (education, agriculture and civic identity management). The high demand for this study from policy-makers and funders provides an excellent opportunity to ensure uptake of high-quality evidence into policy and practice. Specific methods of communicating study findings include a combination of the following: We will ‘embed’ the research strategy development and assessment into policy and practice, working with the Federal Ministry of Health and State Ministries of Health in Ondo and Kano States and the DOH in the FCT. This embedded approach, developed by the Nuffield Centre of the University of Leeds, has been used in many countries to improve the quality and effectiveness of scaled-up programmes.23–25 We will engage decision makers throughout the process in a research-policy partnership to facilitate adoption and scale-up of eHealth tools to other states in Nigeria.26 Patients were not involved in the development or design of the study. We will work with patient advocacy groups to ensure that plain language summaries of study findings are shared to both participating service users and wider patient groups.
