Financially incentivized knowledge assessments to improve provider compliance with treatment guidelines: a cluster-randomized controlled trial

Background: Despite increasing access to health care, under-5 mortality remains high in many parts of Sub-Saharan Africa. Interventions to improve quality of care have mostly focused on additional training for medical staff, but generally shown little impact. We will assess the impact of financially incentivized quarterly provider knowledge assessment on compliance with Integrated Management of Childhood Illness (IMCI) protocols in Congo, DRC. Methods: Out of a total of 1738 facilities currently receiving results-based financing under an ongoing health financing program, 110 facilities were chosen for this study. All health care workers providing outpatient services to children under age 5 in these facilities will be included in the study. Facilities were randomized with equal probability to control and treatment. Treatment facilities will receive quarterly medical staff knowledge assessments using interactive vignettes. Performance on these vignettes will be rewarded through financial bonus payments to facilities. A baseline survey of health worker knowledge was conducted in 2018. An endline assessment is scheduled to start in the second half of 2021. The primary outcome of interest is health worker compliance with Integrated Management of Childhood Illness (IMCI) guidelines. Compliance will be verified through direct observation of medical staff-patient interactions. Discussion: This is to our knowledge the first trial assessing whether linking health financing to health care worker performance on knowledge assessments can increase compliance with under-5 case management protocols. Trial registration: ClinicalTrials.gov NCT04634019. Registered on November 18, 2020.

This trial will be conducted in DRC and is nested within an ongoing results-based financing program in the provinces of Kwango, Kwilu, and Mai-Ndombe in DRC. Despite the rapid economic growth in recent years and some improvements in some human development indicators, the DRC has some of the worst health and nutrition indicators in the world. Life expectancy was 58 years only in 2016, and under-five mortality was estimated at 88 deaths per 1000 [12]. The maternal mortality ratio is 670 per 100,000 live births.1 Chronic malnutrition among children under five is common with an estimated 42.7% stunted according to the 2014 DHS, and almost half of children under five are moderately or severely anemic [13]. Evidence from a recent facility survey suggests that general health care worker compliance with under-5 treatment protocols is very low [14] and that areas with low compliance on average face higher mortality rates [15]. All health care workers (medical doctors, nurses, and other clinical staff) providing outpatient care to children under age 5 in 110 randomly selected health facilities will be eligible for the study. Surveyors (study staff) have training either as nurses or medical doctors and will be specifically trained for these assessments. At the beginning of each sick child clinic visit, surveyors will ask both the child’s caregiver and the attending health worker to consent to them sitting in (and documenting) the medical consultation. No additional consents will be collected. In order to make the treatment and control areas as comparable as possible, the study focused only on facilities previously enrolled in a results-based financing program. Under this program, quarterly payments are made to all facilities based on the number or pre-specified services delivered as well as a generic quality score. Facilities in the control group thus are identical in terms of the general monitoring and reporting—the only difference is that a part of the quarterly performance-based payments is made conditional on knowledge test performance in the 55 facilities randomly selected for this intervention. The intervention is a quarterly payment to the health facilities linked to the performance of a randomly selected health care worker on a knowledge-based assessment performed each quarter. Health facilities in the intervention group will receive a supervision visit each quarter. During the supervision visits, one health care worker is randomly chosen for the knowledge assessments and is tested on two randomly chosen vignette cases. For the intervention, 12 clinical vignettes were created. These vignettes cover typical patient cases such as malaria, diarrhea, and respiratory infections and assess health workers’ ability to correctly diagnose and treat hypothetical questions. All patient scenarios were generic and covered typical symptoms seen in everyday clinical practice without any particular adaptations to local settings. All medical staff members were informed that there would be a knowledge assessment based on these vignettes every 3 months and that the results of this assessment would determine the total bonus payment made to the facility. In order to create a quarterly performance score, the scores on the two vignettes are then averaged. Fifty percent of the overall facility quality score is determined by the general quality checklist that captures the basic structural and process features of the facility. The remaining fifty percent are directly determined by the health care workers’ performance on the knowledge assessments. If the overall quality score is below 50%, no quality payments are made. If the quality score is ≥ 50%, facilities can receive a top-up payment of up to 25% of the quantity-based payments. The total bonus percentage is determined by multiplying the quality score (with ranges between 0 and 100%) with the maximum 25% bonus. As summarized in Fig. ​Fig.1,1, any facility with a score below 50% does not get any bonus. A quality score of 50 results in a bonus of 25% × 0.5 = 12.5% − a quality score of 100 results in 25% × 1 = 25% bonus. Incentive structure The knowledge assessments started in the last quarter of 2018 after the completion of the baseline survey. Figure ​Figure22 summarizes the overall timeline of the study as well as key measurement points. SPIRIT overview over study activities No such criteria were established. The knowledge assessment intervention is embedded in the larger PBF program supervised by the Ministry of Health with support from the World Bank. Completion of the knowledge assessment tests is required in order to determine the quarterly financial transfers to the selected facilities and strictly enforced. A robust monitoring system has been developed for the program as the data is used to determine financial transfers to health facilities. Moreover, the knowledge assessment tests are conducted with tablets, and the data is uploaded to the project portal. All subjects participating in the trial will have access to routine health care services throughout the study. None. The primary study outcome is compliance with under-5 diagnostic guidelines. Compliance will be measured through up to 8 direct observations of under-5 sick child visits at each facility collected over a 4-day period. For each visit, 10 diagnostic process steps will be measured. Compliance will be defined as at least 8 out of 10 diagnostic processes completed. Secondary outcomes will be health care worker’s knowledge of IMCI protocols and compliance with the IMCI treatment guidelines. IMCI knowledge will be assessed through a series of vignettes provided to the medical staff. For compliance with treatment guidelines, we will focus on the administration of antibiotics and anti-malarials and referrals to higher level health centers as well as compliance with all diagnostics steps specified in the IMCI guidelines. Primary and secondary outcomes will be measured as part of a facility survey that will be conducted between May and December 2021. All facilities in the treatment and control areas will be visited by an assessor team for 3–5 days, who directly observe all under-5 sick child visits up to the targeted 8 observations during this facility visit. For smaller facilities, we expect only about one visit per day—for larger facilities, the target of 8 visits can likely be completed within less than two days. Similar assessments have been conducted through a baseline survey in 2018. These prior assessments will be used to assess baseline balance; the average 2018 quality scores will also be used as controls in adjusted impact estimates. Baseline facility assessments in the targeted facilities were conducted in 2018. The knowledge assessment was integrated into the quality scoring of selected facilities since the last quarter of 2018. The data collection to determine the intervention’s impact is currently scheduled to be collected in the third and fourth quarters of 2020. A total of 110 facilities participate in the trial. Out of these 110, 50% will be randomly selected for the treatment. Within each facility, up to 8 under-5 sick child visits will be observed. We anticipate an average of 6 direct observations in each facility, resulting in a sample of 660 direct observations for our primary outcome. At baseline, compliance with diagnostic protocols was 10%. The study is powered to detect a 15% point increase in diagnostic compliance (from 10 to 25%) with p = 0.9, assuming alpha = 0.05 and an intra-class correlation of 0.2 (DEFF of 2). All targeted facilities had previously been enrolled in the larger RBF program and were randomly selected for this sub-study. To collect data on the quality of under-5 consultations, the study team will spend four full days at each facility observing health care workers during their outpatient consultations with children under the age of 5 years. Based on previous studies in this area, we do not anticipate refusals to be a major challenge. Randomization was done at the health facility (cluster) level. Within each health zone, 50% of health centers were randomly selected for the intervention using a simple random number draw created in the Stata statistical software package by GS. Given the personal interactions required for the intervention, all treated health care workers will be fully aware of this program. Assessors conducting the direct observations will however not be informed regarding the treatment status of the facilities surveyed. All facilities had previously been enrolled in the PBF program. Random assignment of the intervention was based on a random number draw created by GS. Selected facilities were then visited by the project team and introduced to the intervention. Only the outcome assessors will be blinded to the treatment. All health care workers in the intervention group will be fully aware of the program and treatment allocation. NA—health care workers will be fully aware of the treatment, and we do not foresee any reasons for unblinding assessors. The data collection tools are based on tools developed by the Health Results Innovation Trust Fund for the portfolio of impact evaluations of results-based financing projects (https://www.rbfhealth.org/resource/impact-evaluation-toolkit-provides-hands-guidance). The tools that were further calibrated to local settings are part of the baseline survey conducted in 2015. All tools have been tested through pre-study pilots as well as through the baseline survey and the midline survey in 2015. A copy of the direct observation tool is in the Appendix of this document. We do not foresee any closure of facilities. It is possible that some of the health care workers who were assessed as part of the knowledge tests no longer work at their prior facilities. This will not affect the sample size overall but will be considered when analyzing the data (larger treatment effects expected for medical staff directly selected for the knowledge test). All data will be collected electronically using the Open Data Kit (ODK) platform. The ODK software allows to enhance data quality through pre-specified range checks as well as internal consistency checks. Data will be stored at an encrypted local server. Only the PIs will have access to the identified data, and they will de-identify the data before sharing it with the rest of the research team. All personal information will be removed from data files prior to analysis. NA—no biological specimens will be collected or analyzed as part of this trial.

N/A

• Dima Health