Use of Lot quality assurance sampling surveys to evaluate community health worker performance in rural Zambia: a case of Luangwa district

Background: The Better Health Outcomes through Mentoring and Assessment (BHOMA) project is a cluster randomized controlled trial aimed at reducing age-standardized mortality rates in three rural districts through involvement of Community Health Workers (CHWs), Traditional Birth Attendants (TBAs), and Neighborhood Health Committees (NHCs). CHWs conduct quarterly surveys on all households using a questionnaire that captures key health events occurring within their catchment population. In order to validate contact with households, we utilize the Lot Quality Assurance Sampling (LQAS) methodology. In this study, we report experiences of applying the LQAS approach to monitor performance of CHWs in Luangwa District. Methods: Between April 2011 and December 2013, seven health facilities in Luangwa district were enrolled into the BHOMA project. The health facility catchment areas were divided into 33 geographic zones. Quality assurance was performed each quarter by randomly selecting zones representing about 90% of enrolled catchment areas from which 19 households per zone where also randomly identified. The surveys were conducted by CHW supervisors who had been trained on using the LQAS questionnaire. Information collected included household identity number (ID), whether the CHW visited the household, duration of the most recent visit, and what health information was discussed during the CHW visit. The threshold for success was set at 75% household outreach by CHWs in each zone. Results: There are 4,616 total households in the 33 zones. This yielded a target of 32,212 household visits by community health workers during the 7 survey rounds. Based on the set cutoff point for passing the surveys (at least 75% households confirmed as visited), only one team of CHWs at Luangwa high school failed to reach the target during round 1 of the surveys; all the teams otherwise registered successful visits in all the surveys. Conclusions: We have employed the LQAS methodology for assurance that quarterly surveys were successfully done. This methodology proved helpful in identifying poorly performing CHWs and could be useful for evaluating CHW performance in other areas. Trial registration: Identifier: NCT01942278 . Date of Registration: September 2013. Between April 2011 and December 2013, seven health facilities in Luangwa District were enrolled into the BHOMA project LQAS survey study. Each health facility catchment area was divided into geographical zones, where each zone included up to a maximum of 300 households. Existing and new CHWs from within the communities were recruited through established community participatory methods by engaging traditional leaders and neighborhood health committee members from all villages within the zone to ensure representativeness for all zones. Using the existing structures in each community, both existing and new CHWs were recruited following open advertisement and interviews. Selected CHWs met minimum qualifications of reaching grade 7 and ability to read and write. A total of 70 candidate CHWs were interviewed from which 33 were recruited according to the number of the district zones for the study. Of these 14 were female (age range 18 to 35) and 19 were male (age range 20 to 48). The median educational level among CHWs was grade 12 (range 7–12); see Table 1. CHW survey performance by round, gender and educational level of responsible CHW The CHWs underwent project-specific orientation on a “community care” package. The training package included data entry using mobile phones and collection of demographic data, age bands, tracking and recording of mortalities. They were also taught on how to collect data on HIV, pregnancy, immunizations, and were to conduct various health-related topics. CHWs conducted monthly household surveys with the goal of reaching all households in their respective zones each quarter and thereafter submit electronically the data collected to the data management team based at the BHOMA central office in Lusaka. If they found a patient with a life-threatening condition or a clinical “danger sign”, they were trained to provide start doses of basic drugs from the CHW drug kit such as paracetamol and oral rehydration salts and then refer the patient to the nearest heath centre. Clinical “danger signs” include failure to drink or breastfeed, continuous vomiting, convulsions, lethargy/unconsciousness, chest in-drawing, severe shortness of breath, severe bleeding, and severe palmer pallor. The danger signs were classified according to the Zambia Ministry of Health integrated management of childhood illnesses (IMCI) classification guidelines. Pregnant women who had not started antenatal care were also referred to the maternal child health facility. As part of quality assurance process, CHW supervisors were trained to coordinate the activities of CHWs and to use the LQAS survey questionnaire (see Additional file 1) to confirm CHW household visitation. Based on the households visited by CHWs in the previous quarter, the data management team at the BHOMA central office in Lusaka generated a random list of sampled households according to LQAS methodology [13]. The BHOMA study was reviewed and approved by both the University of Zambia Biomedical Ethics and the University of North Carolina at Chapel Hill Ethics Committees. In each of the selected zone, 19 households were sampled by the CHW supervisors using a standardized questionnaire to be completed by either the head of the household or any member above 16 years as listed in Additional file 1. A sample size of 19 provides an acceptable level of error for decision-making by managers; at least 90% of the time, it identifies areas that have reached the set target or below the average coverage of the programme [17]. Since LQAS uses the binomial formula to calculate smaller samples and to come up with a decision criteria for grouping CHWs by their performance using a three triage assessment system; adequate, inadequate and very inadequate [15]. In this study, the same criteria was used to assign the CHW to the triage system and these were; coverage as been adequate if 75% or more of the targeted households were visited, inadequate if the coverage was between 50 and 75% and very inadequate if 50% or less. For instance in a sample of 19 households if only five or fewer have not been visited, then the CHW is said to have provided adequate coverage. If more than five households have not been visited, the CHW performance is considered as inadequate. The CHW supervisors validated the data by counterchecking the unique household identity number and a summary of the health information discussed. Once the questionnaires were completed, they were submitted to the district study team for further data completeness checks before forwarding the forms to the data management team at the central office in Lusaka. CHW coverage performance was assessed by confirming physical visitation of a CHW to each household listed on the quarterly sample within each zone and delivery of health information on at least one health topic. A threshold was set apriori at 75% as the minimum standard for coverage. To use LQAS decision rules, the study applied two rules for analysis; Thus the LQA sample size depends on performance standards, the classification error and the number of permissible error and all of them are interrelated. A detailed theory can be found elsewhere [15]. Data was collected on paper and captured on a CHW LQAS access database by the data management team at the study central office in Lusaka. The variables analyzed were the mean performance of households % visited in each of the survey round and also the frequency of at least one health topic being discussed by the CHWs. The data was analyzed using descriptive statistics. Chi-squared tests were used for testing associations. ANOVAs were used to compare mean performance between CHWs. Each CHW performance was evaluated and assigned a performance score. Mean performance scores were computed for all the CHWs as a sum of total households visited by each CHW/total households to be sampled × 100 as determined by the data management team at the central office in Lusaka.