A quality assessment of Health Management Information System (HMIS) data for maternal and child health in Jimma Zone, Ethiopia

Health management information system (HMIS) data are important for guiding the attainment of health targets in low- and middle-income countries. However, the quality of HMIS data is often poor. High-quality information is especially important for populations experiencing high burdens of disease and mortality, such as pregnant women, newborns, and children. The purpose of this study was to assess the quality of maternal and child health (MCH) data collected through the Ethiopian Ministry of Health’s HMIS in three districts of Jimma Zone, Oromiya Region, Ethiopia over a 12-month period from July 2014 to June 2015. Considering data quality constructs from the World Health Organization’s data quality report card, we appraised the completeness, timeliness, and internal consistency of eight key MCH indicators collected for all the primary health care units (PHCUs) located within three districts of Jimma Zone (Gomma, Kersa and Seka Chekorsa). We further evaluated the agreement between MCH service coverage estimates from the HMIS and estimates obtained from a population-based cross-sectional survey conducted with 3,784 women who were pregnant in the year preceding the survey, using Pearson correlation coefficients, intraclass correlation coefficients (ICC), and Bland-Altman plots. We found that the completeness and timeliness of facility reporting were highest in Gomma (75% and 70%, respectively) and lowest in Kersa (34% and 32%, respectively), and observed very few zero/ missing values and moderate/extreme outliers for each MCH indicator. We found that the reporting of MCH indicators improved over time for all PHCUs, however the internal consistency between MCH indicators was low for several PHCUs. We found poor agreement between MCH estimates obtained from the HMIS and the survey, indicating that the HMIS may over-report the coverage of key MCH services, namely, antenatal care, skilled birth attendance and postnatal care. The quality of MCH data within the HMIS at the zonal level in Jimma, Ethiopia, could be improved to inform MCH research and programmatic efforts. Jimma Zone is located in Oromiya region, Southwest Ethiopia, approximately seven hours from Addis Ababa, Ethiopia’s capital. The total population of this Zone is estimated at approximately 3.3 million inhabitants. Jimma Zone is further divided into 21 Woredas or districts. Three of the districts in Jimma Zone were selected for a large cluster-randomized trial designed to address barriers to safe motherhood (ClinicalTrials.gov identifier: {“type”:”clinical-trial”,”attrs”:{“text”:”NCT03299491″,”term_id”:”NCT03299491″}}NCT03299491): Gomma, Kersa, and Seka Chekorsa. Ten PHCUs are located in Gomma, nine in Seka Chekorsa and seven in Kersa, resulting in a total of 26 PHCUs. A PHCU consists of one health centre associated with five health posts. Each health post is managed by two government-employed health extension workers, trained in the delivery of primary care to communities within their catchment area. The primary level of health service delivery and data collection in Ethiopia occurs in PHCUs. The compiled data from PHCUs are forwarded to and reviewed by the district health offices before being submitted to the zonal health office, where the data are entered into the electronic HMIS and made accessible to regional and national levels policy-makers [21]. For our study, we consulted with the Jimma Zonal Health Office and extracted electronic monthly service reports for the 2015 fiscal year (i.e. July 2014 to June 2015) for Gomma, Kersa and Seka Chekorsa district health offices and their respective PHCUs. Monthly reports from the two previous fiscal years (i.e. July 2012-June 2013 and July 2013-June 2014) were also retrieved to assess internal consistency. As part of the baseline evaluation for the above-mentioned cluster-randomized trial designed to address barriers to safe motherhood in Jimma Zone, a community-based population-representative cross-sectional survey was performed with women aged 15–49 years from the three districts, who had had a birth outcome (live birth / stillbirth / miscarriage / abortion) within the same time frame as the HMIS extracted reports. Results from this survey are the subject of a separate publication and are not presented herein. To attain the sample size for the trial, a two-stage sampling strategy was used. Twenty-four PHCUs or clusters were first randomly selected from the 26 available in the three study districts. From each PHCU catchment area, 160 eligible women were then randomly selected from registration lists compiled by volunteer community health workers in each village or kebele. A total of 3,784 women provided information on their past use of maternal and child health services as well as on their experiences when they were pregnant, during childbirth, and after delivery (see survey questions in S1 Appendix). After obtaining informed consent, face-to-face surveys were conducted at the women’s households by trained interviewers. For our purposes, we only consider those participants who reported attending any of the MCH services in one of the PHCUs found in the three districts. This resulted in the exclusion of 2.5% of the total sample of women who attended a health facility located in another district. Recognizing the need for developing countries to regularly evaluate the quality of their routine health information system, the WHO developed several tools to assist in assessing common data quality dimensions, including the Data Quality Report Card (DQRC), which we used in this study. The DQRC considers several data quality dimensions and represents a relatively easy and quick quantitative method to identify inaccuracies and inconsistencies in HMIS data [22]. Four dimensions of data quality are included in the DQRC: completeness of reporting; the internal consistency of reported data; the external consistency of population data; and, external consistency of coverage rates [23]. These four dimensions are further sub-divided into specific elements or indicators (Table 1). Given the unavailability of appropriate district-level census data, we did not assess the external consistency of population data. ANC1 –Antenatal Care First Visit; ANC4 –Antenatal Care Fourth Visit; DTP1 –Diphteria, Tetanus, Pertussis first dose; DTP3 –Diphteria, Tetanus, Pertussis third dose; MCH—Maternal and Child Health; PHCU—Primary Health Care Unit a Definitions adapted from the Data Quality Report Card (DQRC) guideline for this study b We only assessed MCH indicators for which no true zero values would be expected c Women who seek care during their pregnancy are also more likely to seek care for their children [23] d The number of ANC4 visits should either be approximately similar to or lower than the number of ANC1 visits recorded but never higher e The number of DTP3 doses should either be approximately similar to or lower than the number of DTP1 doses administered but never higher f This indicator cannot be calculated at the sub-national (i.e. district) level given the absence of UN population projections at that level. In this data quality assessment, we examined several key maternal health services and immunization coverage indicators recommended by the WHO [23], including: antenatal care first (ANC1) and fourth (ANC4) visit coverage; deliveries attended by a skilled birth attendant (SBA) in health facilities; access to early postnatal care (PNC); and, infant receipt of Diphtheria, Tetanus, Pertussis vaccine first (DTP1) and third (DTP3) dose. We also included two additional indicators: malaria in pregnancy and stillbirth rate. We generated appropriate descriptive statistics for the first three dimensions of data quality according to the DQRC, using percentages, means, and standard deviation estimates. All analyses were performed using SAS statistical software version 9.4. (SAS Institute Inc., Cary, NC, USA). To assess the fourth dimension of data quality (i.e. external consistency of coverage rates), we considered rigorous statistical tests that were not part of the DQRC. We generated scatter plots along with Pearson correlation coefficients and intraclass correlation coefficients to determine whether coverage rates from HMIS and survey data were linearly associated and the amount of variability between the two methods, respectively. We calculated ICCs using the Deyo’s method, as described by Szklo and Nieto [24]. Deyo’s method quantifies the variance of the difference between measurements taken by two scorers or readings (i.e. between-scorers/readings) as well as the variability within the repeated measurements recorded by each scorer (i.e. within-scorer). In this study, the ICC calculations took into account the variability between the two methods of MCH coverage rate estimation (i.e. HMIS vs. cross-sectional survey) and the variances reported within district by each method. We used Bland-Altman analysis, a standard method to assess agreement between two quantitative measurements, to estimate the concordance between MCH indicator values from the HMIS and the survey [25]. Given the number of data points in each district (i.e. ten, nine and seven PHCUs in Gomma, Seka Chekorsa and Kersa, respectively), adherence to the normal distribution might have compromised the results of the Bland and Altman analysis. We therefore estimated the limits of agreement using non-parametric methods [26]. Results using limits of agreement defined by the WHO are also available upon request.