Multisite, mixed methods study to validate 10 maternal health system and policy indicators in Argentina, Ghana and India: A research protocol

Introduction Most efforts to assess maternal health indicator validity focus on measures of service coverage. Fewer measures focus on the upstream enabling environment, and such measures are typically not research validated. Thus, methods for validating system and policy-level indicators are not well described. This protocol describes original multicountry research to be conducted in Argentina, Ghana and India, to validate 10 indicators from the monitoring framework for the ‘Strategies toward Ending Preventable Maternal Mortality’ (EPMM). The overall aim is to improve capacity to drive and track progress towards achieving the priority recommendations in the EPMM strategies. This work is expected to contribute new knowledge on validation methodology and reveal important information about the indicators under study and the phenomena they target for monitoring. Validating the indicators in three diverse settings will explore the external validity of results. Methods and analysis This observational study explores the validity of 10 indicators from the EPMM monitoring framework via seven discrete validation exercises that will use mixed methods: (1) cross-sectional review of policy data, (2) retrospective review of facility-level patient and administrative data and (3) collection of primary quantitative and qualitative cross-sectional data from health service providers and clients. There is a specific methodological approach and analytic plan for each indicator, directed by unique, relevant validation research questions. Ethics and dissemination The protocol was approved by the Office of Human Research Administration at Harvard University in November 2019. Individual study sites received approval via local institutional review boards by January 2020 except La Pampa, Argentina, approved June 2020. Our dissemination plan enables unrestricted access and reuse of all published research, including data sets. We expect to publish at least one peer-reviewed publication per validation exercise. We will disseminate results at conferences and engage local stakeholders in dissemination activities in each study country. This observational study explores the validity of 10 indicators from the EPMM monitoring framework. It uses mixed methods, including (1) cross-sectional review of secondary policy, legal and regulatory data, (2) retrospective review of facility-level patient and administrative data and (3) collection of primary, quantitative, cross-sectional data from health service providers and clients. Standard approaches for assessing the validity of policy and health system indicators are not available; therefore, we developed a specific methodological approach to validate each indicator, tailored to test the validation questions that reflect the specific aims and research questions relevant to each indicator undergoing validation and its underlying construct. Because there is no standard approach (metric or framework) for assessing validity of indicators of upstream health system functionality, we have developed a tailored analytical plan with appropriate statistics to compare the values of the reported indicators to evidence collected in each case. In two specific cases, two indicators designed to monitor a similar construct are compared with each other to explore their convergence and whether indicator adjustment could improve measure validity for that construct. These two indicator pairs share the same validation research questions and are studied in tandem. Thus, the validity of the 10 EPMM indicators is evaluated via seven separate assessments, or validation exercises. The 10 EPMM indicators under study and the specific validation research questions for each indicator appear in table 2. Nine indicators will be validated in all countries, and one additional indicator is to be validated in Ghana only due to local interest. Data collection began in January 2020 was suspended due to COVID-19, resumed May 2020, and is expected to be completed by November 2021 in all settings. Indicators for validation and validation questions B/CEmONC, Basic and Comprehensive Emergency Obstetric and Neonatal Care; BEmONC, basic emergency obstetric and neonatal care; EmOC, Emergency Obstetric Care; EmONC, Emergency Obstetric and Neonatal Care; ICM, International Confederation of Midwives; ILO, International Labour Organization; MPs, Midwifery Professionals. The research will be coordinated by a multicountry team of partners from all three countries and the USA. Country partners were selected through a competitive process based on proposal strength and geographic diversity. One application was selected from Africa, Asia and Latin America/Caribbean, respectively, based on World Bank classification.15 The research will comprise national and subnational data; however, fieldwork will be conducted in subnational settings in each country. Four districts/provinces in each country were selected for primary data collection. Sites were selected through a purposive, two-staged sampling approach based on a composite index of key maternal health indicators reflecting antepartum, intrapartum and postpartum care coverage and MMR, used as a proxy of health system performance. First, one state/region in the highest-performing quartile of the index and one state/region in the lowest-performing quartile were selected. Second, one highest-performing district/province and one lowest-performing district/province were selected within each state/region. In Argentina, some adjustments to the standard site selection protocol were implemented. Due to low population density, terciles were used. In addition, because there was almost no geographic variability in skilled birth attendance and early postnatal care coverage in data from Argentina where most births take place in facilities, Uterotonic Administration at Birth was substituted in the index for this country. Finally, to avoid over-representation of data from Buenos Aires province due to its disproportionate size (total population of over 16.5 million), Region V of the province was selected in consultation with the National Ministry of Health to represent the province. Region V of Buenos Aires province comprises 13 counties, a total population of 3 432 962, 16 hospitals and 319 primary health centres and reflects similar sociodemographic, geographic and health system characteristics as the entire province in table 3. National and subnational research settings Data required for validation vary by indicator; details of the data sources, participants and sampling for each indicator are presented in table 4. Participants and sampling plan detailed by validation exercise B/CEmONC, Basic and Comprehensive Emergency Obstetric and Neonatal Care; GIS, Goegraphic Information System; PSU, primary sampling unit. In general, three types of data will be collected: policy/administrative, facility, and individual data. We will systematically search for national and subnational policies, laws and regulations through a comprehensive desk review of relevant source documents in each country. Country research teams will consult with subject matter experts and data custodians to ensure that all relevant documents were captured. Country-specific data will also be collected from global databases and repositories, as required by each indicator. Furthermore, administrative and patient-level data will be collected from district/provincial-level health management information systems (HMIS). Facilities will be selected based on data requirements for each indicator, using a multistage sampling plan (figure 2). In the first stage, we will conduct a census of all public and private registered health facilities in each study district/province. For some indicators, data will be collected from all facilities in the census. Next, we will determine which maternal health-related services are provided at each facility in the census. We will collect information on provision of services within the five categories in the WHO Maternal Newborn Child and Adolescent Health (MNCAH) Policy Survey: (1) caesarean section, childbirth (normal delivery), delivery-related pharmaceutical products and medical supplies, (2) family planning, (3) antenatal care and insecticide-treated bed nets, (4) postnatal care for mother, (5) testing and treatment for sexually transmitted infectious diseases and cervical cancer screening.16 Although infertility management is included in the WHO MNCAH Policy Survey, it is not in our study. Schematic of standard sampling plan for facilities. DHS; Demographic and Health Surveys, PSU; primary sampling unit, MICS; Multiple Indicator Cluster Surveys, TAB;therapeutically induced Abortion, MH; maternal health. Thereafter, we will replicate the methodology used in Demographic and Health Surveys (DHS)17 to define primary sampling units (PSUs), which are typically census tracts or discrete villages, depending on the country. We will randomly select 20 PSUs in each study district/province based on probability proportionate to size. Finally, we will define eligible facilities for each indicator within the sampled PSUs based on the services they provide relevant to the specific validation questions for that indicator. Eligible facilities for each indicator will include all lower level primary health facilities within the PSUs that provide the relevant maternal health-related services, plus all higher level facilities across the district/province. Within study districts/provinces, we will collect primary, quantitative, individual-level data from study participants via surveys conducted at facilities and in communities. Eligible facility-based participants will include administrators; maternity care clinicians (midwives/midwifery professionals and clinical cadres legally authorised to provide induced abortions); women who received an included maternal-health related service at an eligible facility and their chosen companions if they had a complicated childbirth or caesarean birth. Within eligible facilities, we will obtain a sample of staff participants as detailed in table 4. We will enrol 1040 women of reproductive age who received maternal health services in each country, representing 20 women per service/district for 260 women total per district. Eligible community-based participants will include women of reproductive age (15–49 years). We will use the same 20 PSUs to obtain the community-based sample of women. Within each, a house listing exercise will identify households with women of reproductive age (15–49 years). From this list, 18 households per PSU will be randomly selected and 1420 women will be recruited, based on the following sample size calculation: n=Z2*pqd2, where Z is the standard normal deviate, p is the proportion of population with characteristic, q is the proportion of population without characteristic, d is the degree of accuracy required. The sample size derived through this calculation (n=96) was further adjusted to reflect an estimated 10% non-response rate, a design effect of 2 to account for clustering and a multiplier of 1.68 to account for the low prevalence of modern contraception in each country, yielding a final sample size of 355 women per district/province. Household surveys are infeasible in Argentina due to low population density, vast distances between households and lack of cultural acceptance. Therefore, interviews will be conducted with a random sample of 360 women per district exiting from eligible facilities. Facility eligibility criteria are detailed above. Participants will be considered eligible if they belong to one of the targeted participant groups listed above, and/or have received an included maternal health-related service and meet the age of majority to consent or else provide assent along with parental consent if younger (less than 18 years old in Ghana and India; less than 16 years old in Argentina). Exclusion criteria include not being proficient in the local language; not meeting the age of majority in the country, district or province unless they can provide parental consent; being unable, unwilling or lacking capacity to provide consent or assent. No patients were involved in the design, conducting, reporting or dissemination of this study. We will engage local country stakeholders in a dissemination activity in each study country. We will disseminate results to district/provincial government units and participating health facilities as appropriate, to ensure that they can be used to drive progress and improvement in the study settings. In the following section, we describe in detail the specific methodology and analytical plan for each indicator. (1) To verify that the ‘legal status of abortion’ indicator reported globally by each country accurately reflects the laws and statutes on record; and (2) to look for variation at the provider and facility level of the application of the legal categories under which abortion is lawful (legal grounds) and, thus, the accessibility of induced abortion. This validation exercise will use mixed methods exploring two validation questions to test the global indicator on legal status of abortion. We will conduct a desk review of the legal grounds for induced abortion expressed in national laws (subnational laws, in Argentina), also capturing any requirements for eligibility on each legal ground articulated in the legal statutes. We will conduct surveys with health professionals whose scope of practice authorises them to provide abortion services in each setting to explore provider knowledge of the legal grounds for abortion in their jurisdiction and provider practices for determining patient eligibility on each legal ground, providing abortion services or referrals. For the first validation question, we will compare and describe any differences between legal statutes in each country, reported data in the Countdown indicator, and the WHO GAPP database. For the second, we will tabulate the number of accurate survey responses among abortion providers on the legal grounds for abortion in their jurisdiction. We will explore any variance in provider requirements to access abortion for each legal ground in the country to look for differences in the application of the law across providers and facilities. Descriptive statistics will be reported and we will stratify the results to look for systematic variance. To verify that no charges, formal or informal, are assessed for services included in the indicator that are supposed to be free by law and to describe variance between the law and primary data sources. We will conduct a desk review of national and subnational laws and policies on free care provision. We will administer surveys to chief financial officers (or similar administrative position) within participating health facilities to collect data on formal fees or payments charged for any included services and the rationale. We will conduct interviews with women exiting eligible facilities to ask about formal and informal charges for any services received. If a woman had a complicated birth or caesarean section and a companion of choice (eg, family member or friend) is present who was at the facility during the birth, we will interview them as well about any charges they may have paid on her behalf. We will use comparative analysis to detect and describe differences between service categories designated as free to users in the national statutes, and the most recent data reported by the country in the WHO MNCAH Policy Survey. We will estimate the per cent of women paying fees for each type of service. Universal applicability of the policy implies that 0% of women pay fees for maternal health services in the public sector. We will test the significance in the difference using a one-sample test of proportion. We will use a χ2 test to determine whether fees are levied in a systematically different way to various types of women using the EPMM standard equity stratifiers. Results will be reported by service type and client demographics, and the value of the indicator expressed each way will be compared with explore optimal construct validity. To strengthen measurement of midwifery workforce adequacy. Three aspects of adequacy are reflected: density (number to meet need), distribution (accessibility) and both competency and authorisation to provide essential care (availability). Two nested validation exercises are included. The aims of the first one are: (1) to compare midwifery professionals’ scope of practice in each country to international reference standards from the International Labour Organization’s (ILO) definitions for midwifery professionals and associate professionals and to the International Confederation of Midwives (ICM) Essential Competencies for Midwifery Practice and (2) to compare estimates derived from two indicators to measure the same construct (density and distribution of midwives), to explore consistency (convergent validity), evidence that one measure is more accurate or a more efficient way to capture the construct and whether adjusting the numerator and/or denominator provides a better estimate. The second validation exercise aims to verify whether midwives and midwifery professionals are authorised to perform basic obstetric and neonatal care (BEmONC) functions and whether they do so in practice. We will conduct document review to compare the national scope of practice for midwifery professionals on record in each country to the ILO and ICM descriptions for midwifery personnel. We will review national laws and regulations that authorise midwifery professionals’ scope of practice in each country to verify what is reported by the country in the MNCAH Policy Survey. Then, we will recruit a representative sample of midwifery professionals employed within all participating facilities providing maternal health-related services in each study district. We will administer a survey asking respondents whether they have the skills necessary to perform each competency and/or BEmONC signal function; how they obtained those skills; the frequency and recency of behaviours related to each competency or reasons for non-performance of these behaviours in their current job. We will report the percent agreement between the national scope of midwifery practice and the ILO tasks, the ICM competencies and the variance between them. We will calculate the percent (%) of midwives whose current practice meets the international standard reflected in the ICM competencies as well as the average competency of midwives in the sample, stratified by facility type (public, private) and geography (urban, rural). Last, we will compare the value of the indicator for density and distribution of midwives, adjusted using different numerators and denominators. For numerators, we will calculate the value using the number of midwives on facility rosters, those who meet the ILO definition, and those who meet the ICM competencies. For the denominator, we will examine the value of the indicator using different population parameters: total population/district; women of reproductive age/district; number of births/district and number of pregnancies/district. We will compare midwives’ authorisation to perform BEmONC signal functions with the country’s most recent Countdown 2030 country profile and response to the most recent WHO RMNCH Policy Survey. We will then compare the tasks that midwives and midwifery professionals are authorised to perform to their reported actual performance of those tasks over the last 90-day period in facilities, where emergency maternal and newborn care are available in each study setting. We will report any variance between midwifery professionals’ authorisation, training, and practice patterns. To explore two dimensions of availability of B/CEmONC facilities: availability of all B/CEmONC signal functions within designated B/CEmONC facilities, and sufficient number of B/CEmONC facilities to meet the needs of the population (coverage). The aim is to compare the value of estimates emphasising different dimensions of availability of B/CEmONC facilities, based on different measurement approaches and data sources, to explore their external consistency or convergent validity. We will review records at all participating facilities where births take place to look for evidence that they have performed emergency signal functions within the previous 90 days and offer services 24 hours per day/7 days/week. We will perform geospatial analysis to estimate the travel time to each facility within the sample for various segments of the population. We will use a publicly available global population model for these estimations. We will compare and report any variance between B/CEmONC designation and functionality across all facilities. We will calculate and compare the value of the indicator in each study district using the following denominators: 500 000 population/district; 20 000 births/district; 30 000 pregnancies/district. Last, we will use the travel time estimates obtained from the geospatial analysis to ascertain the number of facilities that are within a 2-hour travel time for the total population, for women of reproductive age, and for the number of births and pregnancies occurring to women within each study district. We will explore how the value of the indicator differs based on the denominator used and compare the values of the indicator reflecting these various approaches to measuring EmONC availability and report differences. To validate both numerator and denominator of the indicator ‘Maternal death review coverage’, defined as the percentage of maternal deaths occurring in a facility that were audited, in the study settings. Both numerator and denominator are subjected to threats to validity due to under-reporting and misclassification of maternal deaths. We will collect documentary evidence of maternal death and maternal death reviews in all facilities through chart and record review. We will perform retrospective review of secondary data obtained from district HMIS on both maternal deaths and maternal death reviews reported from all facilities. We will compare the number of facility-based maternal deaths reported through HMIS to the district to the verified number of maternal deaths in all facilities in the district in patient registers. We will trace individual deaths by dates and other reported details to verify they have been reported to the district. Once validated, we will aggregate all maternal deaths reported for comparison. We will review facility death review committee records for the last 1-year period to extract the number of maternal death reviews conducted and the content of each review. We will compare the number of maternal death reviews reported to each district with the number of reviews validated through facility record review that met the definitional standard for quality18 in the same district. Finally, we will tabulate maternal death review coverage using primary data for the numerator and denominator to the official value reported in the indicator in each country. ‘Demand for family planning satisfied through modern methods of contraception’ uses a macroeconomic lens to look at contraceptive supply and demand, aggregating data from individual women; however, it is uncertain how well it correlates with women’s own subjective perceptions of their personal demand for contraception through modern methods or how well that demand has been satisfied. This study has two aims: (1) at the individual level, to assess whether women’s self-reported demand for family planning and its satisfaction converges with the standard DHS-derived measure and (2) at the population level, to examine how the value of the indicator changes based on the use of derived data from the standard calculation versus self-reported data reflecting women’s own perceptions. We will administer a community-based survey to a sample of women in each study setting that includes direct questions to women about their desire for and use of contraception, their satisfaction with their current method and their experience of care during their most recent family planning encounter. We will then ask all the questions, in order, in the DHS algorithm used as the global standard to calculate the indicator. We will compare the results for individual women of two different approaches to measuring the construct of ‘demand for family planning satisfied through modern methods of contraception’ using matched t tests. We will disaggregate by women’s characteristics to identify patterns. Finally, at the population level, we will compare the value of the indicator calculated from primary data we collect to the aggregate district/province level data reported through DHS where available to explore convergence. Sustainable Development Goal 5.6.2. tracks the ‘Number of countries with laws and regulations that guarantee full and equal access to women and men aged 15 years and older to sexual and reproductive healthcare, information and education’. Weaknesses with the indicator scoring methodology have the potential to change its value and affect its interpretation. The aim of this exercise is to verify the laws and regulations reported for this indicator in Ghana and to explore whether the value of the indicator changes using new estimation methods to calculate its score compared with the established method, to improve interpretation. We will conduct a comprehensive desk review of legal statutes and regulations related to the 13 components in the indicator metadata. We will conduct secondary analysis of results from the UN 12th Inquiry Among Governments on Population and Development, Module II (Fertility, Family Planning and Reproductive Health) Survey,19 which reports on existing laws along with barriers and enablers. We will compare the laws and regulations on record in Ghana to the 13 components reported in the indicator for completeness and accuracy. We will calculate scores for the data collected from the UN Module II survey using the original UN scoring method and alternative scoring methods to look for differences in resulting values of the indicator. Values will be compared and sensitivity analyses conducted to explore the range of variation in the value of the indicator and the associated impact on its interpretation as a measure of sexual and reproductive health and rights.