Consensus-based approach to develop a measurement framework and identify a core set of indicators to track implementation and progress towards effective coverage of facility- based Kangaroo Mother Care

Background As efforts to scale up the delivery of Kangaroo Mother Care (KMC) in facilities are increasing, a standardized approach to measure implementation and progress towards effective coverage is needed. Here, we describe a consensus-based approach to develop a measurement framework and identify a core set of indicators for monitoring facility-based KMC that would be feasible to measure within existing systems. Methods The KMC measurement framework and core list of indicators were developed through: 1) scoping exercise to identify potential indicators through literature review and requests from researchers and program implementers; and 2) face-to- face consultations with KMC and measurement experts working at country and global levels to review candidate indicators and finalize selection and definitions. Results The KMC measurement framework includes two main components: 1) service readiness, based on the WHO building blocks framework; and 2) service delivery action sequence covering identification, service initiation, continuation to discharge, and follow-up to graduation. Consensus was reached on 10 core indicators for KMC, which were organized according to the measurement framework. We identified 4 service readiness indicators, capturing national level policy for KMC, availability of KMC indicators in HMIS, costed operational plans for KMC and availability of KMC services at health facilities with inpatient maternity services. Six indicators were defined for service delivery, including weighing of babies at birth, identification of those ≥ 2000 g, initiation of facility-based KMC, monitoring the quality of KMC, status of babies at discharge from the facility and levels of follow-up (according to country-specific protocol). Conclusions These core KMC indicators, identified with input from a wide range of global and country-level KMC and measurement experts, can aid efforts to strengthen monitoring systems and facilitate global tracking of KMC implementation. As data collection systems advance, we encourage program managers and evaluators to document their experiences using this framework to measure progress and allow indicator refinement, with the overall aim of working towards sustainable, country- led data systems. The KMC measurement framework and core list of indicators were developed through: 1) scoping exercise to identify potential indicators; and 2) face–to–face consultations with measurement and KMC experts to review candidate indicators and finalize selection (Figure 1). Overview of Kangaroo Mother Care (KMC) framework development and indicator selection process. An initial list of candidate indicators was developed through a review of the grey literature (program documents and surveys) and consultations with KMC and measurement experts. We circulated a request for existing KMC indicators to members of the Newborn Indicators Technical Working Group (NITWG), an inter–agency working group convened by Saving Newborn Lives, and the KMC Acceleration Partnership. A total of 79 candidate indicators and data elements were extracted and summarized in an excel spreadsheet. This list was refined to 55 through sorting and removal of duplicates and organized by a standard results framework (impact, coverage, access, quality, demand, policy/enabling environment). The main sources of indicators in this initial list included the Fundacion Cangaru, Maternal and Child Health Integrated Program (MCHIP) KMC implementation Guide, various facility and household surveys conducted by programs implementing KMC (SNL Malawi Facility Assessment and household survey, Uganda Newborn Study (UNEST) survey, Ghana Newhints survey, Ethiopia household survey, South Africa Facility Assessment) and the Malawi HMIS. A series of face–to–face meetings were convened with KMC and measurement experts working at country and global level over a three month period. A full list of participants, their affiliations and area of expertise (measurement, KMC or both) are included in Table S1 in Online Supplementary Document(Online Supplementary Document). A small group meeting with 12 members of the KAP and the NITWG was held September 5, 2014 to review and score the raw list of 55 indicators. A focused set of five criterion for initial scoring of the indicators was developed, which reflects commonly applied indicator selection criteria: feasibility (data can be collected with reasonable and affordable effort in low resource settings), reliability (data can be collected consistently over time), usefulness for decision–making (data are relevant and will help guide KMC programming), sensitivity (responsive to change), and specificity (focused on specific aspect, not overly broad) [15–17]. The group broke into smaller groups for in–depth discussion and scored each candidate indicator as high, medium, or low for each of the five criteria. The group recommended that a measurement framework specific to KMC should be developed to better organize the indicators and assist with prioritizing selection. Following the meeting, a core team representing the KAP, NITWG and ENAP metrics stream extracted the strongest indicators based on the scoring criteria for further development (preparation of full definitions, data source, and methods) and drafted a KMC measurement framework. The resulting 18 candidate indicators were then organized according to the draft framework. In October, consultations were held with a broader group of newborn programmatic and measurement experts on October 6–7, 2014 in Washington DC. The 24 attendees, representing implementing agencies, donors and researchers, formed three small groups (national level/service readiness; facility–level/service delivery; and coverage) to review the KMC framework and each of the 18 candidate indicators in detail. Groups were provided with a series of questions for each indicator to guide their discussion and decision–making process. The group made recommendations about which indicators to retain, which to drop and areas for further research; further details on the discussion and outputs is available in the meeting report [18]. Following the meeting, the core team consolidated feedback and updated the measurement framework and refined the indicator list down to 11 candidate indicators. A smaller task team was delegated to work specifically on defining a feasible coverage indicator for KMC that could be tested as part of the ENAP metrics measurement improvement plan. This task team, alongside the core group, also undertook a preliminary mapping exercise to see what data were available, with a focus on assessing denominator options for generating a potential coverage indicator for KMC that could be tested as part of the ENAP measurement improvement plan. The final consultation took place on November 15, 2014 in Kigali, Rwanda as part of the KMC Acceleration meeting and focused on country level input. Eighteen participants, including individuals supporting KMC implementation in nine countries (Bangladesh, Malawi, Nigeria, Rwanda, India, Indonesia, Philippines, Uganda and South Africa) gave feedback on the measurement framework and reviewed each candidate indicator to assess availability, feasibility and usefulness considering their country context. Participants were split into two groups. One group worked specifically on the coverage indicator, and the other group focused on readiness indicators and facility level data for tracking service delivery and quality of care. In each group, a presentation was made to provide an overview of progress to date, review each indicator in detail and identify priority areas for discussion. Participants in the service readiness and facility data group were asked to use post–it notes to record information on availability/data source, data users, collection methods, and challenges for each indicator in their setting and then vote whether the indicator was ready to go, needed more work/unsure or should be dropped. Participants in the coverage group, reviewed the work carried out by the ENAP metrics KMC task team (Box 1) and discussed a feasible a measurable coverage indicator. In view of the challenges in measuring a denominator, the group reached consensus through placing individual votes between use of <2500 g, total facility births or estimated live births. Based on the feedback, the core team finalized the framework and list of indicators. What is coverage measurement and why is it challenging for KMC? A coverage indicator aims to measure the number of individuals that receive a specific intervention or treatment within a given population in need of the intervention. The numerator is measured as the total number of individuals that received the intervention and the denominator is the total population, usually those that could have benefitted from that specific intervention or treatment. For KMC, neither the numerator nor the denominator are easy to define or measure. KMC is not a one–off contact with the health system; many of the components of KMC are processes (eg, continuous skin–to–skin contact, follow up care). And measurement of specific interventions is a challenge when only by a small group or sub population benefit from that intervention. Defining whether or not an infant could benefit from KMC requires a level of clinical judgement and more precise metrics than those reported by most routine information systems in LMIC. The ENAP metrics KMC task team ENAP metrics assembled a KMC task team with experts in measurement and programme implementation drawing on expertise from the KMC acceleration partnership and wider groups. Different numerators and denominators were proposed and discussed based on their definition and the feasibility of measurement. Numerator challenges The evidence base for mortality impact of KMC is currently for infants weighing 2000 g or less. However, in some low and middle income countries where programmes have been extended, eligibility criteria for entry to KMC may be for babies up to 2500 g. Coverage of most maternal and newborn interventions in many settings is still measured through household surveys and relies on maternal recall up to five years after the birth in question. Even though mothers can accurately recall KMC, even years after the event, the sample size needed to gather representative data through a household survey may be prohibitively large15. Typically, facility based assessments capture information on infrastructure, processes and service readiness, and are best suited to measure the number of facilities that are prepared to provide components of the service (eg, sufficient trained staff, space, and equipment). In most settings, the number of newborns initiated on facility–based KMC is measured either through hospital admission or care records, but currently these data are rarely reported into national health information systems. Denominator challenges The denominator was the most technically challenging and a list of options were proposed. A large proportion of newborns do not have their weight recorded at birth and even where birthweight is recorded, there is a known tendency for “heaping” of data, especially at measures of 2500 g and 2000 g. Given the difficulty in accurately capturing all those babies in need of KMC, especially through existing data collection systems, using total live births as the denominator to give a proxy was considered. This has been done with other interventions where the aim is not for 100% coverage, such as C–section, to generate a rate that is benchmarked against a target threshold. Recent estimates suggest a variation in preterm birth rates of between 4–18% of total lives births in different countries. This means that the KMC rate in each country may indicate a different unmet need and target thresholds would need to vary between settings to reflect these differences as well as variation in numbers of full–term LBW and pre–term babies. As an important limitation, if total live births is used as a denominator, it does not reflect whether the babies that received it were drawn from the population that could have benefitted from KMC. Proposed indicator The ENAP KMC task team established that it is not possible to capture all of the components of KMC in one coverage indicator as many of these refer to processes that happen over a period of time. Household surveys are unlikely to be a feasible approach to measure KMC coverage and increasingly, health facility assessments are starting to measure key components of KMC care. Of all the available options, the number of newborns initiated on facility based KMC gives a representation of the number of newborns initiating the care. Task teams agreed the indicators would need rigorous testing for validity and feasibility with a variety of different denominator options including, live births in the facility, estimated live births and eventually target population for coverage (total number of newborns ≤2000 g). As a preliminary exercise, the task team approached a select few LMIC countries for data on the KMC numerator, which is available through a limited number of HMIS and many hospital registration systems. To demonstrate the numerator with different denominator options, task teams present three graphs showing the proposed numerator over total reported live births, total reported live births <2500 g and estimated live births for two countries, Malawi and Dominican Republic (Figure 4). Analysis of denominator options for Kangaroo Mother Care (KMC) coverage indicator in Malawi and Dominican Republic. What are the next steps? As national facility based data and health information systems become more advanced, the ideal is to develop more precise indicators, but these are not currently available in most of the countries where the unmet need for KMC is arguably the greatest and there are the most data gaps. It is critical to improve the recording and reporting of birth weight in facilities. Given the importance of prematurity as a direct cause of death and as a risk factor for morbidities and death from other causes (eg, infections), developing simplified tools for measuring gestational age is critical to plan for programmes, to improve the evidence base and to develop more precise indicators of unmet need. If such data were available in more settings, indicators based on specific weight or gestational age criteria could be measured. Existing data sets from countries with established KMC programmes and accurate assessment of gestational age and birthweight could be used for testing the denominators and proposed numerators. The ENAP metrics measurement improvement plan has a five year plan set out to test the validity and feasibility of a number of numerator and denominator options for all the ENAP core indicators with the objective of institutionalizing a KMC coverage indicator in global accountability mechanisms by 2020.