Taking stock: Protocol for evaluating a family planning supply chain intervention in Senegal

Background: In Senegal, only 12 % of women of reproductive age in union (WRAU) were using contraceptives and another 29 % had an unmet need for contraceptives in 2010-11. One potential barrier to accessing contraceptives is the lack of stock availability in health facilities where women seek them. Multiple supply chain interventions have been piloted in low- and middle-income countries with the aim of improving contraceptive availability in health facilities. However, there is limited evidence on the effect of these interventions on contraceptive availability in facilities, and in turn on family planning use in the population. This evaluation protocol pertains to a supply chain intervention using performance-based contracting for contraceptive distribution that was introduced throughout Senegal between 2012 and 2015. Methods: This multi-disciplinary research project will include quantitative, qualitative and economic evaluations. Trained researchers in the different disciplines will implement the studies separately but alongside each other, sharing findings throughout the project to inform each other’s data collection. A non-randomised study with stepped-wedge design will be used to estimate the effect of the intervention on contraceptive stock availability in health facilities, and on the modern contraceptive prevalence rate among women in Senegal, compared to the current pull-based distribution model used for other commodities. Secondary data from annual Service Provision Assessments and Demographic and Health Surveys will be used for this study. Data on stock availability and monthly family planning consultations over a 4-year period will be collected from 200 health facilities in five regions to perform time series analyses. A process evaluation will be conducted to understand the extent to which the intervention was implemented as originally designed, the acceptability of third-party logisticians within the health system and potential unintended consequences. These will be assessed using monthly indicator data from the implementer and multiple ethnographic methods, including in-depth interviews with key informants and stakeholders at all levels of the distribution system, observations of third-party logisticians and clinic diaries. An economic evaluation will estimate the cost of the intervention, as well as its cost-effectiveness compared to the current supply chain model. Discussion: Given the very limited evidence base, there is an important need for a comprehensive standardised approach to evaluating supply chain management, and distribution specifically. This evaluation will help address this evidence gap by providing rigorous evidence on whether private performance-based contracting for distribution of contraceptives can contribute to improving access to family planning in low- and middle-income countries. Senegal is a West African country with a population of approximately 14 million. It is classified as a lower-middle income country, with a Human Development Index rank of 163 out of 187 countries in 2013 [42]. Reproductive data are available from the 2014 Demographic and Health Survey [43], however we use data from the 2012–13 survey here to describe the context at the start of the intervention implementation. In Senegal, half (47 %) of women aged 25–29 were married at age 18 or below in 2012–13 [44]. The total fertility rate was 5.3 children per woman, with an average of 4.1 for women living in urban areas and 6.3 in rural areas [45]. Following the slow rise in modern contraceptive use reaching 12 % of WRAU in 2010–11, the MCPR increased to 16 % by 2012–13 (Fig. 1). An estimated 29 % of WRAU still had an unmet need for FP in 2012–13, implying that almost one third of women in union who did not want to get pregnant within the next 2 years were not using contraception [45]. Sharp differences existed in the MCPR between urban and rural areas (27 % compared to 9 %, respectively) and across educational levels (29 % among WRAU with secondary education and above, compared with 12 % among WRAU with no formal education). The most popular contraceptive method was injectables, accounting for 39 % of modern contraceptive users, followed by the pill (32 %) and implants (17 %) [45]. The public sector supplied the vast majority of contraceptives, with 83 % of modern contraceptive users obtaining their last method in a public SDP, and above 90 % for injectables and IUDs. The private sector supplied 13 % of users (predominantly condoms and pills) [44]. Two sub-objectives will be addressed in the impact evaluation: (1.1) evaluate the effect of the intervention on contraceptive stock availability in SDPs, and (1.2) examine the effect of changes in stock availability on the MCPR at the population level. Table 1 presents the key outcome and impact indicators used for objective 1, based on data collected by two studies. Indicators used for outcome and impact evaluation a SDP Service Delivery Point; FP family planning; WRAU Women of Reproductive Age in Union; WRA Women of Reproductive Age bModern methods include condoms, pills, injectables, implants, intra-uterine devices (IUDs), sterilisation cLong-acting methods include implants and IUDs The first study will follow a non-randomised stepped-wedge design using secondary data. To determine the impact of the intervention on contraceptive stock availability (objective 1.1) we will use data from Service Provision Assessments [46], nationally representative surveys of health facilities conducted annually in Senegal. A general mixed-effects logistic regression model will be constructed using an approach similar to the stepped-wedge analysis method outlined by Hussey and Hughes [47], although the rollout was not randomised. Stock availability will be modelled as a binary outcome variable (stock present or absent on the day of the survey) for all contraceptive methods, and as an ordinal variable for the number of each of the most popular short-acting (condoms and pills), medium-acting (injectables) and long-acting methods (implants and IUDs). Time will be treated as a fixed effect, and region as a random effect, to allow the intervention effect to vary regionally. Independent covariates at regional level, such as density of SDPs offering FP services and density of road network, will be used as well as SDP-level variables such as facility type. Similarly, to examine the effect of the intervention on contraceptive use among WRAU (objective 1.2) we will use information on contraceptive use at the population level from the Demographic and Health Surveys [48], nationally representative surveys of women of reproductive age also conducted annually. As above, a mixed-effects logistic regression model will be built treating region as a random effect, and including woman-level covariates (such as education level) and regional covariates (such as pre-intervention MCPR). For both objectives, secondary analyses allowing for a delay in the effect of the intervention will be explored by allowing the variable for the intervention mode to be fractional. The second study will be a 4-year monthly time series of contraceptive stock availability (objective 1.1) and FP consumption (objective 1.2), constructed from data from a sample of SDPs. Five of the 13 regions where the intervention was implemented by 3PLs (i.e. excluding Saint Louis) will be randomly selected, and 10 districts randomly selected in each region with probability proportional to the number of SDPs in the district. In each selected district, the health centre will be included, as well as three randomly selected health posts, for a total sample of 200 SDPs across urban and rural locations. There is limited guidance on sample size calculations for time series analysis, which usually focuses on the number of time points rather than of sampling units [49]. At least 48 monthly observations on stock inventory and FP consultations will be collected from each SDP, as well as from the selected district and regional storerooms, and the national-level storeroom, ensuring at least 24 monthly time points pre- and post-intervention. Information on contraceptive stock availability will be extracted from stock cards, daily patient registers and stock journals, as well as for several ‘tracer’ stocks (anti-malarials, oral rehydration salts, amoxicillin, and iron tablets) not expected to change as a result of the intervention. Monthly number of FP consultations in which patients receive a contraceptive at the SDP will be extracted from patient registers. A time series analysis will be conducted by constructing a generalised linear segmented regression model of the probability of stock-out for any method on a given month, and for the most popular short-, medium- and long-acting methods. We will examine whether there is a change in the slope or level of the number of methods available and monthly FP consumption at SDPs between the pre- and post-intervention periods. This analysis will be repeated for the tracer stocks to compare their change in stock availability with FP methods as a proxy measure of the ‘normal’ distribution channels through which these commodities are still distributed. The process evaluation will be informed by both qualitative and quantitative studies to gain a better understanding of what the intervention consisted of, the extent to which it was implemented as planned, and how well 3PLs functioned within the health system. Trained researchers in the different disciplines will implement the quantitative and qualitative studies separately but alongside each other, and the different teams will share findings throughout the course of the work to inform each other’s data collection. Quantitative monthly indicator data from the implementer will be analysed to examine to what extent different components of the intervention were implemented as intended. The main indicators of interest will relate to timeliness of deliveries, achievement of stock-out rates below the 2 % target, and availability and use of data for informing stock orders (Table 2). Indicators used for process evaluation a SDP Service Delivery Point; FP Family planning; 3PL Third-party logistician Qualitative data collection will be conducted at all levels of the health system (Fig. 5) by trained anthropologists, supervised by senior academics with in-depth knowledge of the context. A discourse analysis of funding documents, proposals and project reports related to the intervention will be conducted to refine the initial theory of change elaborated by the research team. Data collection methods used in the ethnography of the supply chain Repeat in-depth interviews will be conducted with key national and international stakeholders, during and immediately after the implementation period, to allow researchers to develop a rapport with interviewees, improve data quality and monitor change over time. These interviews will focus on the development of the project, implementation issues and lessons learned, to understand how closely the implementation followed the intervention design laid out in the theory of change. In-depth interviews will also be conducted with all cadres of personnel involved in the supply chain and FP at the regional, district and SDP levels (including SDP staff, programme auditors and 3PL), to examine the acceptability of performance-based contracting, how the intervention was modified in different contexts, and potential unanticipated issues. In-depth interviews will be administered using topic guides, which will be piloted and developed iteratively as data emerge. Three repeat in-depth interviews will be conducted over the evaluation period with up to 10 key informants at the national level. In-depth interviews will be conducted with regional medical officers (up to 11), key implementing staff at the regional level (up to 15), 3PLs and auditors (up to 10), as well as with key stakeholders from the national (n = 1) and regional storerooms (n = 11), district stock managers (n = 11), health workers (n = 60) and pharmacists (n = 10). Up to 160 in-depth interviews will be conducted in total; the final sample size will follow the principle of saturation, wherein data collection continues until new data do not shed further light on the research questions. Sampling for interviews will be purposive to represent urban and rural areas and the diverse types of SDPs as well as distance from regional medical storerooms, and time since introduction of the intervention. Ethnographic work will be carried out to understand the logics and practices of implementation on the ground. Researchers will travel with 3PL while performing deliveries and carrying out stock inventories, and observe the lived realities of both 3PL and SDP staff. Transcripts of interviews and field notes will be translated and independently coded by researchers in London and Senegal using qualitative software. Coding will be undertaken along key themes that are being explored, allowing for unexpected issues that emerge, and jointly discussed by the qualitative team to come to a consensus on major findings. In addition, we will ask 25 SDP managers to fill in fortnightly reflective diaries on issues relating to the implementation, giving them a voice to identify unanticipated issues to the research team. Preliminary findings from the diary study will be fed back to SDP managers to validate our interpretation of their writings and to give them a further opportunity to discuss collectively their perspectives on the project. Similarly, findings from all objectives will be presented to key stakeholders at several points throughout the evaluation, in order to check the validity of results and elicit potential reasons for these. The purpose of this component of the evaluation is to understand what factors influence the effect of stock availability in SDPs on contraceptive use among women, using exploratory qualitative and quantitative methods. We hypothesise that FP uptake in areas with contraceptive availability would be higher in areas with high-quality FP services, where women have better geographical and financial access to contraceptive services, and with more FP-related activities (other than the intervention), such as demand-generation activities. Therefore, several approaches will be used to assess the quality of FP services provided in SDPs, describe women’s physical access to FP services, and estimate the implementation intensity of other FP­related activities in regions throughout Senegal, in order to examine whether the effect of the intervention varied according to these factors. First, the quality of FP services will be assessed using data from the Service Provision Assessments [46], which collect information on choice of methods offered, observations of FP consultations and exit interviews with clients and providers. An index of FP service quality will be developed based on frameworks developed by Bruce, Mauldin and Ross [33, 50], as well as other existing indices [51–55]. The median quality of FP services across SDPs will be calculated for each region of Senegal, and any observed regional differences in the change in MCPR (objective 1) will be interpreted in light of regional FP quality category (high, medium, or low).. Second, GPS information for all SDPs in Senegal will be obtained from the Ministry of Health, coordinates for regional storerooms taken during fieldwork and road shape files obtained from ESRI [56]. QGIS software [57] will be used to map SDPs, storerooms and roads. The 2013 census data [58] will be used to overlay age-specific fertility rates on SDP points, and buffer zones created to determine where women do and do not have access to SDPs providing FP, SDPs with FP stock available and SDPs with high quality FP services. These results will be used to examine whether changes in stock availability have had a larger impact on MCPR in regions where women have better access to FP services. In addition, in-depth interviews will be conducted with women to triangulate these quantitative analyses, with the aim of understanding the barriers they face in accessing FP, the quality of care they experience in SDPs, and what aspects women value when seeking care in public SDPs (cost, counselling, availability of methods etc.). Third, indicators to measure implementation intensity of FP-related activities (other than the intervention) will be developed using Heidkamp et al’s “snapshot” approach [59]. These indicators will be informed by in-depth interviews with different FP actors in Senegal (see Process evaluation), and calculated based on policy documents and reports from these organisations for each region. Results from the impact evaluation (objective 1) will be stratified by region according to the intensity of other FP-related activities in order to examine whether the effect of the intervention on contraceptive use was stronger in regions with other FP programmes active at the same time as the intervention. The purpose of the economic evaluation is to estimate the cost of the intervention (including costs related to capital goods, training, 3PL distribution, and audit teams), and its cost-effectiveness in relation to the current supply model. A micro-costing approach will be used to estimate the cost of the intervention and current supply chain models at national, regional, district, and SDP levels. Costs for 3PLs will be measured as the price paid for by the implementer. Costs will be estimated for the entire supply chain, retrospectively to capture costs pre­intervention, start­up costs (including initial training) and costs post­intervention. A survey of SDPs and district, regional, and national storerooms will be conducted to measure costs. Among the SDPs included in objective 1, up to 80 SDPs will be randomly sampled to include a range of SDP sizes and rural/urban locations; the district and regional storerooms in which these SDPs are located will also be surveyed. A questionnaire will be administered to SDP and storeroom managers in order to record staff involved in supply chain management, time and resources spent collecting stock from higher-level storerooms, and intervention-related activities such as training. Financial and economic costs of the intervention and the current supply chain system will be collected through document reviews and account classification. The development of the survey tools will be informed by the in-depth interviews conducted during the process evaluation (objective 2) in order to ensure all costs are captured. In addition, a time and motion study will be conducted to directly measure and cost time required to manage, operate and deliver the commodities to SDPs. Data on human resource utilisation and flow of services relating to the FP supply chain will be collected through interviews with SDP managers and 3PLs, and time spent on supply chain activities will be estimated using structured observations of SDP staff involved in supply chain activities and 3PLs and self-reported timesheets. Financial and economic costs will be categorised as: training and start-up costs, capital costs (including storage and transport equipment) and recurrent costs (such as staff salaries and maintenance expenditure). Capital costs will be estimated by using current (replacement) costs for all capital goods used in a year, annualised over the expected duration of their working life using a discount rate of 3 % according to World Health Organisation guidelines [60]. The aim of the cost-effectiveness study is to compare the incremental costs and incremental effects of the intervention compared to the current supply chain model for contraceptives. Since the intervention has already been rolled out, the pre-intervention costs of distributing FP commodities will be estimated by calculating costs of the current distribution system for non-contraceptive ‘tracer’ commodities (anti-malarials, oral rehydration salts, amoxicillin, and iron tablets). Estimates for the incremental effectiveness of the intervention compared to the current supply chain model will be obtained from the impact evaluation (objective 1). Several effect measures will be used to calculate incremental cost-effectiveness ratios (Table 3). Data on stock availability in SDPs will be used to calculate FP consumption lost due to stock­outs and the cost per stock­out averted for all stock­outs and for the most commonly used short­, medium­ and long­acting methods. Indicators used for economic evaluation a SDP Service Delivery Point; 3PL Third-party logistician; FP Family planning; MCPR: modern contraceptive prevalence rate Modelling techniques, in addition to assumptions derived from the literature, will be used to estimate the cost per couple­year of protection; cost per additional WRAU accessing modern contraception; and cost per pregnancy averted, unsafe abortion averted, and maternal death averted. In addition, the healthcare costs saved by averting pregnancies (such as costs of antenatal care, post­abortion care, pregnancy and birth complications) will be estimated. All cost estimates will be evaluated using probabilistic sensitivity analyses to identify the variables that had the largest impact on the model results. The study design and data sources used to address the four main objectives are summarised in Table 4. Summary of study designs and data sources used to address each research objective a SDP Service Delivery Point; FP Family planning; 3PL third-party logistician bService Provision Assessments [46] cDemographic and Health Surveys [48] Ethical approval for this study was obtained from the ethics committee of the Conseil National de Recherche en Santé (CNRS) in Senegal (n° 107/MSAS/DPRS/CNERS), and from the London School of Hygiene & Tropical Medicine (Ethics ref: 9925). Informed consent will be sought from participants for primary data collection-related activities, including in-depth interviews, structured observations and SDP diaries. For data extraction from SDPs and district, regional or national storerooms, consent will be sought from the head of the SDP or storeroom chief officer. Participants in qualitative data collection, and SDPs and storerooms, will be identified using a coded identifier, and the key will be stored securely in a password-protected file accessible to selected team members. In the reporting of the data, quotations will be anonymised to ensure individuals are not identifiable. GPS information for SDPs will be stored in a separate password­protected dataset. Audio files will be downloaded onto a password­protected computer hard drive and backed up regularly. All paper and soft copies of field notes will be kept in a locked cabinet and only shared within the study team.