Evaluation of the impact of the voucher and accreditation approach on improving reproductive health behaviors and status in Kenya

Abtsract. Background: Alternatives to the traditional ‘supply-side’ approach to financing service delivery are being explored. These strategies are termed results-based finance, demand-side health financing or output-based aid which includes a range of interventions that channel government or donor subsidies to the user rather than the provider. Initial pilot assessments of reproductive health voucher programs suggest that, they can increase access and use, reducing inequities and enhancing program efficiency and service quality. However, there is a paucity of evidence describing how the programs function in different settings, for various reproductive health services. Population Council, funded by the Bill and Melinda Gates Foundation, intends to generate evidence around the ‘voucher and accreditation’ approaches to improving the reproductive health of low income women in Kenya. Methods/Design. A quasi-experimental study will investigate the impact of the voucher approach on improving reproductive health behaviors, reproductive health status and reducing inequities at the population level; and assessing the effect of vouchers on increasing access to, and quality of, and reducing inequities in the use of selected reproductive health services. The study comprises of four populations: facilities, providers, women of reproductive health age using facilities and women and men who have been pregnant and/or used family planning within the previous 12 months. The study will be carried out in samples of health facilities – public, private and faith-based in: three districts; Kisumu, Kiambu, Kitui and two informal settlements in Nairobi which are accredited to provide maternal and newborn health and family planning services to women holding vouchers for the services; and compared with a matched sample of non-accredited facilities. Health facility assessments (HFA) will be conducted at two stages to track temporal changes in quality of care and utilization. Facility inventories, structured observations, and client exit interviews will be used to collect comparable data across facilities. Health providers will also be interviewed and observed providing care. A population survey of about 3000 respondents will also be carried out in areas where vouchers are distributed and similar locations where vouchers are not distributed. © 2011 Warren et al; licensee BioMed Central Ltd. i). At Facility Level a) Accredited facilities will have a greater increase in average utilization of essential Maternal and New born health (MNH) care and FP services compared to control facilities between baseline and follow-up surveys. b) Accredited facilities will have a greater increase in the proportion of poor clients for essential MNH care and FP services compared to control facilities between baseline and follow-up surveys. (Poverty is measured using three indices: participatory scale, standard household assets scale and a food insecurity scale) c) The quality of essential MNH and FP services in voucher facilities will be equal to or greater than the quality of the same in non-accredited facilities. ii). At Population Level: a) Communities served by voucher distributors for MNH and FP services will have greater increase in the proportion of facility-based births compared to the comparison communities at baseline and follow up surveys b) Communities served by voucher distributors for MNH and FP services will have greater increase in the proportion of facility-based births among the poor compared to the poor in comparison communities at baseline and follow up surveys. This study aims to evaluate the impact and effectiveness of the voucher and accreditation approach in Kenya. 1. To assess the effect of the voucher and accreditation approach on increasing access to, quality of, and reducing inequities in the use of, selected RH services 2. To evaluate the impact of the voucher and accreditation approach on improving reproductive health behaviors and RH status and reducing the inequities at the population level. The study will employ a before and after quasi-experimental design with a control group where surveys will be undertaken among the target population for the voucher program before and after its introduction and also among an equivalent comparison population living in areas not served by a voucher program in order to control for potential time dependent confounding. In order to address the first objective, facilities will be the primary sampling unit to measure access to and quality of care and service statistics. Health facility assessments, including providers’ technical competence, skills and time-utilization, and clients’ perceptions of quality of care at specified intervals at accredited and non-accredited facilities will be undertaken. The district-level administrative unit will be used to generate clusters of heath facilities that are accredited and those that are not. These two sets of facilities will be in the same or similar districts to maximize the likelihood of the populations having similar social, cultural, economic characteristics, and having similar RH behaviors among women aged 15 to 45 years and among pregnant women. As some degree of variability is expected between the districts in terms of the background characteristics mentioned above, four districts will be selected. By the end of Phase I, there were 54 facilities accredited in Kenya. Five of these will be randomly selected from each district and Nairobi making a total of 20 facilities that will then be matched with non-accredited facilities in similar nearby districts. Given that the accredited facilities will self-select to the experimental group through choosing to participate in the voucher program, there is a strong likelihood that they will be different from those not choosing or not invited to participate. We cannot predict a priori how they might be different, for example, perhaps the contracted providers are more entrepreneurial or perhaps more socially motivated. To maximize the equivalence of these groups, thereby enhancing the validity of the design, a sampling design known as pair-wise matching will be used. In this design, the characteristics of interest (those characteristics that may influence a provider’s or facility’s performance above and beyond their use of the voucher and accreditation model) are measured for each accredited provider in the experimental sample upon recruitment, and a profile established for each provider. Researchers then identify ‘equivalent’ non-accredited providers for the control group. Examples of these types of characteristics include type of practice, professional skills mix, profile of clientele, location, and fees charged, among others. The second objective of this research is to conduct population-level surveys among representative samples of women, men and adolescents stratified by socio-economic status in geographic areas served by accredited and non- accredited facilities in the selected districts. The sample size is based on the national proportion of facility-based births; 42% of all births. We assume the national figure is representative of the proportion of facility-based births in the voucher region. To detect a 14% increase in the proportion of facility-based births, we will need 1078 experimental subjects and 1078 control subjects to be able to reject the null hypothesis that the proportion of facility-based births for experimental and control subjects are equal with probability (power) 0.8. The Type I error probability associated with this test of the null hypothesis is 0.05. We will use an uncorrected chi-squared statistic to evaluate this null hypothesis. The survey will measure indicators described in table ​table1.1. The expected results include intervention-dependent RH outcomes (pregnancy-and birth-related complications, unintended pregnancy, inter-birth intervals, reported STI treatment, among others); RH-related care behaviors (antenatal care, ANC; skilled delivery; postnatal care; lactational amenorrhea, LAM, breastfeeding; contraceptive use); awareness of RH issues, use of services, out of pocket spending, and expectations for use of services. Broad indicators for the assessment of OBA project in Kenya Population Council will conduct two health facility assessments [6] to assess quality of care provided in public, faith-based and private study facilities. An initial assessment will be undertaken in both accredited and non-accredited facilities to determine the comparability of the facilities and to provide baseline measures of the quality of care. To determine the sustainability of the quality of care provided an additional assessment will be undertaken at 12-15 months later to determine the extent to which the quality of care has changed. In addition, data collected through routine monitoring of service statistics will provide further information about client load, services mix, and client characteristics. Data collection procedures for each component of these assessments are as follows: An inventory of available resources including facility infrastructure, staffing numbers and skills mix, services provided, staff training undertaken, availability of equipment, commodities, test kits, stationary (client cards and notes), medications required to provide the services within the intervention will be undertaken. The head of the facility will be approached by a nurse/midwife researcher. The researcher will request the in-charge to guide them around the facility to observe and record all relevant information on a checklist. Statistics related to routine program data on utilization of MNH/family planning services for a 6 or 12 month period prior to the assessment visit will be collected. We will also record the number of new and continuing clients coming to a clinic for MNH/FP services as well as other health services. Monthly trends in the numbers of new and continuing ANC, post-natal care (PNC) and family planning clients as well as for other services will be obtained from facility records. All providers at 40 MNH-FP units will be approached for interview. Eligible providers available on the day of data collection at the facility will be interviewed concerning MNH and FP services. Interviews will ascertain their perceptions of barriers and operational challenges that may influence voucher clients’ acceptance of services and the provider’s attitudes towards the accreditation process. In addition there will be assessment of provider knowledge and skills for MNH and FP and other related SRH services, as well as their understanding of the organizational setup and description of related activities. It is expected that four to eight providers will be eligible to participate in the hospitals and between two and four at the health centers and dispensaries. This will give around 80 providers in each group (Total 160). To assess the quality of care at each facility, all providers participating in the facility evaluation and who deliver the MNH/FP services will be asked for permission to observe their consultations. Recruitment of providers will be done following sensitization meetings held with the district health management teams (or equivalent). Researchers will hold group meetings with the management and healthcare providers in each participating facility to introduce the purpose and methods of the study and to request their participation. The tools will also be pre-tested among a small group of women with similar characteristics as the study population to identify potentially negative consequences and modified accordingly. Observations of client-provider interactions (CPI) will be conducted during FP visit, labor, pre-discharge from the maternity unit and delivery, six weeks postpartum. The CPI encompasses both the process (how clients are treated and whether they actively participate) and the content (what they are told, technical competence, accuracy of information, provision of essential information) of a consultation. After obtaining informed consent from the client, a structured non-participatory observation of the client-provider interaction will be undertaken to determine the quality of care provided. Subsequent sessions for which consent has been received will be observed until 6 randomly selected antenatal, delivery, PNC and FP clients in each facility have been observed. This includes government, non-governmental, faith-based and private facilities. We acknowledge that observing client provider interaction may bias in a positive direction the results obtained on quality of care. We will be spending more than one day at each site, so that the presence of the research team becomes more familiar and the behavior of the providers becomes more normative. Samples of clients attending each type of consultation will be recruited if they meet the following eligibility criteria: are accessing family planning or maternity care including postnatal care for themselves (and/or their babies) at delivery or one of the pre-discharge, one-week or six-week postpartum consultation times; are aged over 18 years (the small proportion of clients that are less than 18 years will not justify the difficulties in obtaining parental/guardian permission); are aged below 45 years (the small proportion of women giving birth/accessing FP above this age will be excluded); give their informed consent for their consultation to be observed and the key actions taken recorded, and to be interviewed on exiting from the consultation. All women satisfying these inclusion criteria will be recruited until the required sample sizes have been reached. Exit interviews will be held with each client who was observed by a trained interviewer to ascertain their perceptions of the service received. The client will be introduced to the interviewer following the CPI. To measure the magnitude of changes in the quality of services provided, composite summary scores will be developed for a series of key indicators by aggregating the mean scores of key items being assessed for each individual client-provider interaction being observed. This scoring system will categorize whether an accepted standard of quality has been met. For each study group, a mean score per group will be calculated for each indicator and for the composite summary score to enable statistical comparisons to be made between experimental and comparison groups over time. Examples of the types of individual items and key indicators are presented in Table ​Table22. Examples of indicators to make composite scores of quality of care Population Council will conduct baseline and end-line population level surveys with a randomly selected sample of men and women aged 15-45 years from the catchment communities of all study facilities and have had a pregnancy or a pregnant partner during the last 12 months or started a new FP method. Adolescents 15 – 17 years will only be interviewed following parental consent. Surveys will compare patterns of service use and perception and to compare any differences between communities that have ready access to the voucher and communities that do not have access. Facility catchment areas will be identified as either “experiment” or “control” based on the presence of an accredited facility. A complete list of villages (administratively referred to as “sub-locations”) in the catchment areas will be made and a sample taken from areas with and without an accredited facility. In each selected village, teams will randomly select a seed household and then identify every third household along village roads. At each of these “core” households, teams will inquire about pregnancies in the core household and in the two adjacent households. In this way, teams will identify and visit households in which a pregnancy or recent delivery is reported. Community members will be asked standardized, and sex-specific, questions on access and use of services, attitudes, experiences and reasons for service use/non-use of voucher and RH issues. This will offer a comparison between voucher holders vs. non-voucher holders, as well as offer insights into preferences for the accredited services and reasons for use/non-use of these. Enumerators will be trained on proper technique and ethical conduct. Training of research assistants is likely to take a minimum of eight days including a pretest in the field. Table ​Table33 describes examples of operational results and indicators to compare accredited and non- accredited facilities and communities. Paper questionnaires and portable digital assistants (PDAs) will be used to capture quantitative data. Data from paper questionnaires will be keyed into Epidata 3.1 and exported into Stata 10 for analysis. Data from PDAs will downloaded into an MS Access database before being exported into Stata 10 for analysis. Examples of operational results and indicators to be used to compare results from the accredited and non-accredited health facilities and communities In order to enhance the findings from population surveys and address unforeseen questions arising from other components of the study, in-depth interviews (IDIs) and focus group discussions (FGDs) with healthcare providers and key informants will be conducted. These interviews will be used to gain a deeper understanding into the motivations, perceptions, and priorities of the healthcare providers regarding voucher and accreditation. The provider IDIs and FGDs will focus more specifically on: services offered; attitudes towards voucher and accreditation, including effects on workload; benefits and challenges of the voucher and accreditation; perception of clients’ views; the referral system and other healthcare needs. Before the FGDs and IDIs, participants will be provided with any necessary information to complete their understanding of the nature of the research. The researcher will discuss with the participants their experience with the research in order to monitor any unforeseen negative effects or misconceptions. FGDs will be carried out with groups of male and female voucher and non-voucher users (aged 18 years and over) as well as with providers. These will take place alongside the baseline and endline surveys, as well as on an ad hoc basis when needed during the project. These will be used to gain a deeper understanding into the motivations, perceptions, and priorities of the local community regarding vouchers and service use. The FGDs will address the following broad themes: motivations for healthcare use and selection/use of the RH services; attitudes towards voucher and accreditation; communication/interaction with different providers; contraceptive and sexual health behavior, including communication with partners and other community members about RH services. All participants in the FGDs will be requested to respect confidentiality and to agree to not to divulge any information heard during the discussion outside of the group. FGDs of one to two hours will be held in four randomly selected populations within the surveyed districts: 1 FGD: 6-8 younger women who are currently or have been voucher users (< 25 years) 1 FGD: 6-8 older women who are currently or have been voucher users (25 years & over) 1 FGD: 6-8 younger women who have never used voucher (< 25 years) 1 FGD: 6-8 older women who are have never used vouchers (25 years & over) 1 FGD: 6-8 younger men who/or partners are currently or have been voucher users (< 25 years) 1 FGD: 6-8 older men who are currently or have been voucher users (25 years & over) 1 FGD: 6-8 younger men who/or partners who have never used voucher (< 25 years) 1 FGD: 6-8 older men who/partners who have never used vouchers (> 25 years) The Data Management Unit in Population Council will store all data in password protected computers. Hard copies of questionnaires, anonymised transcriptions and tapes of the group discussions will be stored securely in a locked cabinet, in accordance with the Population Council policy and the Kenya Data Protection Policy. Analyses of facility data will be undertaken and the proportion of women receiving an acceptable quality of service will be calculated. The methodology to calculate the proportion of women receiving an acceptable quality is similar to the Lot Quality Assurance Sampling (LQAS) approach that has been used in Kenya and elsewhere for assessing quality [7]. LQAS follows the principle that an entire group (lot) of services is deemed poor quality if a certain proportion within a small sample does not reach a minimum standard. LQAS applies cumulative probabilities calculated with a binomial formula to select small sample sizes and decision criteria for judging a group of providers. In addition, time series analyses will be conducted in order to estimate: mean monthly number of clients obtaining RH services, by type, at accredited and non-accredited providers; mean monthly number and proportion of clients in the lowest economic quintile obtaining RH services, by type, at accredited and non-accredited facilities and proportion of voucher services among all RH services at accredited facilities. Population-level surveys provide the opportunity to measure reproductive health indicators, including both reported health status, behaviors and healthcare utilization, among populations being served by a voucher program and comparable populations not served by the voucher program. Statistical comparisons between these indicators can then be used to detect any differences between the populations at 1% and 5% level of significance. We will also compare concentration index scores for selected RH indicators calculated from the data collected among accredited and non-accredited populations. The concentration index is a widely used indicator for quantifying the degree of income related inequality in a specific health indicator and will be used to provide evidence of the extent to which voucher and accreditation approach reduces inequities. Qualitative data will be captured on paper and audio tapes and later transcribed into MS Word before exporting into QSR Nvivo 8 for analysis using thematic framework.