Protocol for the evaluation of a pay for performance programme in Pwani region in Tanzania: A controlled before and after study

Background: The use of supply-side incentives to increase health service utilisation and enhance service quality is gaining momentum in many low- and middle-income countries. However, there is a paucity of evidence on the impact of such schemes, their cost-effectiveness, and the process of implementation and potential unintended consequences in these settings. A pay for performance (P4P) programme was introduced in Pwani region of Tanzania in 2011.Methods/design: An evaluation of the programme will be carried out to inform a potential national rollout. A controlled before and after study will examine the effect of the P4P programme on quality, coverage, and cost of targeted maternal and newborn healthcare services and selected non-targeted services at facilities in Tanzania. Data will be collected from a survey of 75 facilities, 750 patients exiting consultations, over 75 health workers, and 1,500 households of women who delivered in the previous year, in all seven intervention districts. Data will be collected from the same number of respondents in four control districts. A process evaluation will examine: whether the P4P programme was implemented as planned; stakeholder response to the programme and its acceptability; and implementation bottlenecks and facilitating factors. Three rounds of process data collection will be conducted including a review of available P4P documents, individual interviews and focus group discussions with key informants working at facility and district level in five of the intervention districts, and at the regional and national levels. An economic evaluation will measure the cost-effectiveness of P4P relative to current practice from a societal perspective.Discussion: This evaluation will contribute robust evidence on the impact and cost-effectiveness of P4P in a low income setting, as well as generate a better understanding of the feasibility of integrating complex intervention packages like P4P within health systems in resource poor settings. © 2013 Borghi et al.; licensee BioMed Central Ltd.

The impact evaluation will employ a controlled before and after study design. Surveys will be undertaken within all seven districts in Pwani region where P4P is being implemented before and after its introduction and also among four control districts with no P4P, namely: Kilwa, Mvomero, Morogoro town, and Morogoro rural. Control districts were selected from two neighboring regions such that they were as similar as possible to intervention districts in relation to poverty and literacy rates, the rate of institutional deliveries, infant mortality, population per health facility, and the number of children under one year of age per capita. Care was also taken to avoid districts where programmes were underway to improve maternal and child health, which could confound results. The impact evaluation relies on four tools that will be administered at baseline and endline: a health facility survey, a health worker survey, a survey of patients exiting facilities, and a survey of women who delivered in the previous 12 months (Figure 2). The facility survey, health worker survey, and exit interviews will be conducted at all sampled facilities. The household survey will be administered to households within the catchment areas of these facilities to complement the data compiled during the facility survey [11]. Overview of impact evaluation data collection tools and sampling strategy. The health facility survey aims to measure the effects of P4P on service availability and provision at the sampled facilities. It is comprised of three sections. In the first section, questions focus on basic service provision within the facility (staffing levels, opening hours, facility management, as well as facility infrastructure). The second section of the survey compiles equipment and drug availability data. The third section captures HMIS data on service utilisation, facility expenditures and revenues for the 12-month period before P4P was implemented (January to December 2010 (at baseline) and the period from January 2011 to December 2012 (at endline). The health facility survey will be administered to the facility in-charge or in his/her absence to a knowledgeable health worker or administrator. The health worker survey tool aims to measure the effects of P4P on health workers’ working conditions and attitudes towards work at the selected facilities. The exit interview survey primarily intends to measure the effect of the P4P initiative on a range of subjective and objective indicators of quality of care for targeted and selected non-targeted services. The survey will also examine the effect of P4P on the cost of these services. Respondents eligible for interview include women of reproductive age (aged between 16 to 49 years) attending antenatal or postnatal care, or women with children under-one year of age coming for a preventive check up or an immunisation for the baby. These patients will respond to questions linked to the services targeted by P4P. Patients attending care for non-targeted services will also be interviewed. Non-targeted service users will include: women of reproductive age who are not pregnant, or children under five years of age accompanied by a woman of reproductive age, reporting with fever and no cough (as a proxy for malaria), or fever and cough (as a proxy for acute respiratory infection – ARI), or diarrhoea. These conditions were chosen as they were the three most significant conditions reported at outpatient departments in Tanzania in 2009. A survey of women who had delivered within the previous 12 months will also be carried out. The women’s survey addresses the effects of P4P on service use during pregnancy, place of delivery, birth weight and postpartum care and care for the newborn as well as related costs and service satisfaction. Household socioeconomic status is also measured in this survey. The core indicators for each of the surveys are shown in Table 1. Overview of core indicators for each of the surveys The health facility is the primary sampling unit. Facilities were sampled from those that were eligible to participate in the P4P scheme (they offered reproductive and child health services and had submitted a one year backlog of HMIS data, enabling performance targets to be measured). All eligible hospitals (n = 6) and health centres (n = 16) from the intervention districts were included in the sample along with all eligible non-public dispensaries (n = 11). An equivalent number of facilities in control areas were sampled by level of care. Public dispensaries were sampled at random with probability proportional to the number of public dispensaries in a given district (n = 42). In control areas, hospitals and health centres were sampled to match as closely as possible with selected intervention facilities in terms of annual outpatient care visits and staffing levels. A total of 75 health facilities were sampled from intervention districts, and 75 were sampled from control districts (Figure 2). In Pwani region, 46% of all facilities in the region were included in the sample. The aim of the sampling procedure for the selection of health facilities was to seek district representation, while for the health worker survey it was to obtain the views, attitudes, and perceptions of at least one health worker per facility. No sample size calculation was therefore carried out. In dispensaries, one health worker will be interviewed. If more than one health worker is on duty, preference will be given to someone other than the in-charge to avoid overburdening them with questions (as they will be interviewed for the facility survey). In health centres and hospitals, two health workers will be interviewed. The health workers will be selected at random from those who are on duty at the facility on the day the interviewers are present. For the exit and household surveys, the sample size calculation was based on the formula by Hayes and Bennett, 1999, adjusted for the cluster design of the study at the facility level [23]. We estimated the size needed to detect a 17% reduction in waiting time from 114 minutes (SD 66) [24] to 95 minutes, with a k value of 0.25, 80% power and a significance level at of 5% (two tailed test). We did not increase the sample size to account for non-response because response rates of 100% were observed in previous studies in Tanzania [25,26]. The estimated sample size was 10 exit interviews per facility, equivalent to a total of 750 interviews in intervention and control areas respectively. A balance in the number of interviews between antenatal, postnatal clients and non-targeted services will be sought. Exit interview patients will be approached by interviewers upon entry to the health facility and asked a series of screening questions to check their eligibility. Eligible patients will then be asked for their informed consent to participate in the study. This process will be repeated until the required number of eligible consenting respondents has been attained. Participants will then be monitored by the interviewers from their time of arrival at the facility until their time of departure, and the waiting and consultation times will be measured using a stopwatch. The cadre of the provider seen by the woman/child will also be recorded by the interviewer. The survey tool will be administered to patients upon completion of their consultation in a quiet location within the facility, at distance from providers and other patients. For the household survey, we estimated that the required sample size to detect an 11 percentage point increase in institutional deliveries (from 50 to 61%), with k value of 0.25, 90% power, and a significance level at of 5% (two tailed test), and a 90% response rate, was 20 households per cluster, equivalent to 1,500 women per study arm. The following process was followed to identify eligible households. First, villages were sampled from the facility catchment area; for all dispensaries, the village where the facility is located will be selected by the research team; for health centres and hospitals, two villages will be selected at random from all villages lying within the ward where the facility is located. Second, all hamlets (comprising approximately 100 households) within this village/these villages, and located within the catchment area of the facility will be identified; a random sample of four of these hamlets will then be selected. In the case of dispensaries, all four hamlets will reside within the selected village. In the case of health centres and hospitals, two hamlets will be sampled from each village. Third, five households will be sampled from each of the selected hamlets, amounting to a total of 20 households within each facility’s catchment area; households will be selected at random from the selected hamlets using a modified Expanded Programme of Immunisation (EPI) type sampling scheme that ensures an equal chance of any household being selected.1 The process evaluation aims to compare what was planned to what is actually happening in practice, incorporating issues related to the acceptability of the scheme to various stakeholders, as well as the context of implementation, in an attempt to understand potential facilitating or debilitating factors that might explain variation in implementation. We will track indicators of health worker and manager satisfaction, supervision and verification, and implementation constraints and facilitators; as well as examine how specific issues such as facility ownership, facility resources, and governance structures affect the implementation of the programme at the facility level. The process data will be used to get a better understanding of the reasons for any P4P effect or a lack of effect, as well as potential unintended consequences. Table 2 presents a list of qualitative indicators used to monitor the progress of implementation at district level. Process monitoring subthemes Three rounds of qualitative data will be collected throughout the programme life cycle to explore how perspectives and knowledge change over the course of implementation and to inform the continued implementation of the programme. The first round of data collection will take place in a sample of 15 health facilities from five of the seven intervention districts. A total of 54 interviews and four focus group discussions (FGDs) will be conducted with key stakeholders at facility, district, regional, and national levels. Districts were selected to offer variation in relation to geographical location, skilled birth attendance coverage and achieved performance targets during the first cycle. Three facilities were purposively selected within each district to offer a mix of ownership and level of care as well as differing levels of baseline performance. In rounds two and three, data will be collected from facilities in a subsample of the round one districts using a mix of individual interviews and focus group discussions. Interview guides for each round were developed for the following stakeholder groups (the CHMTs, the district P4P focal person, health facility in-charges, health workers, the RHMT, the Health Facility Governing Committee (HFGC), and national level stakeholders). Contextual information will also be collected in each district. The objective of the economic evaluation is to ascertain whether P4P represents value for money. The study will be carried out from a societal perspective, which includes all agencies or bodies that are involved in implementation or who incur costs or may be affected by the intervention, for example: the implementers (e.g., government (MOHSW, the Prime Minister’s Office – Regional Administration and Local Government (PMO-RALG)), the NHIF, and CHAI; as well as those who are affected by implementation and may incur costs as a result (e.g., households, health workers, district health managers). We will estimate both the financial costs of each activity (i.e., what is paid out by the funding body—all financial transactions), as well as the economic costs, which include the value of all resources valued at their opportunity cost. Similarly, any donated or subsidised items will be valued at market prices. Costs and cost-effectiveness of the P4P programme will be compared to the current situation (doing nothing). Costs will be classified according to project activities (start-up activities and ongoing activities) as well as by resource inputs (recurrent items such as staff, supplies, transport etc., as well as capital costs such as equipment, vehicle etc.). Capital items will be annualised over the lifetime of the project. Start-up costs will include those resources used during training of providers and other stakeholders; contracting; entering baseline HMIS data, and target setting; provision of guidelines; and establishment of a steering committee. Ongoing costs will include those resources used during data processing; verification; strategies adopted by facilities, districts, and regions to meet targets; fund payout; and review and modification of targets. Programme cost data will be derived from project accounts and through interviews with key implementation stakeholders. Interviews with district, regional, and national stakeholders will be undertaken to allocate staff time to activities as well as to identify and value resources and time invested in the programme that is not paid for. Facility data will be used to ascertain to what extent increased service use results in costs to the health system in terms of additional staff or beds, for example, or to what extent there is spare capacity and such an increase can be readily absorbed within the system. Household costs will be captured during the baseline and endline household surveys and will enable the measurement of the societal costs of an eventual service increase due to the programme, as well as changes in the levels of out of pocket payments. To provide quality assurance within the impact evaluation, survey data will be checked by supervisors at the end of each day of data collection. Household, exit, and health worker interview data will be collected using hand held devices (Samsung Galaxy tablets 7.0 and Huawei IDEOS phones) with skip and quality check functions to minimize data entry error. Facility data will be captured on paper and double entered. Data will be backed up on CD each day in a Microsoft Access Database, and converted to Stata for analysis. Hard copies of questionnaires will be stored in a lockable room. Electronic output will be anonymised. Interviews and focus groups conducted as part of the process and economic evaluation will be conducted in Kiswahili and recorded using sound digital recorders. Audio sound files will be transcribed and translated into English by the bilingual researchers who conducted the interviews. All translated data obtained from interviews and focus group discussions will be entered into QSR Nvivo 9 for data management, for the process evaluation, and into Microsoft Excel for the economic evaluation. Impact data will be checked first for consistency and after export to Stata, data cleaning will be undertaken. Binary variables (Yes = 1, No = 0) will be created for all categorical variables. All binary and continuous variables will be summarized by calculating means and standard deviations. A comparison of all variables between intervention and control arms will be made at baseline. Tests of differences in means between intervention and control groups will be conducted using the Adjusted Wald F-test. Principal component analysis (PCA) will be used for creating socioeconomic status (SES) indices for household and exit interview data analysis using data collected on household size and characteristics, access to utilities, durable asset ownership, food security, household expenditures, head of household marital status, highest level of education attained, and main occupation. Data that use a Likert scale (e.g., dissatisfied = 1, neither satisfied nor dissatisfied = 2, satisfied = 3) will be analyzed by calculating individual mean scores for each variable. Factor analysis will be used on patient satisfaction and health worker motivation data for identifying the underlying factors or themes within the data. At endline, we will compare the main outcome indicators for each of the survey tools between intervention and control arms, using data for twelve months of intervention. We will estimate a multivariate regression specification of the difference – in – difference model in which an individual’s (woman, patient, health worker, facility) outcome is regressed against a dummy variable, indicating whether the facility was eligible for P4P (i.e., providing reproductive and child health services, had submitted the backlog of HMIS data, and performance reports), a facility fixed effect, a year indicator, and a series of individual and household characteristics (in the case of patients and women). For household and exit data, we will calculate robust standard errors, clustered at the facility level to correct for correlation of the error terms across patients within facilities, and across households in facility catchment areas. Transcripts from interviews and focus group discussions will be read systematically and independently by each of the researchers, and coded applying thematic content analysis, which identifies recurrent themes that form a cluster of linked categories containing similar meanings. To validate findings, we will triangulate data across respondent groups and look for supporting documentary evidence, where available. Analyses will be undertaken on an ongoing basis as transcripts and other information from the study sites become available. The evaluation study was approved by the Institutional Review Board of the Ifakara Health Institute and the Ethics Review Board of the London School of Hygiene & Tropical Medicine. The study design and protocol were also approved by the P4P Management Team that includes members of the MOHSW. Presentations of the proposed research methods were also made to the P4P steering committee that includes CHAI, MOHSW and the Government of Norway. Prior to undertaking data collection, letters were sent to respective District Executive Directors (DEDs) copied to District Medical Officers (DMOs) informing them of the study and its objectives. Subsequently, visits were made to the DMOs to agree on dates for data collection. An information sheet was left at the DMO’s office. Information sheets and consent forms were provided to all those participating in the study. Written consent was obtained prior to undertaking all interviews and FGDs.

N/A

• Dima Health