Evaluating global health partnerships: A case study of a gavi HPV vaccine application process in Uganda

Background: Global health partnerships have grown rapidly in number and scope, yet there has been less emphasis on their evaluation. Gavi, the Vaccine Alliance, is one such public-private partnership; in Gavi-eligible countries partnerships are dynamic networks of immunization actors who work together to support all stages and aspects of Gavi support. This paper describes a conceptual framework-the partnership framework-and analytic approach for evaluating the perceptions of partnerships’ added value as well as the results from an application to one case in Uganda. Methods: We used a mixed-methods case study design embedded in the Gavi Full Country Evaluations (FCE) to test the partnership framework on Uganda’s human papillomavirus (HPV) vaccine application partnership. Data from document review, interviews, and social network surveys enabled the testing of the relationships between partnership framework domains (context, structure, practices, performance, and outcomes). Topic guides were based on the framework domains and network surveys identified working together relationships, professional trust, and perceptions of the effectiveness, efficiency, and legitimacy of the partnership’s role in this process. Results: Data from seven in-depth interviews, 11 network surveys and document review were analyzed according to the partnership framework, confirming relationships between the framework domains. Trust was an important contributor to the perceived effectiveness of the process. The network was structured around the EPI program, who was considered the leader of this process. While the structure and composition of the network was largely viewed as supporting an effective and legitimate process, the absence of the Ministry of Education (MoE) may have had downstream consequences if this study’s results had not been shared with the Ministry of Health (MoH) and acted upon. The partnership was not perceived to have increased the efficiency of the process, perhaps as a result of unclear or absent guidelines around roles and responsibilities. Conclusion: The health and functioning of global health partnerships can be evaluated using the framework and approach presented here. Network theory and methods added value to the conceptual and analytic processes and we recommend applying this approach to other global health partnerships to ensure that they are meeting the complex challenges they were designed to address. In this section, we elaborate on the evaluation design and implementation so that it can be easily replicated. We provide sample data collection tools in Supplementary Files 1 and 2. Applying the partnership framework required a mixed-methods case study design37 embedded in an ongoing prospective evaluation of Alliance support (the FCE).15 Among the four FCE countries, Uganda was chosen as the initial case country to implement the partnership analysis because of local interest and the recent completion of a potential case – the development of the funding application to the Alliance for HPV vaccine introduction at the national level. In the partnership analysis approach, the ‘case’ can be defined as a process with a specified outcome of the process, for example an application for new vaccine support, resulting in a submission to the Alliance; or the planning process culminating in the launch of a new vaccine. Defining the case as process facilitates cross-country comparisons and encourages empirical identification of the network of actors who have participated. It ensures that the outcomes (Figure 1, Box 5) are well-defined and measurable, thus, enabling attribution. The HPV vaccine application process was chosen as a suitable case because of its timeliness in relation to planned data collection and the potential of applying lessons learned to both the ongoing implementation of HPV vaccine in Uganda as well as to other new vaccine applications in Uganda and elsewhere. In addition, it was of theoretic and programmatic interest to understand how a partnership around HPV vaccine, which targets adolescent girls to prevent cervical cancer, might involve a different group of stakeholders than traditional childhood vaccines. As the Partners’ Engagement Framework moves to include ‘expanded’ partners,14 the process and outcomes of adding new partners is of particular relevance. Finally, although certain immunization activities (ie, vaccine implementation) involved partners spanning administrative levels and jurisdictions, a national level process was chosen for ease of access to potential respondents and data collection. Data collection included document review and in-depth interviews, including a structured network survey, with key informants. The document review component informed the development of topic guides and supported the identification of initial interview respondents. Documents included the final report of Uganda’s HPV vaccine demonstration project,38 minutes of application meetings, the application submission to Gavi,39 and Gavi’s decision letter. In-depth semi-structured interviews were conducted with individuals involved in the HPV vaccine application process. Respondents were identified based on the local research team’s in-depth knowledge of the process, and augmented by document review. Interviewers, who were members of the local FCE research teams, followed topic guides based on the partnership framework domains (see Supplementary File 1 for sample topic guide). The aim of the interview was to elicit the respondent’s perception of the overall application process, as well as of constructs within each domain in order to test the relationships between the constructs and domains. All interviews were conducted in Kampala, the capital city of Uganda. Interviews lasted on average 45 minutes. Notes were taken during the interview and were expanded immediately following the interview. During the interviews, the research team also administered a structured network survey adapted from Provan et al31 (see Supplementary File 2 for sample survey). This survey was administered orally at the start of the interview. One attempt to administer it over email was unsuccessful and required visiting that respondent in person; attempts to leave it to the end of the interview resulted in rushing through it, which is a particular challenge when interviewing policy-makers and other policy elites. The survey began by asking the respondent to provide the names of the individual people he or she worked with on the HPV vaccine application. This open-ended ‘name generator’ encourages the empirical identification of the true actors in a network, as opposed to a roster approach (ie, a list of names), which might bias the network towards who is named in formal documentation.33 We chose to define our units of survey observation, as well as the node level units of analysis in the networks, as individuals rather than organizations. This decision was based on the local research team’s in-depth understanding of partner dynamics and previous policy network mapping studies that suggested that decision-making was largely relational on an individual level, and that personal behaviors and attributes should not be ignored.35 For each name provided, the respondent was also asked whether they shared information with that person during the HPV vaccine application process, how many years they have known that person, and their level of professional trust for that person. Professional trust was defined using the following prompt: “When we say ‘trust,’ we mean can you trust that organization to keep their word, to do a good job, and to respond to your organization’s needs?” and rated on a scale of 1-4 (little trust – high trust; see survey Supplementary File 2 for more details). The question of trust often led to open-ended responses which were recorded and probed on. Interviewers used a fluid approach where they probed on responses during the survey to discuss related domains in the interview topic guide. Following the network survey, respondents were asked to indicate their perceptions of the effectiveness, efficiency, and country ownership of the partnership. To do this, the interviewers read a list of statements adapted from Provan and Milward18 and were asked to indicate “occurred” or “did not occur” for each. For example, “Planned activities were executed with greater quality” was one indicator of effectiveness; “Reduction in financial cost of process” was one indicator of efficiency, and “Increased legitimacy of decisions made” was one indicator of country ownership. Negative statements (eg, “Unnecessary management burden on my organization”) were also included (see Supplementary File 2 for complete list of questions). Interviewers found that this component of the survey was easier to complete if they passed the survey instrument to the respondent to read and self-administer. Again, interviewers recorded verbal comments and open-ended responses and probed where appropriate. The survey also included questions about the respondent’s basic job and demographic characteristics, which were completed by the research team prior to the interview to the extent possible. Job and demographic characteristics were also completed, to the extent possible by searching meeting minutes or the Internet, for identified network members who were not surveyed. A ‘snowball,’ or respondent-driven sampling, approach was used to identify respondents for this study.33 This approach was used because it encourages the empirical identification of true actors in a network as opposed to providing a list of people to the respondents which might bisas the network. Due to the prospective nature of the evaluation, the team identified the initial respondents through document review and participant observation during planning meetings. The goal of this process was to identify a sub-set of individuals who were likely to be central to the network without being overly homogenous or closed as a group. Using an open name generator, names of individuals mentioned in the interviews were added to a master list of network members and these were then approached for an interview. In this approach to defining a network — as opposed to the roster approach which assumes a fixed and known member list — the researcher must define a boundary for data collection but attempt to capture the entire census of network members within that boundary. We chose to stop including new names (ie, the network boundary) when a round of names elicited fewer new names than the previous round. This is a common decision point for dispersed networks, such as policy networks, as it suitably limits bias while not overtaxing resources.40 The research team read the qualitative interview notes together and coded text segments by hand according to a pre-determined coding structure based on the framework categories. Additional codes included codes for interactions between categories, and the application process for inactivated poliovirus vaccine (IPV) which had occurred more recently than the HPV vaccine application process and was mentioned often during interviews. The team read the coded text segments and wrote a memo for each theme summarizing key interpretations and findings. Immediately following interviews, network data were entered in matrix form in a MS Excel workbook. Names were entered in rows and columns and the existence of a working relationship was entered in each node-alter cell, weighted by the reported trust score. A second matrix recorded information exchange ties, entered as binary (0 = no tie was reported; 1 = a tie was reported). Respondent attributes and perceived outcomes were entered in a third worksheet, and descriptive analyses performed in Stata. The two resulting network sociomatrices were imported into UCINET software41 where the maximum value of two ties was taken in cases of asymmetry. Density, centralization, and degree centrality were computed based on existing algorithms; a dichotomized version of the working relationship network was also analyzed. Network maps were produced for the networks using NetDraw.42 Uganda National Expanded Program on Immunization (UNEPI) is responsible for immunization under the Ministry of Health (MoH) in Uganda and is headed by the Assistant Commissioner of Health Services also known as the Expanded Program on Immunization manager (EPI manager). UNEPI is situated in the Department of National Communicable Diseases Control (NDC) within the Directorate of Clinical and Community Services. The UNEPI program is responsible for policy, standards and priority setting, capacity building, coordinating with other stakeholders and partners, resource mobilization, monitoring, and technical support supervision to districts.43 UNEPI links with other MoH departments and divisions through Technical Working Groups as well as Senior and Top Management committees. EPI activities are organized during monthly technical committee meetings consisting of EPI country partners led by the EPI manager. EPI has various partners including (1) Public partners such as National Medical Stores (NMS), district administrations and health facilities; (2) Development partners such as United Nations Children’s Fund (UNICEF), the WHO, and bilateral donors; (3) International non-governmental organizations (INGOs) such as PATH, SABIN Vaccine Institute, African Field Epidemiology Network (AFENET), and Maternal and Child Health Integrated Program (MCHIP). Uganda has benefited from Alliance support since 2001 with the introduction of hepatitis B vaccine and immunization services support (ISS); since that time it has introduced Haemophilus influenzae (Hib) vaccine and pneumococcal conjugate vaccine (PCV), and utilized cash support for injection safety (INS) and HSS windows, receiving a total of $190.6M in Alliance funds to date. Gavi support to Uganda was suspended in 2006 following financial irregularities and was re-commenced in 2012 when those irregularities were resolved. In Uganda, cervical cancer accounts for 40% of all cancers recorded by the cancer registry, and over 80% of women with cervical cancer are diagnosed with advanced disease.44 Cervical cancer is caused by HPV which is a sexually transmitted infection. In Uganda, the annual age-standardized incidence of cervical cancer is estimated at 44.4 per 100 000 women per year and age-standardized mortality rate estimated at 27.2 per 100 000 women per year (2012 estimates).45 From WHO projections, deaths are predicted to increase by nearly 80% by 2030 in mostly low- and middle-income countries, Uganda inclusive.44 Fortunately, the disease can easily be prevented through HPV vaccinations, screening and treatment. A vaccine to prevent cervical cancer was approved and licensed in the United States in 2004 and in Uganda in 2007. In order to benefit from Alliance support for HPV vaccine, countries are required to demonstrate their ability to deliver HPV vaccines to adolescent girls prior to application for national rollout. Uganda was among the countries selected by PATH to undertake an HPV Vaccines project along with India, Peru, and Vietnam. The project in Uganda was implemented by UNEPI of the MoH with technical support from PATH, and operations research was conducted by the Child Health and Development Centre (CHDC) and PATH. The demonstration project aimed at assessing the feasibility, acceptability and cost of delivering HPV vaccine. The demonstration project was initially implemented in two selected districts in 2008, each testing a different approach, but later scaled up to 12 additional districts in 2009. In Nakasongola district, delivery of HPV vaccine was tested through the biannual Child Days Plus (CDP) approach and the target population was girls 10 years of age. In Ibanda district, a school-based approach was used and the target population was based on school grade (Primary 5) or 10 years of age for girls who were not attending school. The demonstration was considered successful and the accompanying report indicated that the HPV vaccine was highly acceptable in communities and that implementation was feasible.38 A coverage survey in 2009 showed 88.9% coverage with the school-based delivery strategy and 60.7% coverage with the CDP delivery strategy.38 Based on the success in the two districts, the demonstration project shifted to using a combined approach of integrating the CDP with school-based immunization, and HPV vaccine immunization was extended to 12 additional districts in 2012. The new combined approach targeted all girls in Primary 4, regardless of age, and 10-year-old girls who were not in school. Vaccination of the first cohort of girls in the 12 new districts began in September 2012, the second dose was administered in November 2012, and the third dose between March and August 2013. Its success provided evidence to the Government of Uganda about when and how best to introduce the HPV vaccine country wide prior to application for national rollout as this was a Gavi requirement. The HPV vaccine delivery model decided upon then however, has changed due to its feasibility and financial sustainability. Following the successful demonstration, Uganda then made a decision to apply to Gavi for national introduction of HPV vaccine. The application for national introduction of HPV vaccine was prepared between May and September 2013. The initial application was submitted by the Government of Uganda in September 2013 and was approved by Gavi in March 2014.