Validation and calibration of a computer simulation model of pediatric HIV infection

Background: Computer simulation models can project long-term patient outcomes and inform health policy. We internally validated and then calibrated a model of HIV disease in children before initiation of antiretroviral therapy to provide a framework against which to compare the impact of pediatric HIV treatment strategies. Methods: We developed a patient-level (Monte Carlo) model of HIV progression among untreated children <5 years of age, using the Cost-Effectiveness of Preventing AIDS Complications model framework: the CEPAC-Pediatric model. We populated the model with data on opportunistic infection and mortality risks from the International Epidemiologic Database to Evaluate AIDS (IeDEA), with mean CD4% at birth (42%) and mean CD4% decline (1.4%/ month) from the Women and Infants' Transmission Study (WITS). We internally validated the model by varying WITS-derived CD4% data, comparing the corresponding model-generated survival curves to empirical survival curves from IeDEA, and identifying best-fitting parameter sets as those with a root-mean square error (RMSE) <0.01. We then calibrated the model to other African settings by systematically varying immunologic and HIV mortality-related input parameters. Model-generated survival curves for children aged 0-60 months were compared, again using RMSE, to UNAIDS data from >1,300 untreated, HIV-infected African children. Results: In internal validation analyses, model-generated survival curves fit IeDEA data well; modeled and observed survival at 16 months of age were 91.2% and 91.1%, respectively. RMSE varied widely with variations in CD4% parameters; the best fitting parameter set (RMSE = 0.00423) resulted when CD4% was 45% at birth and declined by 6%/month (ages 0-3 months) and 0.3%/month (ages >3 months). In calibration analyses, increases in IeDEA-derived mortality risks were necessary to fit UNAIDS survival data. Conclusions: The CEPAC-Pediatric model performed well in internal validation analyses. Increases in modeled mortality risks required to match UNAIDS data highlight the importance of pre-enrollment mortality in many pediatric cohort studies. © 2013 Ciaranello et al. This work was approved by the Partners Healthcare IRB. We developed a microsimulation model of pediatric HIV disease progression, the CEPAC-Pediatric model. As in the adult CEPAC model, clinical events are first simulated and validated in the absence of ART (a “natural history” model), in order to describe disease progression in the absence of ART and to provide a framework against which to compare the impact of HIV treatment [4,5]. In collaboration with the International Epidemiologic Databases to Evaluate AIDS (IeDEA) consortium [26,27], we derived model input parameters for the CEPAC-Pediatric model, reflecting outcomes in HIV-infected children prior to the initiation of ART. These model input data included rates of WHO Stage 3 and Stage 4 clinical events, tuberculosis (TB), and mortality [28], stratified by age and CD4%. Internal model validation is a formal methodology to assess the validity of model structure. In internal calibration, the empiric data values used in the modeling analysis (“model inputs”) are compared to model-generated results (“model outputs”), in order to assess model performance for analyses related to a single data set [19,24,25,29–31]. We conducted internal model validation by comparing model-generated results to the clinical event and mortality risks observed in the same IeDEA cohort that contributed model input data. For internal validation, selected immunologic parameters that were not available from IeDEA were based on data from the Women and Infants’ Transmission Study (WITS) [32–34]. Model calibration is a methodology distinct from validation. In model calibration, sometimes referred to as “model fitting,” investigators identify the values for key data parameters that will allow model projections to match empiric observations. Calibration seeks to explicitly modify model input parameters, in order to make the model useful for predicting outcomes in cohorts or datasets distinct from the dataset used in internal validation [19,23–25,29]. The IeDEA East African cohort represents a highly selected population of children with excellent access to HIV care. In order to produce analyses more generalizable to other African settings, we identified data parameter sets that allowed model output to match published survival curves from a pooled UNAIDS analysis of >1,300 untreated, perinatally HIV-infected children in eight sub-Saharan African countries [30,35–38]. The CEPAC-Pediatric model is a first-order, patient-level Monte Carlo simulation model (Figure 1). Infants enter the natural history model at birth, and are assumed to have been HIV-infected either in utero or during delivery (intrapartum). A random number generator is used to draw from user-specified distributions of maternal HIV status (CD4 ≤350/μL or >350/μL; receiving or not receiving ART), PMTCT exposure; breastfeeding or replacement feeding; and infant CD4% at birth (percentage of total lymphocytes that are CD4+ cells). We modeled CD4% as the primary immunologic measure for children <5 years of age because absolute CD4 count declines dramatically with age, even in the absence of HIV infection, and CD4% is therefore a more stable marker of immune function as children age [3]. In the absence of ART, each simulated child's CD4% declines monthly at a user-specified rate until they reach age five. Older children, adolescents, and adults can be simulated in dedicated analyses using the CEPAC adult model; in conjunction with the CEPAC-Pediatric model, this permits projections over the lifetimes of HIV-infected children [4–6]. A schematic of the Cost-Effectiveness of Preventing AIDS Complications (CEPAC)-Pediatric natural history model (see Methods for details). Disease progression in the CEPAC-Pediatric model is characterized by monthly transitions among health states, including chronic HIV infection, acute clinical events, and death (Figure 1). Transitions between these health states depend on current age (0-2, 3-5, 6-8, 9-11, 12-17, 18-23, 24-35, 36-47, and 48-59 months) and CD4% (<15%, 15-24%, and ≥25%) during each month of the simulation. Simulated patients face monthly risks of up to 10 types of acute "clinical events," including opportunistic infections and other HIV-related illnesses. For this analysis, reflecting available IeDEA data, we modeled 3 mutually exclusive categories of clinical events: WHO Stage 3 events (WHO 3, excluding pulmonary and lymph node tuberculosis (TB)), WHO Stage 4 events (WHO 4, excluding extrapulmonary TB), and TB events (at any anatomic site) [28]. The CEPAC-Pediatric model simulates three types of mortality. First, children with no history of acute clinical event face a monthly risk of HIV-related death ("chronic HIV mortality"), stratified by current age and CD4%. Second, children who experience a clinical event face "acute mortality" risks in the first 30 days post-event, stratified by current age. After this 30-day "acute mortality" period, children return to “chronic HIV mortality,” though with increased monthly risks compared to age/CD4%-matched children without a history of clinical events. Third, in addition to HIV-related mortality, the model includes a monthly risk of "non-AIDS death," derived from UNAIDS age- and sex-adjusted, country-specific mortality rates that exclude the impact of HIV [39]. For each simulated infant, the model tracks clinical events, changes in CD4%, and the amount of time spent in each health state. After an individual simulated patient has died, the next infant enters the model. Large cohorts (often 1 million-10 million patients) are simulated in order to generate stable model outcomes. Once the entire cohort has been simulated, summary statistics are tallied, including number and type of clinical events and the proportion alive each month. Additional information about CEPAC-Pediatric model structure, data sources, and procedures for initiating new collaborative projects are available at web2.research.partners.org/cepac/model.html. IeDEA: International Epidemiologic Databases for the Evaluation of AIDS; WHO: World Health Organization; TB: tuberculosis; WITS: Women and Infants Transmission Study. a. WHO Stage 4, Stage 4, and TB events defined according to WHO classifications for HIV disease staging in children [3]. b. The publicly available WITS dataset includes 193 perinatally HIV-infected children (positive HIV co-culture or PCR by 4-6 weeks of age), with a median of 5.2 months of follow-up prior to initiation of 3-drug ART (Interquartile Range (IQR): 2.1-12.1 months; AZT monotherapy was permitted during the follow-up period) [33]. Of the 193 perinatally HIV-infected children included in the WITS dataset, 180 (93%) had at least one CD4% measurement before ART initiation, 152 (79%) had at least two values, and 121 (63%) had at least three; the first recorded CD4% was observed at a median age of 5.0 days (IQR: 1.0-29.0 days) c. See derivation of ranges for sensitivity analyses in Methods. IeDEA: International Epidemiologic Databases for the Evaluation of AIDS; WHO: World Health Organization; TB: tuberculosis; WITS: Women and Infants Transmission Study. a. WHO Stage 4, Stage 4, and TB events defined according to WHO classifications for HIV disease staging in children [3]. b. The publicly available WITS dataset includes 193 perinatally HIV-infected children (positive HIV co-culture or PCR by 4-6 weeks of age), with a median of 5.2 months of follow-up prior to initiation of 3-drug ART (Interquartile Range (IQR): 2.1-12.1 months; AZT monotherapy was permitted during the follow-up period) [33]. Of the 193 perinatally HIV-infected children included in the WITS dataset, 180 (93%) had at least one CD4% measurement before ART initiation, 152 (79%) had at least two values, and 121 (63%) had at least three; the first recorded CD4% was observed at a median age of 5.0 days (IQR: 1.0-29.0 days) c. See derivation of ranges for sensitivity analyses in Methods. d. UNAIDS HIV-deleted mortality rates from these eight countries were weighted by the proportion of children from each country included in the UNAIDS pooled analysis used as a calibration target [35,36]. IeDEA is an international consortium of AIDS care and treatment centers [26,27,40]. In previous work, we estimated incidence rates of first clinical event (WHO3, WHO4 and TB), acute mortality (<30 days after clinical event), and chronic HIV mortality among untreated, HIV-infected children at seven clinical sites in the IeDEA East Africa region [28]. Additional details about the IeDEA East Africa sites, as well as methods for derivation of model input parameters, have previously been described [28,41]. In the IeDEA East African cohort, all children enrolled in care prior to 12 months of age (median: 5 months); 52% were female [28]. We translated observed IeDEA event rates into monthly transition probabilities (risks), stratified by age and CD4% (Table 1). In children <6 months old, clinical event risks ranged from 5.2-7.8%/month for WHO3, 1.6-3.5%/month for WHO4, and 0.5-1.1%/month for TB. For children ≥6 months of age, clinical event risks ranged from 3.3-11.6%/month for WHO3, 1.4-6.4%/month for WHO4, and 0.8-3.8%/month for TB (Table 1). Modeled risks of subsequent clinical events were assumed to be equal to risks of first events, within each age and CD4% stratum. For children with no history of clinical events, monthly risks of chronic HIV mortality ranged by CD4% from 0.3-0.4%. For children with a clinical event, the 30-day risk of acute mortality following a WHO3 or WHO4 event was 3.4%, and the risk following TB events was 2.8%. After the 30-day period post-event, monthly risks of "chronic HIV mortality" ranged by CD4% from 0.4-2.4%. Non-AIDS death risks (reflecting age- and sex-adjusted mortality rates) were held at zero for internal validation analyses, since all observed deaths in the IeDEA cohort were coded as HIV-related and thus considered either acute or chronic HIV-related mortality. For calibration analyses, "non-AIDS" mortality rates were from UNAIDS HIV-deleted life tables for the eight sub-Saharan countries which were included in the study (Table 2) [35,36,39]. Because IeDEA lacked adequate longitudinal CD4 data, we derived CD4% at birth and rate of monthly CD4% decline from the US-based WITS, a longitudinal cohort study (1990-2006) of HIV-infected women and their infants during pregnancy and the post-partum period [32–34]. Using a mixed effects model for the primary analysis, we estimated a mean CD4% at birth of 42.0% (standard deviation, 9.4%), and a monthly CD4% decline of 1.4%/month prior to ART initiation [42]. In a secondary analysis in which CD4% was permitted to decline by different rates in months 0-2 and 3+ of life, we estimated a mean CD4% of 50.0% at birth, monthly decline of 6.4%/month for months 0-2, and 0.3%/month in months 3+. Due to high variability around the point estimate for this latter variable, likely due to small numbers of CD4% data in older infants, we used these results to inform the ranges of CD4% parameters for sensitivity analyses, rather than for the primary analysis. For internal validation analyses, we simulated a population of HIV-infected infants from birth (assuming intrauterine or intrapartum infection), with clinical characteristics of patients in the IeDEA cohort. To most closely match the observed IeDEA data, we evaluated model-generated results for children from 5-16 months of age, reflecting a median age at enrollment in the IeDEA cohort of 5 months and a median of 11 months follow-up [28]. We compared model-generated survival curves from 5 to 16 months after birth to Kaplan-Meier survival curves directly from the IeDEA East African regional data. We first assessed model results using base-case parameter estimates. We then performed two-way sensitivity analyses in which we simultaneously varied the two parameters from WITS (CD4% at birth and monthly CD4% decline). First, CD4% at birth was varied in 1.0% increments from 42% (the result in the primary WITS analysis) to 50% (the result from the secondary WITS analysis). This range includes the value of 47%, which was the mean percentage recorded in the first 1-2 days of life in a study in Durban, South Africa [43,44]. Second, the monthly rate of CD4% decline was varied from 0.3% (the lowest value from the secondary WITS sensitivity analysis) to 8.0% (an average of published values in the first three months of life [43–45]). To reflect observations that CD4% may decline more rapidly in the first few months of life [43,44], we permitted CD4% to decline at different rates for "younger" and "older" infants. We defined "younger" and "older" age groups using threshold values of 3, 6, or 12 months of age, and examined all combinations of CD4% at birth and monthly CD4% decline in which CD4% decline was faster in “younger” compared to “older” children. For each parameter set, we compared model-based survival curves to the empiric IeDEA survival curves at each month of the simulation using root-mean-square error (RMSE) [30]. RMSE was calculated as the square root of the average of the squared difference between observed and projected survival proportions at each month over the course of the simulation (5-16 months). We defined the best-fitting survival curves as those with a RMSE <0.01. This method was chosen because it is intuitive, computationally feasible with complex models, and appropriate for data drawn primarily from a single source [23–25,30]. In addition to examining survival results, we also compared the model-generated rates of clinical events to the observed rates in the IeDEA cohort. Because model-based analyses do not rely on a single convention for comparing model results to data [24,25], we defined a good-fitting result as one where model-projected incidence rates were within 10-15% (relative) of observed data, based on previous work [5]. To reflect as closely as possible the IeDEA clinical cohort, simulated infants entered the model at birth, with the initial CD4% distribution and rates of monthly CD4% decline identified in the best-fitting parameter set in the internal validation survival analyses described above. Model-based incidence rates for first clinical events between 5 and 16 months of age were projected for infants. Number of events and time at risk are not stratified by CD4% in the current model output, because they were not anticipated for use in future policy analyses. To directly compare model output with IeDEA data, we re-analyzed IeDEA event rates for all children (combining all CD4% strata) at ages <6 and ≥6 months of age. Following internal validation of the CEPAC-Pediatric model, we compared model-generated results to survival data reported in a pooled UNAIDS analysis of perinatally HIV-infected children in sub-Saharan Africa [35–38]. In this UNAIDS analysis, data were from 12 PMTCT studies in Burkina Faso, Côte d'Ivoire, Kenya, South Africa, Tanzania, Uganda, Zimbabwe and Botswana, reflecting >1,300 perinatally infected infants (defined by a positive PCR test before 6 weeks of age). Among untreated infants, survival was estimated by Weibull survival analysis to be 64% at 6 months, 49% at 12 months, 35% at 24 months, 25% at 36 months, 17% at 48 months, and 12% at 60 months [35–37]. To compare model-generated results to these data, we used the CEPAC-Pediatric model to simulate a cohort of infants with in utero or intrapartum HIV infection from birth through 60 months of age. We anticipated that there would be substantial differences in the CD4% at birth, rate of CD4% decline, and mortality risks between children in the UNAIDS and IeDEA East Africa cohorts. To calibrate the model against UNAIDS data, we varied all CD4% and HIV-related mortality parameters, individually and in combination, applying multipliers of 0.2 to 20 to the mortality risks observed in the IeDEA cohort (Tables 2 and ​and3).3). CD4% decline was modeled to be more rapid in the first 3 months of life, based on results of the internal validation analysis. Monthly risks for clinical events (WHO3, WHO4, and TB) observed in the IeDEA cohort were similar to or greater than those reported in the literature [45–50], and were therefore not varied in calibration analyses. Non-AIDS mortality rates were also held constant, using a weighted average of UNAIDS HIV-deleted mortality data for the eight countries in the UNAIDS analysis (Table 2) [35,36,39]. IeDEA: International Epidemiologic Database to Evaluate AIDS, East African region. m: month. a. Values for monthly CD4% decline reflect more rapid decline in the first three months of life than after age three months, based on published literature [43–45], and the results of internal validation analyses. b. Acute mortality risk: risk of death within 30 days of a clinical event (WHO Stage 3, WHO Stage 4, or tuberculosis; see Methods). c. Chronic HIV mortality: monthly risk of death for patients with no history of a clinical event, or for patients >30 days following a clinical event (see Methods). In all evaluated parameter sets, multipliers for chronic HIV mortality were limited to ranges in which multipliers applied at younger ages were ≥ multipliers at older ages. Risks were therefore permitted to remain constant or decrease (but not increase) with age. This leads to a total of 294,660 parameter combinations of chronic HIV mortality multipliers, and 141.4 million total parameter sets examined (see Methods). Model-generated results were compared to empiric data in a step-wise fashion based on six key time points after birth (6, 12, 24, 36, 48, and 60 months). We first identified all combinations of CD4% values and mortality risk multipliers (Table 3) that led to model-generated mortality within 1% of the UNAIDS mortality estimate at 6 months of age (63-65%). For each of those parameter sets, multipliers were next applied to chronic HIV mortality risks for ages 7-12 months. All parameter sets producing model-generated mortality risks within 1% (absolute) of the target 12-month risk (48-50%) were retained in the next step. Chronic HIV mortality risk multipliers were then applied to ages 13-24 months; parameter sets leading to results within 1% of the 24-month target (34-36%) were retained. This process was repeated for time points of 36, 48, and 60 months. In all evaluated parameter sets, multipliers for chronic HIV mortality were limited to ranges in which multipliers applied at younger ages were greater than or equal to multipliers at older ages. Risks were therefore permitted to remain constant or decrease (but not increase) with age. Finally, all parameter sets leading to model results within these ranges were compared again to the UNAIDS mortality rates to identify all parameters sets that resulted in a RMSE <0.01%.