Can an educational podcast improve the ability of parents of primary school children to assess the reliability of claims made about the benefits and harms of treatments: Study protocol for a randomised controlled trial

Background: Claims made about the effects of treatments are very common in the media and in the population more generally. The ability of individuals to understand and assess such claims can affect their decisions and health outcomes. Many people in both low- and high-income countries have inadequate aptitude to assess information about the effects of treatments. As part of the Informed Healthcare Choices project, we have prepared a series of podcast episodes to help improve people’s ability to assess claims made about treatment effects. We will evaluate the effect of the Informed Healthcare Choices podcast on people’s ability to assess claims made about the benefits and harms of treatments. Our study population will be parents of primary school children in schools with limited educational and financial resources in Uganda. Methods: This will be a two-arm, parallel-group, individual-randomised trial. We will randomly allocate consenting participants who meet the inclusion criteria for the trial to either listen to nine episodes of the Informed Healthcare Choices podcast (intervention) or to listen to nine typical public service announcements about health issues (control). Each podcast includes a story about a treatment claim, a message about one key concept that we believe is important for people to be able to understand to assess treatment claims, an explanation of how that concept applies to the claim, and a second example illustrating the concept. We designed the Claim Evaluation Tools to measure people’s ability to apply key concepts related to assessing claims made about the effects of treatments and making informed health care choices. The Claim Evaluation Tools that we will use include multiple-choice questions addressing each of the nine concepts covered by the podcast. Using the Claim Evaluation Tools, we will measure two primary outcomes: (1) the proportion that ‘pass’, based on an absolute standard and (2) the average score. Discussion: As far as we are aware this is the first randomised trial to assess the use of mass media to promote understanding of the key concepts needed to judge claims made about the effects of treatments. Trial registration: Pan African Clinical Trials Registry, PACTR201606001676150. Registered on 12 June 2016. http://www.pactr.org/ATMWeb/appmanager/atm/atmregistry?dar=true&tNo=PACTR201606001676150. This will be a two-arm, parallel-group, individual-randomised trial evaluating the impacts of a podcast series designed to teach concepts of evidence-informed decision-making to parents of primary school children in Uganda (Fig. 1). Study flow chart The study will be conducted in central Uganda within the communities in which participants live and work. Uganda is a landlocked country in East-Central Africa. It is a multiethnic country with slightly over 70% of the population aged below 25 years. The country has a life expectancy at birth of 54.9 years, total fertility rate of 5.9 children, literacy rate of 78.4%, maternal mortality ratio of 343 per 100,000 live births, and infant mortality rate of 59 per 1000 live births. It has 0.1 physicians per 1000 people [33]. Our study population will be parents of primary school children in schools with limited educational and financial resources. We will draw a sample of parents of children from schools participating in a cluster-randomised trial of IHC primary school resources, designed to help primary school children understand and assess claims made about treatment effects [34]. All public and private primary schools in the central region of Uganda will be eligible to participate in the IHC primary school resources trial. The only exclusions will be international schools, special needs children’s schools for the deaf and blind, and schools or classes that participated in the user-testing and piloting of the teaching resources. We intend to include at least 120 schools in that trial. We will randomly select at least 30 schools among the 120 participating in the related trial evaluating the effectiveness of IHC materials designed for primary school children [34]. This number is informed by findings from our earlier engagements with parents, in which only about 10 to 20 parents attended meetings that are not on the usual school programme. About half of the schools selected will be from the intervention arm and the other half from the control arm of the schools participating in the trial evaluating primary school resources. In each of the selected schools we will create a list of parents of P.5 children and invite them to a meeting at the schools. At these meetings we will provide parents with information about the podcast trial and seek their consent to participate. Because parents who attend meetings might be different than those who do not, we will also make an effort to reach parents who will not have attended the meetings through phone calls, where possible. Parents who accept to participate will be asked to sign a Consent Form (Additional files 1 and 2) before randomisation to begin their participation in the study. We will provide information about the trial to participants in a brochure designed as part of a package inviting parents to participate. This information will also be available on the participants’ Informed Consent Forms and will be written in the two most commonly spoken languages (English and Luganda). We will stratify our sample proportionately by the highest level of education attained (primary school and below, secondary school, or tertiary education) and whether their children are in a school that is in the intervention or control arm of the IHC primary school resources trial. We will use computer-generated blocked randomisation. A statistician or other senior researcher at the clinical trials unit at the College of Health Sciences, Makerere University who is not a member of the IHC research team will generate the allocation sequences for the two comparison groups. A web-based random number generator at www.sealedenvelope.com will be used to obtain random permuted blocks with varying block sizes of 4 and 6, and equal allocation ratios, both within blocks and within strata, as shown in Table 1 below. Excerpt from a sample randomisation lista aList generated from www.sealedenvelope.com As shown in Table 1 above, the list will have study codes that are unique for each participant (column 7), against which we will attach an ordered array of numbers (column 6) spanning the full breadth of our sample size within each stratum. Each study code will be printed on a separate opaque envelope. The allocation group corresponding to each study code will be printed on a separate slip, inserted and sealed in the corresponding opaque envelope. As shown in the list above, the first participant to be recruited among those with primary or no education will have the study code YL9 on their envelope, inside of which will be a small note with their allocation group (group A). To reduce the risk of bias during allocation a separate list containing the study participant number, and corresponding study code and number within each stratum, will be generated for each of the three education-level strata (up to and including primary, at least secondary and tertiary), as shown in Table 2 below. This list – which will not identify the study arm – will be given by the statistician to a research assistant who will be in charge of group allocation. Sample participant recruitment lists for each stratum Upon completion of procedures for confirming consent of a willing participant the research assistant responsible for recruiting the participant will call the research assistant in charge of group allocation. The research assistant responsible for allocation will then open the next available envelope in the stratum, corresponding to the participant’s education level and study arm of the child’s school, to determine the study group to which the parent will be allocated. Participants in the intervention group will listen to a series of podcast episodes designed to teach people to assess claims made about treatment effects (the IHC podcast). Each episode includes a short story with an example of a treatment claim, a simple explanation of a concept used to assess that claim, another example of a claim illustrating the same concept, and its corresponding explanation. In each story there is a question about the trustworthiness of a claim, which is resolved by applying the relevant key concept. All episodes have a conclusion with a take-home message emphasising the concept of evidence-informed, health decision-making. The examples used in the podcast are for claims made about treatments for health conditions, which are of interest to the target audience, such as malaria, diarrhoea, and HIV/AIDS. We have also included claims made about some common practices, such as contraception, which are of interest to our audience. The topics and claims were identified from scanning recent mass media reports and interviewing parents [32]. There are eight main episodes in the series covering the nine core concepts (Table 3). Each episode lasts about 5 min. One of the episodes (episode 1) covers two closely related key concepts (1.1 and 5.1 in Table 3). Two additional episodes introduce the podcast and summarise the key messages from the first eight episodes, respectively. The podcast also has a song (the IHC theme song), sections of which play at the beginning, in the background and at the end of the episodes. The lyrics of the song were designed to reinforce the messages of the podcast and focus on being careful about claims, and the important questions to ask when we hear claims made about treatment effects. The final structure, content, presentation of the content in each episode, and the series as a whole was informed by an iterative user-centred process of development and user-testing described in another paper [32]. This process involved consultation with various stakeholders, including parents in our target audience, on the appropriate content to be included, and the presentation of this content in each episode, and in the podcast as a whole [32]. The numbering in the first column in the table below relates to numbering of the key concepts in their respective domains, as presented by Austvoll-Dahlgren et al. [31]. The nine conceptsa included in the IHC podcast aFrom Austvoll-Dahlgren et al. [31] A summary of the concepts contained in each episode of the podcast is shown in Table 4 below. The conceptsa contained in each episode of the podcast aSee Table 3 for the full description of the concepts and their implications Participants in the control group will receive typical public service announcements about conditions of interest to our target audience, including malaria, HIV/AIDS, and diarrhoea, that have been included in the podcast. These audio messages are already being aired or have in the recent past been aired on radio. They are freely available and will be sourced from radio stations and media houses involved in disseminating information from the Ministry of Health. The interventions will be delivered over a period of 7 to 10 weeks to participants in the communities where they live or work. At least 20 research assistants will help in the field recruitment, administration of the intervention, follow-up of participants in the community, and administration of the outcome assessment tool. Each research assistant will be allocated about 25 participants to follow up through the duration of the study. The research assistants will visit all their allocated participants every week, delivering the interventions on portable media players to an average of four participants per day. In this study setting this approach is currently the most practical and most reliable way of delivering the interventions for this trial. One-time brief interventions have not been shown to improve knowledge of evidence-based medicine principles [35, 36] among health workers. So, as part of our intervention, at every visit we will play a recap of the previous episodes before playing the new episodes. Each participant will receive at least one supervised listening session of every episode, delivered by the research assistants over 7 to 10 weeks. In addition to the messages delivered by the research assistants, we will provide participants with the complete podcast on MP3 players, so that they can replay the episodes at their convenience. For the intervention group we will start with simpler concepts and gradually introduce more difficult concepts as the podcast series progresses, as summarised in Table 5 below. For the control group we will introduce episodes with health conditions that are similar to those being delivered in the intervention group at a defined period of time. The timing of study procedures is summarised in Fig. ​Fig.22 below. Sequence of delivery of the episodes of the podcast aStudy evaluation tool (Additional file 3). Schedule of enrolment, interventions, and assessments We will use a modified version of the Claim Evaluation Tools as the outcome measure [37]. The Claim Evaluation Tools consist of multiple-choice questions that assess an individual’s ability to apply 32 concepts that people need to be able to understand and apply to assess the validity of treatment claims and to make informed health care choices. In addition, it includes questions that assess intended behaviours, self-efficacy, attitudes, and satisfaction. Development of the Claim Evaluation Tools has been based on extensive qualitative and quantitative feedback from methodological experts, health professionals, teachers, and members of the public. We have conducted psychometric testing and Rasch analysis with a diverse sample of 1000 people, including P.5 children exposed to pilot versions of the IHC primary school resources, children who were not exposed, adults with little or no exposure to the concepts, and adults who are familiar with the concepts. The evaluation tool was found to have high reliability (Cronbach’s alpha 0.81), and to be unidimensional (there was no evidence of subdimensions measuring different traits). Furthermore, there was weak or no dependence between items (no items were redundant). Results from the pilot revealed that parents in the intervention group performed slightly better compared with those in the control group. Many of the participants in the pilot preferred Luganda to English and had difficulty reading the questions in English. After removing some questions, we modified the remaining questions that did not perform well according to the Rasch analysis, and revised and simplified the text where needed. The items were also translated into Luganda and adapted for audio administration. In the second psychometric testing, the Claim Evaluation Tools were administrated to a similar sample as described in the first psychometric test, but approximately half of the sample received the items as a written questionnaire in English and the other half received the Luganda audio versions. The results of this test suggested that the items administered in English performed very well according to the Rasch model, and with high reliability. Furthermore, the items were also less difficult than what was found in the first psychometric testing before the revisions. The results also suggested that the Luganda versions of items had evidence of under-discrimination and differential item functioning in seven out of 29 items. These items were revised to improve fit to the Rasch model. Based on these two psychometric tests, a selection of 18 multiple-choice items addressing the nine concepts that the IHC podcast covers will be used (Additional file 3). Each key concept is evaluated by two items. We chose items with high reliability (fit to the Rasch model) and those with an appropriate difficulty level. Parents in both arms of the trial will complete the questionnaires after they have listened to the podcast or public service announcements. Research assistants will deliver the questionnaires, ensuring that the parents have adequate time to answer the questions and ensure that the questionnaires have been completed fully. We will use an absolute (criterion referenced) standard to set a passing score for the version of the Claim Evaluation Tools that we will use, i.e. based on how much the parents know and are able to apply. Parents will be counted as ‘passing’ or ‘failing’ depending on whether they meet a specified criterion. We used a combination of Nedelsky’s [38] and Angoff’s [39] methods to determine the criterion, which is a cut-off for a passing score, as described in Additional file 4. In addition, we determined a second cut-off for a score that indicates mastery of the nine concepts, using the same methods, as described in Additional file 4. The criterion for passing is a minimum of 11 out of 18 questions answered correctly. The criterion for mastery is a minimum of 15 out of 18 questions answered correctly. The intervention and control groups will complete the Claim Evaluation Tools after listening to all of the IHC episodes or public service announcements, respectively. We will evaluate the effects of the IHC podcast series again after 1 year, using the same outcome measures. We will try to locate participants who will have moved from their original residence or work place and document those who are lost to follow-up, as well as the reasons, where possible. Because of the nature of the intervention, the research assistants who will deliver the intervention, the principle investigator supervising them, and the participants in the study will not be blinded. The statistician who will analyse the study results will not be told which group is the intervention group until after the analyses are completed. To ensure uniform performance in delivery of the intervention and assessment of outcomes, all study staff will receive joint training before the start of the trial and will receive refresher training in the course of the trial. We will have standard operating procedures to guide interactions with participants. We will maintain a database of participants’ contact details and maps to their homes to enable follow-up. The research assistants will keep a study logbook and use a progress tracking checklist (Tables 6 and ​and7)7) to keep track of their progress and note any problems with follow-up of participants. We will discuss these problems and ways of addressing them and preventing loss to follow-up at weekly meetings. Study log for monitoring progress of the intervention for each study participant Sample study procedures checklist for all participants We will use paper questionnaires for the outcome assessment and we will collect baseline sociodemographic characteristics using a paper form. For participants who prefer listening to the Luganda version of the podcast but are unable to read Luganda we will use an interviewer-administered, standardised Luganda audio version of the evaluation tool. A research assistant will enter their responses to the audio evaluation tool on a paper questionnaire. To reduce any wrong entries and missing values the research assistants will check the questionnaires for completeness and for unclear responses immediately after completion. We will use Epidata version 3 software [40] to enter all data from the paper questionnaires and convert them into an electronic dataset. To reduce errors during data entry we will double-enter all the data. All data entrants will be trained on the meaning of variables and how they should be entered into the database. Questions that remain unanswered will be scored as ‘wrong’. The lead investigator will double-check the entries. All copies of Consent Forms will be kept separate from the completed questionnaires and baseline characteristics of the participants. Both signed Informed Consent Forms and completed questionnaires will be kept under lock and key. All electronic data will be password-protected and kept on a separate hard disc, which will be backed up every week to prevent data loss. The hard discs will also be kept under lock and key and access will be restricted to study investigators. Trial progress reports will be compiled by the study team and submitted to the regulatory bodies (Uganda National Council for Science and Technology and School of Medicine Research and Ethics Committee (SOMREC), Makerere University College of Health Sciences), in accordance with their respective guidelines. Final project reports (including trial reports) will be compiled by the study staff and submitted to the relevant bodies (Research Ethics Committees (RECs) and funders), as per their requirements. We will present participants’ age, sex, and highest level of education attained as frequencies and proportions. For the primary and secondary outcomes, we will use fixed-effect models with the stratification variable (education) modelled as a fixed effect, using logistic regression for dichotomous outcomes, linear regression for continuous outcomes and Poisson regression for count outcomes. For the questions that assess applied knowledge or understanding, missing values will be counted as ‘wrong’ answers. We will report the proportion (for dichotomous outcomes), mean and standard deviation (for continuous outcomes), or median and interquartile range (for count outcomes) in each group, the estimated difference, the estimated confidence interval for the difference, and the P value for the difference between groups from the statistical model for each of the primary and secondary outcomes. For attitudes and intended behaviours (Additional file 3), we will dichotomise each outcome by combining, for example, ‘very unlikely’ with ‘unlikely’ and as ‘likely’ with ‘very likely’; and we will report the proportion in each of the four categories. Education might affect people’s ability to understand and apply the concepts and to answer questions in the Claim Evaluation Tools. We will explore whether there are differences in the effects of the intervention for participants with primary school education versus secondary school education versus tertiary education. We will conduct tests for interaction for the primary outcomes, and we will use published guidelines to interpret the results of these subgroup analyses [41], using the models described above. The children of all participants will be participating in a parallel trial evaluating the effects of IHC teaching resources, designed to teach P.5 school children concepts that they need to understand and apply to assess treatment claims [34], but only half of them will be receiving the IHC teaching resources. By testing for interaction between IHC teaching resources and the podcast in the statistical models described above we will evaluate whether the combination of the podcast and the IHC teaching resources improves outcomes among parents, compared to the podcast alone. We used the method described by Donner [42] to calculate the sample size, based on calculation of odds ratios. The smallest difference between the intervention and control groups that we want to be able to detect in the proportion of respondents with a passing score is 10 percentage points. Assuming that 10% of the control group will achieve a passing score (a conservative estimate, based on data from the pilot [32], power of 0.90, a two-sided P value of 0.05, we estimate that 397 participants will be needed to be able to detect an improvement of 10% in the intervention group. Studies of the effects of drug fact boxes and a primer to help people understand risk suggest that this is likely to be an adequate sample size [25, 26]. Allowing for a 20% loss to follow-up gives us a sample size of 497 participants. More specific information about the trial and this protocol is summarised in the SPIRIT checklist (Additional file 5). The National Council of Science and Technology in Uganda has given this study a very low rating for risk to participants. Nonetheless, we will monitor unexpected adverse events and problems that might arise. We have engaged health researchers, journalists, other media and communications practitioners, teachers, and the general public in developing, revising, piloting, and user-testing the podcast [32]. Authorship of publications arising from this study will be according to contribution. Publications and the resources will be open access, allowing free noncommercial use, distribution, reproduction and further development of the work provided the source is properly cited. We will acknowledge all staff participating in the trial. The results of this trial will be publicised through a variety of channels. All of the resources will be made available on the IHC project and Testing Treatments interactive websites, and through the Critical Appraisal Resource Library (CARL). They will be disseminated internationally by the IHC project team and through members of our international advisory group, the Cochrane Collaboration (www.cochrane.org), the Evidence-informed Policy Network (www.evipnet.org), the WHO, the Campbell Collaboration (www.campbellcollaboration.org), and other relevant networks and organisations.