Determining the agreement between an automated respiratory rate counter and a reference standard for detecting symptoms of pneumonia in children: Protocol for a cross-sectional study in Ethiopia

Background: Acute respiratory infections (ARIs), primarily pneumonia, are the leading infectious cause of under-5 mortality worldwide. Manually counting respiratory rate (RR) for 60 seconds using an ARI timer is commonly practiced by community health workers to detect fast breathing, an important sign of pneumonia. However, correctly counting breaths manually and classifying the RR is challenging, often leading to inappropriate treatment. A potential solution is to introduce RR counters, which count and classify RR automatically. Objective: This study aims to determine how the RR count of an Automated Respiratory Infection Diagnostic Aid (ARIDA) agrees with the count of an expert panel of pediatricians counting RR by reviewing a video of the child’s chest for 60 seconds (reference standard), for children aged younger than 5 years with cough and/or difficult breathing. Methods: A cross-sectional study aiming to enroll 290 children aged 0 to 59 months presenting to pediatric in- and outpatient departments at a teaching hospital in Addis Ababa, Ethiopia, was conducted. Enrollment occurred between April and May 2017. Once enrolled, children participated in at least one of three types of RR evaluations: (1) agreement—measure the RR count of an ARIDA in comparison with the reference standard, (2) consistency—measure the agreement between two ARIDA devices strapped to one child, and (3) RR fluctuation—measure RR count variability over time after ARIDA attachment as measured by a manual count. The agreement and consistency of expert clinicians (ECs) counting RR for the same child with the Mark 2 ARI timer for 60 seconds was also measured in comparison with the reference standard. Results: Primary outcomes were (1) mean difference between the ARIDA and reference standard RR count (agreement) and (2) mean difference between RR counts obtained by two ARIDA devices started simultaneously (consistency). Conclusions: Study strengths included the design allowing for comparison between both ARIDA and the EC with the reference standard RR count. A limitation is that exactly the same set of breaths were not compared between ARIDA and the reference standard since ARIDA can take longer than 60 seconds to count RR. Also, manual RR counting, even when aided by a video of the child’s chest movements, is subject to human error and can result in low interrater reliability. Further work is needed to reach global consensus on the most appropriate reference standard and an acceptable level of agreement to provide ministries of health with evidence to make an informed decision on whether to scale up new automated RR counters. This study aims to understand whether an ARIDA RR count agrees with an expert panel of pediatricians counting RR by reviewing a video of the child’s chest for 60 seconds (reference standard) for children aged younger than 5 years with cough and/or difficult breathing. The primary objective of this study is to determine the performance of an ARIDA, as defined by agreement and consistency, in children aged younger than 5 years with cough and/or difficulty breathing. The secondary objective is to determine the performance of expert clinicians (ECs) counting RR, as defined by agreement and consistency, in children aged younger than 5 years with cough and/or difficulty breathing. The third objective is to measure RR fluctuation over time after ARIDA device attachment in normal breathing children aged 2 to 59 months. The study is a cross-sectional study comprising three types of RR evaluations: agreement, consistency, and RR fluctuation over time. The study was conducted in the pediatric in- and outpatient departments at Saint Paul’s Hospital and Millennium Medical College in Addis Ababa, Ethiopia. This hospital was selected based on the high incidence of pneumonia in outpatient and inpatient departments, interest and willingness of hospital managers to host the study, availability of Integrated Management of Neonatal and Child Illness (IMNCI)-trained [14] expert clinicians (ECs), and availability of a suitable study room, reliable electricity supply, and access to treatment including amoxicillin and oxygen. Ethical approval was obtained from the Armauer Hansen Research Institute/ALERT Ethics Review Committee (a biomedical research institute in Ethiopia) on March 7, 2017 (ref. PO02/17) and favorable ethical opinion received by the Liverpool School of Tropical Medicine Ethics Committee. All participants consented to the study by reading and signing the information and consent form. All children attending in- and outpatient departments at Saint Paul’s Hospital and Millennium Medical College in Addis Ababa between April 5 and May 22, 2017, were potential participants in the study and were systematically screened for eligibility. Children aged 0 to <2 months were excluded from the consistency evaluation due to the anticipated difficulty in attaching two devices at once to a small child. Children aged 0 to <2 months and those with fast breathing were excluded from the fluctuation evaluation due to anticipated difficulty in measuring RR in this group for an extended period of time and also to isolate the effect of the ChARM attachment on RR from other causes of RR fluctuation. All other children aged younger than 5 years who were accompanied by a caregiver aged 18 years and older, not too agitated to be assessed by a research nurse, who did not present with general danger signs or IMNCI referral signs or device manufacturer safety exclusion criteria (wearing supportive device at area of chest/belly, skin not intact in chest/belly, born before 37 weeks of gestation [<2 months only]), were not an inpatient being managed by barrier nursing (such as severe burns, child with neutropenia, severe infectious diseases), and were not advised against research procedures by the supervising clinician were eligible to participate in the study. General danger signs for newborns (<28 days) include active convulsions or fits, not feeding well, fever (37.5℃ [99.5℉] or above), low body temperature (35.5℃ [95.9℉] or below), movement only when stimulated, or no movement even when stimulated and for all other age groups include lethargy or unconsciousness, not able to drink/breastfeed, vomiting everything, and active convulsions or fits. IMNCI referral signs for all children include stridor in a calm child, chest indrawing, severe dehydration, severe persistent diarrhea, very severe febrile disease, severe complicated measles, mastoiditis, complicated severe malnutrition, and severe anemia. Written informed consent was obtained from the caregiver before enrollment. Two ECs with extensive experience in assessing and treating children with suspected pneumonia using IMNCI guidelines were selected. They were required to have BSc nursing qualification and an IMNCI certificate. The primary outcome for the agreement evaluation was the mean difference in RR between ARIDA and the reference standard, which summarized the lack of agreement by calculating the average deviation between measures. Similarly for the consistency evaluation, the mean difference in RR between two ARIDA devices was calculate. Table 1 shows all the outcome measures for the study by objective. Primary objective and outcome measures. aARIDA: Acute Respiratory Infection Diagnostic Aid (Philips Children’s Respiration Monitor device). bVEP: video expert panel (reference standard). cRR: respiratory rate. dRMSD: root mean squared difference. eEC: expert clinician. Data were collected using an electronic data collection platform (CommCare, Dimagi) installed onto password-protected 7C Pro tablets (Tecno Mobile) and backed up to a protected cloud server. Four-digit unique identification codes were used to anonymize patient data. All videos were transferred using password-protected external hard drives, and paper consent forms were stored in opaque carriers in locked cabinets. All RR evaluation data were entered by two independent research assistants. The data manager downloaded data daily and entered it into a data checker with in-built validation checks. Source videos showing the ARIDA device with the RR count displayed were used to verify ARIDA counts should the two research assistants disagree. Other inconsistencies were rectified by tracing back to paper data entry forms or querying the counts directly with the research team. The video expert panel (VEP) members and ECs were trained for two days on the WHO IMCI method to count RR including practice for half a day using videos of known RR counts [3]. They were individually evaluated using different RR videos to ensure that they were able to count RR ±2 breaths per minute (bpm) from the known RR in 4 out of 5 training videos [15]. All VEP members and ECs passed the competency assessment before starting video review. Following training, an 8-day pretesting of procedural activities including patient screening, patient flow, and data collection was conducted to ensure the research team was conversant with the data collection procedures, devices, and videography equipment to be used in the study. There was also a pretest of the video panel reference standard evaluations and refresher training on general danger signs, breath-counting, signs of stridor, and chest indrawing with the IMNCI training video. Patients were screened by a research nurse in the in- and outpatient departments of the hospital using a screening checklist to ascertain the child’s eligibility. An RR classification was made during prescreening by the research nurse using the Mark 2 Acute Respiratory Infection (MK2 ARI) timer to determine whether the child had fast or normal breathing. The prescreening assessment was conducted in a separate part of the hospital and not communicated to the ECs to blind them to the RR classification. Children were enrolled prospectively based on eligibility determined by the screening procedure. The research team then decided, based on the state of the child, age, and breathing status, how many elements of the study to conduct on each child—agreement, consistency, and/or RR fluctuation. Table 2 shows the number of participants aimed to be enrolled to each element of the study by age group and breathing status. Enrollment targets for each type of evaluation by age group and breathing status. For the agreement evaluation, the research assistant attached an ARIDA to the child and ensured the child was positioned correctly according to device instructions: with his/her back fully supported, either on the lap of the caregiver or lying down on a bed, and the device in line with the child’s belly button and one of the nipples [16]. Once the child was calmed, usually by the research assistant clicking their fingers, the videographer started recording and the ARIDA and EC RR count started simultaneously. The EC was blinded to the ARIDA RR count by placing ARIDA on the far side of the child’s belly and using a paper cover to shield the screen. The time taken to get an ARIDA count (from when the OK button was pressed to when the device beeped to signal completion of the RR count) was also obtained by a research assistant using a stopwatch. After 60 seconds, if the EC had not obtained an RR count, the EC attempt was recorded as unsuccessful and repeated for both the EC and ARIDA. After 5 minutes or if the ARIDA displayed an error message, the ARIDA attempt was recorded as unsuccessful (with a reason) and repeated for both EC and ARIDA. If the third attempt was still unsuccessful for either device or EC, the evaluation was recorded as a failure. Fifteen different ARIDA devices were used and rotated systematically for all evaluations. The consistency evaluation followed the same procedure as the agreement evaluation with two ARIDA devices attached to a child using a single belt, positioned in line with each nipple and measured RR from the same starting point. Time taken to obtain each ARIDA RR count was recorded by two research assistants using stopwatches. To measure the consistency between ECs, two ECs conducted separate manual RR counts with MK2 ARI timers over an identical 60-second period. For an EC or ARIDA attempt to be successful, both ECs or both ARIDA devices had to get an RR count. For the RR fluctuation evaluation, an EC counted RR with the MK2 ARI timer for 60 seconds. Following this, the ARIDA was attached to the child and the EC did three more RR counts for 60 seconds in the following time periods: 0 to 1 minute, 2 to 3 minutes, 4 to 5 minutes. On completion of the evaluation, the research team debriefed the caregiver and ensured medical management for the child was completed by the relevant hospital staff. The reference standard for the agreement evaluation was a video review by two to four independent VEP members. They were all practicing pediatricians with over 5 years’ experience managing pneumonia in children aged younger than 5 years and who had received refresher training in counting RR as per WHO IMCI guidelines (3). First, two VEP members independently watched a video of the child’s chest movements, edited with the layover of the ChARM start and stop sound and a 60 second timer, and counted the number of breaths observed in a full minute. Beep sounds were added by the videographer in sync with the original sounds made by the ChARM device at the start (when the start button on ChARM is pushed) and at the end (when the ChARM displays the result). The sound recorded at the time of recording was also muted to allow the VEP to focus on the sound of the start and stop beeps. If the first two VEP members agreed (≤±2 bpm), a mean RR count was used as the reference standard. If they disagreed (>±2 bpm), a third VEP member reviewed the video and if two out of three counts agreed (≤±2 bpm), the mean of the two closest RR counts was used. If all three VEP members disagreed (>±2 bpm), the video was sent for review to a fourth VEP member. If the fourth VEP member’s count agreed (≤±2 bpm) with any of the first three VEP members’ counts, the mean of the two closest counts was used. If all four panel members disagreed (>±2 bpm), the data from this evaluation were excluded from the agreement analysis. The primary outcome on which sample size was based was the agreement between the ARIDA and VEP RR counts. As per Bland-Altman [17], we conducted a precision-based sample size calculation based on the confidence interval for the 95% limits of agreement. The formula estimates the required number of children per age group (n) based on the desired width of the confidence interval. Using normal approximation and allowing a confidence interval of ±0.5 standard deviations of the difference between the two devices, a sample size of 46 children per age group was required for the agreement and consistency evaluations, adjusted to 52 per group for failure to get a reference standard count. For the RR fluctuation evaluation, a sample size of 30 children was used. Data analysis for all three RR evaluations was conducted in Stata 13 (StataCorp LLC) and Excel (Microsoft Corp). First, the number of children screened, eligible, consented, and enrolled in each type of evaluation was described. Baseline characteristics (age and sex) by screening breathing status (normal/fast) for those enrolled were described. All full-length source videos were reviewed for quality assurance purposes, and descriptive information on the video quality was recorded, including those where all four VEP members disagreed on the RR count. For the ARIDA and EC agreement and consistency evaluations, mean difference, root mean square difference, absolute mean difference, proportion of RR counts ±2 bpm from the reference standard, and positive and negative percentage agreement with 95% confidence intervals were calculated in Stata 13 by age group, and Bland-Altman plots with limits of agreement and 95% confidence intervals by age group and breathing status were created. Percentage of unsuccessful attempts and failures (defined by three unsuccessful attempts) and mean time to get an ARIDA RR count were calculated. A per-protocol analysis was used whereby children were excluded from the analysis if an RR could not be obtained simultaneously by the ARIDA and by the EC, with a VEP RR reading where at least two of the panel members were within ±2 bpm of each other. For the RR fluctuation evaluation, mean difference in the RR count between baseline and 1 minute, 1 and 3 minutes, and 3 and 5 minutes were calculated. The proportion of children with fast or normal RR classification at baseline and the change between RR classifications over time were analyzed. Malaria Consortium and UNICEF (Supply Division and Ethiopia Country Office) conducted quality assurance visits every 2 weeks to the research site during data collection. All data collected from the screening and RR evaluations were checked and verified by the data manager daily. A sample of three videos was sent weekly to an independent study advisor for RR evaluation using WHO IMCI guidelines [3] and to Malaria Consortium HQ for quality assurance. The project had an 11-person Advisory Committee made up of experts on maternal and child health who provided technical oversight and reviewed the study protocol.