Study protocol for a pilot prospective, observational study investigating the condition suggestion and urgency advice accuracy of a symptom assessment app in sub-Saharan Africa: the AFYA- € Health’ Study

Introduction Due to a global shortage of healthcare workers, there is a lack of basic healthcare for 4 billion people worldwide, particularly affecting low-income and middle-income countries. The utilisation of AI-based healthcare tools such as symptom assessment applications (SAAs) has the potential to reduce the burden on healthcare systems. The purpose of the AFYA Study (AI-based Assessment oF health sYmptoms in TAnzania) is to evaluate the accuracy of the condition suggestions and urgency advice provided by a user on a Swahili language Ada SAA. Methods and analysis This study is designed as an observational prospective clinical study. The setting is a waiting room of a Tanzanian district hospital. It will include patients entering the outpatient clinic with various conditions and age groups, including children and adolescents. Patients will be asked to use the SAA before proceeding to usual care. After usual care, they will have a consultation with a study-provided physician. Patients and healthcare practitioners will be blinded to the SAA’s results. An expert panel will compare the Ada SAA’s condition suggestions and urgency advice to usual care and study provided differential diagnoses and triage. The primary outcome measures are the accuracy and comprehensiveness of the Ada SAA evaluated against the gold standard differential diagnoses. Ethics and dissemination Ethical approval was received by the ethics committee (EC) of Muhimbili University of Health and Allied Sciences with an approval number MUHAS-REC-09-2019-044 and the National Institute for Medical Research, NIMR/HQ/R.8c/Vol. I/922. All amendments to the protocol are reported and adapted on the basis of the requirements of the EC. The results from this study will be submitted to peer-reviewed journals, local and international stakeholders, and will be communicated in editorials/articles by Ada Health. Trial registration number NCT04958577. This AFYA Trial is a prospective, observational study conducted at the waiting room of the Mbagala Rangi Tatu Hospital, Dar es Salaam, Tanzania. The trial protocol was developed in accordance with the current Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT).19The SPIRIT checklist is available in the supplemental files as online supplemental file 1. bmjopen-2021-055915supp001.pdf The accuracy and comprehensiveness of the SAA used in this study have been validated in several studies, and it has a higher accuracy (73%) in comparison to other apps (38%) in condition suggestion.8 To date, there have been no studies on SAAs thus far in LMICs, creating a need for data to determine its usefulness in these settings. The SAA evaluated in this study is a European Union (EU) regulatory-approved medical device Conformitè Europëenne (CE)-marked with study-related modifications, which was developed by Ada Health (Berlin, Germany).20 It offers artificial intelligence-powered symptom assessment technology to users. After users input their symptoms onto the platform, the tool uses AI to suggest a list of conditions that the user might have, and the probability associated with each suggested condition. In 2020, the SAA was shown to have market leading accuracy of condition-suggestion and urgency-advice accuracy at the same level as UK general medical practitioners.8 The SAA’s reasoning engine infers disease probability estimations based on a representation of medical knowledge, which is used to define a Bayesian network, on which approximate inference is carried out, and following which, information-theoretical methods are used to decide which questions to ask to the user.6 The SAA’s knowledge base was built and reviewed by a large team of medical doctors with clinical experience in a curated process of knowledge integration from medical literature. It consists of disease models of all common conditions and several hundred rare diseases, including their corresponding symptoms and clinical findings. The disease models and their related symptoms are added to the knowledge base and modelled according to fundamental knowledge gained by their medical education and clinical experience, as well as evidence from medical textbooks and peer-reviewed medical literature. The SAA’s medical knowledge is expanded continuously following this standardised process and has been specifically optimised for a sub-Saharan Africa setting through a Foundation Botnar research grant (grant number 6270) to improve SAA applicability for LMIC populations. This has included symptoms/clinical-finding refinement based on additional attributes, for example, intensity or temporality and epidemiological data. These have been used to specify the prior probabilities of diseases to allow for correct probability estimations for this setting. It has also included optimisation of maternal and neonatal conditions, infectious diseases, non-communicable diseases, sexually transmitted infections, trauma-related injuries, mental health issues and neglected tropical diseases. In October 2019, the Ada SAA was made available at no cost to users in Swahili, via the Apple and Google app stores, and currently has over 92,000 users in Tanzania who have completed over 94,000 symptom assessments. In a recent clinical investigation in a mental health use case of Ada in an outpatient clinic, the average completion time of an Ada assessment was 7.90 (SD 3.39) minutes and an average of 31.90 (SD 8.11) questions were asked.21 The Ada app is available both on Android and iOS devices in seven languages, and business-modified versions of the app can be found through various enterprise solutions with Ada business partners. A screenshot of the SAA used in the study in both English and Swahili is presented in figure 1. The left panel shows the Ada starting screen in English, and the right panel shows the screen in Swahili. After the starting screen, the user is guided through a series of questions about their presenting complaint and symptoms. The Ada app is currently available in seven languages. This study will assess children (2–119 months), adolescents (10–19 years) and other adults (20 years and above) who arrive at the study site. All patients who enter the clinic and are willing/able to provide consent will be included, with the exception of (1) Patients with severe injury/illness requiring immediate treatment, (2) Patients with traumatic injury (many of these patients require minimal anamnesis, and it is not rational to include them in a pilot study), (3) Patients incapable of completing a health assessment (e.g., due to illiteracy, mental impairment, inebriation, or other incapacity). Data from patients dropping out of the study or deviating from protocol will be excluded from analysis. Inclusion of patients will be monitored throughout the study in order to ensure recruitment of a study sample of patients with a comprehensive spectrum of symptoms, constellations and conditions: this is to ensure that this pilot study tests the performance of SAA on a broad range of scenarios, and that it not only provides detailed testing for the most commonly presenting patient scenarios. Study recruitment will be carried out to a target of enrolling between two and five patients for each of the following categories (including at least one adult and one child in each category): (1) Conditions related to abdominal pain or gastrointestinal issues, (2) Conditions related to the lower respiratory system, (3) Conditions related to the upper respiratory system, (4) Conditions related to mental health, (5) Conditions related to ophthalmology, (6) Conditions related to orthopaedics, (7) Conditions related to the cardiovascular system, (8) Conditions related to the genitourinary system, (9) Conditions related to ear, nose and throat (ENT), (10) Conditions related to the skin, (11) Conditions related to the female reproductive system and obstetrics, and (12) Conditions related to the neurological system. Once a total of five patients have been enrolled for a given category, no further patients will be included. There will be cases in which the presenting complaint does not match the condition category which the patient is ultimately diagnosed with; for this reason, the physician diagnoses will be aggregated on a dashboard recruitment adapted to optimise recruitment according to the categories listed above. The monitoring of this detailed recruitment is possible by the work of two study trackers, employed at the study site, who have nurse-midwife-level and clinical-officer-level medical training, respectively, and who can ensure the category tracking is followed; the recruitment is expected to become slower towards the end of the study because it is more difficult to enrol condition categories that still need to be filled. The pilot study will be preceded by a feasibility and process optimisation phase, in which 15 patients will be recruited. This phase is for the optimisation of general study procedures, of patient tracking and information recording in the busy clinic environment, and to determine if staff training has been adequate. If deficiencies in study process or staff training are identified, then these will be rectified, and a period of up to 2 weeks has been allowed in study planning for this. There will be no alteration in usual care for these patients and their data will not be analysed for the investigation of the study hypotheses. Following the feasibility phase, at least 50 patients will be recruited for the pilot study. The patient’s journey in the study will consist of three stages (see overview in figure 2). The patient journey in the study. Study recruitment staff will coordinate closely with hospital staff to determine when there is a potentially eligible patient in the waiting room, a system that has been successfully used to recruit patients in previous studies at this site. The study team will approach potentially eligible patients, provide details of the study, and obtain written informed consent. Parents/caretakers will be asked to consent for their children’s participation. In addition, all children aged between 9 years and 18 years will be asked to provide an assent to their participation. The consent form will be in Swahili language (see online supplemental file 2 for the English version). bmjopen-2021-055915supp002.pdf Each patient will be assigned a single study ID (this study ID is exclusive to the study and is not part of the usual care electronic health record). The SAA will be described to the patient, who will then use it independently to assess their symptoms. If the patient asks for assistance in SAA use, study staff will assist them and will record, on a modified Likert Scale, the degree of assistance provided. The results of the symptom assessment will not be shared with the patient or any of the health workers in the clinical setting. The patient will then proceed to usual care, which will either be a consultation with a clinical officer, an assistant medical officer or with a medical doctor (here referred to collectively as ‘usual care health practitioner’). A study-structured consultation form will be completed by the usual care health practitioner, which will either be a paper-based case report form (CRF) or tablet-based eCRF (using a REDCap application). Additionally, the usual care physician will fill in the standard hospital forms, collect vital signs and plan for investigations, as required. All patients will then proceed to a consultation with a study-provided physician, who will also complete the appropriate structured consultation form as a tablet-based eCRF. The study-provided physician can only, at the end of their consultation, refer to the notes collected by the usual care health practitioner for effective management whenever needed, so as to avoid bias. After the patient has completed the full study process, the patient will be asked to complete a purpose-designed survey about the SAA. This survey can be found in the supplemental files as online supplemental file 3 and asks the patient questions surrounding how they liked the Ada assessment and to what extent using the Ada assessment before their doctor consultations made them approach the visit differently than they normally would. bmjopen-2021-055915supp003.pdf As this is an observational, prospective study, no experimental or control interventions are conducted. The condition-suggestion accuracy and comprehensiveness of the SAA on a pilot level, evaluated against the gold standard differential diagnoses determined by the review panel, reported in the context of the accuracy of the usual care health practitioner. This current study is a pilot assessment of feasibility for the planning of a later study, and the number of subjects required to determine true accuracy and comprehensiveness will be included in a later study. In this pilot study, we determine a preliminary measure of accuracy and comprehensiveness. The process of data collection and physician panel assessment consists of five stages (see overview in figure 3). The first three stages in this process have been described above (ie, (1) The patient SAA use; (2) The consultation with the usual care health practitioner; and (3) The consultation with the study-provided physician). The subsequent steps are: Data flow chart. Gold standard urgency advice levels. Data analysis will then be carried out on these lists (see section ‘Data analysis’ below). Patient questionnaires and usual care health practitioner questionnaires will be analysed and described using methods appropriate to modified Likert scale questionnaire data.22 As this is an observational (ie, non-interventional) study that does not pose any risk to the patient, there is no need for additional safety management, for example, a data monitoring committee. Patients requiring immediate medical care and clinically unstable patients are excluded from recruitment. There will be no delay in the diagnosis and treatment of any patients, since if they are called into their appointment before completing the SAA assessment, they will then be excluded from the study process and analysis and instead proceed to their usual care. Study-enrolled patients will receive one extra consultation with the study-provided physician, which is highly unlikely to delay the patient’s diagnosis or treatment, as this will generally require less than 10 minutes. Data entry will take place according to the guidelines prepared in the study data management plan. For the paper CRFs completed by usual care healthcare practitioners, double data entry will be carried out (data will be entered by two operators separately). All consent forms will be paper based. All other data will be collected electronically. Data will be collected at the study sites through a secured local area network, which will allow data sharing on-site. A clinical trial Electronic Data Capture (EDC) system (REDCap), will be used for data capture. Study personnel will be trained on the system and be provided a unique username and password. Paper records will be kept in a locked cabinet in the facility and will only be accessed by study-specific personnel. At the point of EDC, the research assistant will verify the data and then commit it to the EDC. The data will then be automatically locked, and the research assistant will no longer have access to the data. Data collected from the study will be stored for a minimum of 3 years from the date of the last patient out by Muhimbili University of Health and Allied Sciences. Source-data verification will be conducted for 10% of the digitised usual care consultation notes. Top-1, top-3 and top-5 accuracy (known as M1, M3 and M5) and comprehensiveness as defined by Gilbert et al8 of the SAA will be evaluated against the gold standard differential diagnoses, as will the accuracy and safety of the urgency-advice levels from the panel. Using the standard of M1, M3 and M5 in studies such as this one has been determined through its use in many other similar studies, as it shows the percentage of cases where the top-1, top-3 or top-5 condition-suggestion matches the gold standard main diagnosis.8 17 18 21 Data analysis will be conducted as described by Gilbert et al.8 Briefly, condition-suggestion accuracy and urgency-advice accuracy will be compared using descriptive statistics and tests appropriate for categorical data. χ2 tests will be used to test whether the proportion of correct condition suggestions from the SAA, from the usual care medical practitioners and from the study-provided physicians are drawn from the same distributions. In case of a significant difference, two-sided post hoc pairwise Fisher’s exact tests will be used to compare the SAA and practitioners. The usual care health practitioner rating will be strictly anonymised, as the purpose of the study is to compare the SAA to usual care and will not be used to audit individual usual care health practitioners. Condition categories as defined in the section ‘Study population and eligibility criteria’ will be broad and are used to gain a general overview on how the Ada app and doctors perform over different categories. Analysing this will provide an important insight into the SAAs’ strengths and limitations. There is limited research in clinical settings exploring how SAAs perform in different disease types of affected body systems. The study was designed as a guide to a later larger trial, and in line with literature on pilot study design.23 24 Therefore, the sample size was estimated on the basis of having sufficient patients to assess accuracy and comprehensiveness on a pilot scale, survey completion rate, and to determine if there were any safety-related considerations that might be needed in a later larger study. Aspects that will be piloted are: (1) Trialling of new procedures and enabling power calculations intended to be used in a later single or multicentre randomised controlled trial; (2) Determining a pilot-based overview of accuracy and comprehensiveness for a comprehensive range of symptoms and conditions in varied age groups; (3) Establishing how many patients and/or healthcare professionals can be recruited and the feasible level of completed patient and physician questionnaires; and (4) Evaluating the general technical and logistic feasibility of a full-scale study, including issues of data collection and questionnaire design. This study is anticipated to start in July 2021 and the duration of patient recruitment to last for 2 months.