Brenda Y Miao, BAa ∙ Madhumita Sushil, PhDa ∙ Ava Xu, PharmDa,b ∙ Michelle Wang, PharmDa ∙ Douglas Arneson, PhDa ∙ Ellen Berkley, PharmDc,d ∙ Meera Subash, MDe,g ∙ Rohit Vashisht, PhDa ∙ Prof Vivek Rudrapatna, MDa,f ∙ Prof Atul J Butte, MDa,h
Summary
Digital therapeutics (DTx) are a somewhat novel class of US Food and Drug Administration-regulated software that help patients prevent, manage, or treat disease. Here, we use natural language processing to characterise registered DTx clinical trials and provide insights into the clinical development landscape for these novel therapeutics. We identified 449 DTx clinical trials, initiated or expected to be initiated between 2010 and 2030, from ClinicalTrials.gov using 27 search terms, and available data were analysed, including trial durations, locations, MeSH categories, enrolment, and sponsor types. Topic modelling of eligibility criteria, done with BERTopic, showed that DTx trials frequently exclude patients on the basis of age, comorbidities, pregnancy, language barriers, and digital determinants of health, including smartphone or data plan access. Our comprehensive overview of the DTx development landscape highlights challenges in designing inclusive DTx clinical trials and presents opportunities for clinicians and researchers to address these challenges. Finally, we provide an interactive dashboard for readers to conduct their own analyses.
Introduction
Digital therapeutics (DTx) are a somewhat novel class of US Food and Drug Administration (FDA)-regulated software that help patients prevent, manage, or treat disease. Beyond providing additional therapeutic options for patients, the method of delivery of DTx also enables the delivery of continuous and personalised care at scale.1,2 Examples of approved DTx include the Propeller platform, which uses smart devices and paired consumer applications to improve medication adherence and reduces hospital admissions in patients with asthma and chronic obstructive pulmonary disease (COPD),3,4 and EndeavorRx, a video game that helps improve attention function in children with attention-deficit hyperactivity disorder.5 Although DTx have the potential to help bridge gaps in access to care, there are concerns that these software will require access to compatible devices or high digital literacy, and widen disparities in health outcomes.1,6 There is also substantial interest from health-care and regulatory institutions to analyse the clinical development landscape and quality of clinical evidence available for DTx.6,7
ClinicalTrials.gov is the main website in the USA for registering clinical trials, as required by the FDA Amendments Act of 2007.8 Several studies have previously used the ClinicalTrials.gov registry to characterise the level of clinical evidence for drug therapeutics, including analysis of clinical trial design and applicability of trial results to real-world populations.9–11 Analogous studies of clinical trials involving digital interventions12–14 have focused on structured data fields, and only a few have attempted to provide additional insights through manual free-text analysis. However, manual analysis is time-consuming, requires specialised expertise, and is difficult to keep up to date as new DTx trials occur, and so automated tools are necessary to provide real-time insight into emerging trials.
In the past 5 years, developments in natural language processing (NLP) have made automated information extraction readily available for biomedical text. Software tools, such as SciSpacy, provide open-source access to text analysis pipelines and NLP models, which are pretrained on large biomedical datasets and can achieve high accuracies on information extraction and other language tasks.15,16 These pipelines can also map extracted concepts to existing biomedical vocabularies, such as MeSH categories, for standardisation and downstream analysis. Several NLP methods have been applied to analyse drug therapeutic clinical trials,11,17 but have not yet been used to characterise the clinical development of DTx.
Given the increasing availability of DTx and their corresponding clinical trials, we did a systematic review to describe the characteristics of trials on DTx. We took advantage of modern NLP methods to better understand the characteristics of DTx clinical trials and the quality of evidence available for these novel therapeutics. Finally, we provide an interactive dashboard for readers to do their own analyses of DTx studies using structured and unstructured data fields from ClinicalTrials.gov.
Methods
Search strategy and selection criteria
Digital therapeutics clinical trials were identified through the ClinicalTrials.gov application programming interface by use of a set of 27 search terms related to DTx, including “digital therapeutic”, “digital therapy”, “smartphone”, “mobile app”, and “video game” (appendix p 1). Searches were limited to the fields for BriefSummary, BriefTitle, InterventionName, InterventionDescription, Keyword, DetailedDescription, EligibilityCriteria, or OfficialTitle, and only trials registered for FDA-regulated devices and not listed as having a “basic science purpose” were included. We used the ClinicalTrials.gov field IsFDARegulatedDevice to identify trials “studying a device product subject to section 510(k), 515, or 520(m) of the Federal Food, Drug, and Cosmetic Act”.18 Thus, even if FDA clearance or approval had not been granted for any of these trials, there was a high degree of confidence that they were for FDA-regulated products. Basic science studies were identified with the DesignStudyPurpose field and were removed to focus on trials of DTx with an established mechanism of action. By use of the OverallStatus field, trials that had been terminated, withdrawn, suspended, or had an unknown status were also excluded to limit analysis to active trials. The scope of the systematic review was also limited to studies with start dates occurring after 2010, or expected completion dates listed after 2030. Following these filtering steps, the full record from each remaining DTx trial was then extracted from the complete ClinicalTrials.gov dataset, which was downloaded on Aug 3, 2022. We report our findings in line with PRISMA guidelines. Since this systematic review does not assess health outcomes, no protocol is registered on PROSPERO. The full list of data fields available for each trial can be found on ClinicalTrials.gov on the Protocol Registration Data Element Definitions page.18
Analysis of clinical trial characteristics by use of structured data fields
We compared the number and duration of interventional and observational trials, with duration calculated as the number of years between reported start and completion dates. Clinical trials were also analysed on the basis of sponsor and collaborator types, visualised with a Sankey diagram. To understand the geographical distribution of clinical trial facilities in the USA, each entry in the LocationState field was mapped to a state code with the pgeocode software package (version 0.3.0) and the number of trials in each state was plotted as a choropleth map. The density of clinical trial facilities in each state was also calculated as a ratio of trial locations to the population of each state, by use of the 2021 estimated US Census Bureau values.19
We analysed correlation between the number of clinical trial locations and the area deprivation index (ADI), a metric of socioeconomic status in each region. ADIs for the five states with the highest number of clinical trial locations were downloaded from the University of Madison Neighborhood Atlas and mapped to each listed facility’s zip code.20,21 National and state ADIs were analysed, with national ADI score given as a percentile across the entire country. At the state level, ADI is provided on a scale from 1 to 10. Higher scores represent greater socioeconomic disadvantage for both state and national ADIs. Only trials with available features in each data field were considered for these analyses (appendix p 2).
Extraction of condition and eligibility criteria by use of NLP
Although ClinicalTrials.gov has an internal algorithm to map conditions listed with standardised biomedical vocabulary to MeSH terms, these terms do not correspond to the main MeSH branches and are not available for all clinical trials.22 To create standardised mappings for each clinical trial, medical conditions from the condition free-text field were extracted and mapped to MeSH terms by use of the MeSH EntityLinker from SciSpacy (version 0.5.0),15 with only the first match selected for each condition. Resulting terms were grouped into MeSH categories and the most frequent heading was selected for trials with multiple conditions, with priority given to values under the branches C (diseases) and F (psychiatry and psychology). MeSH terms were manually reviewed to assess the validity of the MeSH EntityLinker on this dataset. With conditions classified into standardised clusters, we compared enrolment counts in each MeSH heading, focusing on non-phase 1, interventional trials in groups with fewer than ten studies. The EnrolmentType field was used to differentiate between actual and anticipated enrolment for each trial.
To analyse the most common types of eligibility criteria, we used the BERTopic topic modelling technique (version 0.11.0),23 which clusters text embeddings to produce interpretable, semantically cohesive clusters. BERTopic has been used in previous studies of biomedical text and has been shown to generate more coherent topics compared with Latent Derelict Aldrich or other topic modelling methods.24 To generate embeddings for BERTopic, text from the eligibility criteria field was first split into inclusion and exclusion criteria, with each line considered a separate document. A language model from SciSpacy pretrained on biomedical text (en_core_sci_lg) was then used to generate embeddings for each eligibility criterion. The SciSpacy embeddings encode semantic relationships between biomedical terms, allowing related terms to be grouped into more semantically cohesive topics, unlike conventional methods that cluster words only on the basis of their frequency and co-occurence.15 A BERTopic model with default settings was used to generate topics from these embeddings, and the top five topics for each eligibility criterion were mapped back to the corresponding clinical trial to analyse the percentage of each topic occurring in each MeSH cluster. Again, a subset of the 200 inclusion and exclusion criteria were manually reviewed to confirm that the eligibility criteria were mapped correctly to these topics. Only MeSH groups with at least 15 studies were analysed. Topic modelling was done on inclusion and exclusion criteria of interventional trials in our dataset not listed as a phase 1–4 trial.
Development of an interactive dashboard for DTx clinical trial analysis
The dashboard for clinical trials data analysis was built with Streamlit. The dashboard implements all the methods described in this systematic review for analysis of study types, sponsor types, conditions, and eligibility criteria.
Statistics
Descriptive statistics are provided for categorical variables as proportions, and averages are reported for continuous variables as medians and IQRs. Spearman’s rank correlation coefficient (r) values were calculated to analyse the correlation between continuous variables. Mann-Whitney U tests were used to establish differences in median enrolment between MeSH categories and Bonferroni correction was used to account for multiple testing. Statistical testing was done with Scipy (version 1.7.3) and p values less than 0·05 were considered significant.25
Results
Using 27 search terms related to digital therapeutics (appendix p 1), we identified 8615 clinical trials involving digital-based interventions. Of these trials, 7386 were active or ongoing, and 7221 had a start date after 2010 and expected completion date before 2030. Since DTx are regulated by the FDA as “software as a medical device”, we only considered studies that were listed as using FDA-regulated devices and conducted for non-basic science purposes, resulting in 449 studies of interest (figure 1). Of these 449 studies, 53 (11·8%) were observational and 396 (88·2%) interventional (figure 2), with 74 interventional studies listing a completion date in 2022, and 88 in 2023. Overall, 150 interventional and 18 observational studies were listed as completed, with median study durations of 1·02 years (IQR 0·57–1·69, range 0·06–5·17) and 0·69 years (0·32–1·59, range 0·05–5·42), respectively (figure 2). 13 observational and 68 interventional studies were first posted to the registry in 2022 (appendix p 3). Because all information on ClinicalTrials.gov is voluntarily reported by the sponsor of each clinical trial, only available data are used for each analysis and missingness is reported in the appendix (p 2).