Clinical data refers to data collected on humans to assess the safety and performance of a medicine, treatment, or device. In the medical device industry, the European Medical Device Regulation (EU MDR 2017/745) defines clinical data as, “information concerning safety or performance that is generated from the use of a device” . According to the US Food and Drug Administration (FDA), clinical study data includes patient demographic information, details of any medical treatment received, and a description of the patient’s medical progress, as well as any other relevant information . Moreover, the clinical data definition according to IMDRF1 MDCE WG/N55 FINAL:2019 (formerly GHTF/SG5/N1R8:2007), introduces ‘effectiveness’ as a source of information to substantiate clinical data: “safety, clinical performance and/or effectiveness information that is generated from the clinical use of a medical device”.
Clinical data is an essential element in determining the benefit-risk profile of a device in the EU under EU MDR 2017/745, and ultimately, whether the device conforms to the relevant regulatory requirements and can be placed on the market. Clinical data provides information on potential risks associated with the device, which may result from device malfunctions, foreseeable off-label use, biological hazards etc (see Figure 1). These risks feed into the manufacturer’s risk management processes, with the aim of reducing them as far as possible without adversely affecting the benefit-risk ratio of the device (in accordance with risk management ISO 14971:2019) . Additionally, clinical data is used to support the performance endpoints of a device and to substantiate the intended clinical benefit . For example, a device that intends to measure blood-glucose levels with the intended clinical benefit of reducing the number of hypoglycaemic events will need clinical data to support the provision of accurate and precise measurements of blood-glucose levels in the intended population(s).
Clinical data can be collected at all stages of the lifecycle of a medical device, from pre-market clinical investigations (CIs) to post-market surveillance (PMS) of real-world usage of the device  as depicted in Figure 1. As such, it is not only used to establish the safety and performance of a device before market-release, but also to establish potential new and emerging risks once on the market. This is achieved through the continuous monitoring and collection of real-world safety and performance data .
Collection of high-quality clinical data is vital in clinical research and evidence collection. To this end, identification of clinical research gaps from all involved stakeholders is required, and the entire processes of data collection must be thoroughly discussed in the related Standard Operating Procedures . Additionally, sponsors are obliged to monitor (either on-site or remotely) all research activities, including clinical data collection . For example, on-site or remote monitoring should evaluate and identify potential data collection errors, compliance with protocols, completeness of collected data, unusual data distribution, and review of data collection in real-time (data supervision) .
Potential clinical data collection sources may include (but are not limited to) clinical trials (CTs), clinical investigations (CIs), electronic health records/medical records, administrative data (e.g., hospital records, pharmaceutical prescriptions), health and post-market clinical follow-up surveys, and patient registries [5, 6, 7, 8, 9] (See Figure 2). The MDR introduces post-market surveillance (PMS), and in particular, post-market clinical follow-up (PMCF) as a clinical data source . Clinical data analysed and reported at a PMCF level are part of a continuous process that updates the clinical evidence substantiated in the clinical evaluation report (according to EU MDR 2017/745 ). This requirement for continuous monitoring and collection of safety and performance data is fulfilled by the collection of real-world data (RWD) . RWD, according to the FDA Framework for Real-World Evidence Program, is the routinely collected data relating to patient health status and/or the delivery of health care, and is applicable to medical products in general (not just medical devices) . Real world evidence (RWE) is the result of the analysis of RWD and provides evidence on the performance and the potential benefits or risks of the product during real-world use.
The type of data to be collected will determine the methods and processes used for data collection and data analysis. Paper-based, as well as electronic and hybrid methods (combination of paper-based and electronic means), are the most common methods used for data collection activities [8, 12]. Depending on the design of the clinical study and the related endpoints, clinical data are generated and/or collected by the investigator(s), research staff, and/or participants/patients . Clinical research studies (CTs and CIs included) must be conducted in accordance with appropriate scientific principles, good manufacturing practices (GMP), and ethical considerations (Helsinki declaration, good clinical practice (GCP)) to ensure that meaningful data is generated [1, 14, 15, 16]. From the early stages of participant/patient recruitment to the final stages of data analysis and reporting, clinical data management (CDM) plays a vital role in data handling by monitoring and ensuring compliance with regulations and standards [13, 17, 18]. CDM processes are essential for the design of all clinical data collection steps, providing an “error-free, valid, and statistically sound” data collection agenda .
The process of clinical data collection from CTs/CIs can be both time-consuming and costly and can therefore hinder the development of new health-related innovations . Costs can arise both internally from the sponsor and externally from participant/patient recruitment, data management and monitoring, and regulatory planning and fees, to name a few [19, 20]. Clinical data collection in drug development studies is a great example of cost inefficiency, as multiple factors (e.g., variation of drug type, regulatory policies, therapies, manufacturer’s company size) create ongoing and substantial study costs [19, 21, 22, 23]. An overview of the elements involved in clinical data collection and the associated costs is shown in Figure 3.
Sufficient clinical evidence refers to the “amount and quality of the clinical data and the clinical evaluation results which allow a qualified assessment of whether the device is safe and achieves the intended clinical benefits when used as intended by the manufacturer” . Sufficient clinical evidence is required to evaluate potential (new) risks associated with the device under normal conditions of use, as well as to identify and substantiate the device’s clinical benefit (i.e., the positive impact of a device on the health of an individual or the public ). Sufficient clinical evidence is necessary to support a positive benefit-risk profile of a device.
One of the major challenges for manufacturers under the EU MDR is to be able to provide sufficient clinical evidence to support conformity with the General Safety and Performance Requirements (GSPRs). This is due to the more strict criteria for what constitutes an acceptable source of clinical data (i.e. some sources previously accepted under MDD are no longer accepted under MDR), the up-classification of devices (e.g. stand-alone software can now potentially be categorized as a Class III medical device ) and the strict requirements around equivalent devices. For example, for a manufacturer to claim equivalence, the device in question needs not only to meet requirements for similarity on technical, biological and clinical characteristics to the device to which they are claiming equivalence, but they must also be able to prove that they have sufficient levels of access to the reference device’s technical documentation .
One of the sources of clinical data, as suggested in the MDR, is from the real-world use of the device (clinical experience). Although MDR doesn’t use the term ‘real-world data’ (RWD) or ‘real-world evidence’ (RWE), RWD appears as a term in MDCG 2020–6 . The term RWE has been used more recently by the FDA, such as in the 2017 FDA guidance on “Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices” , and the 2018 “Framework for FDA’s Real-World Evidence Program” . Based on this, we use the term RWE. Sources of RWE can be seen in Figure 4.
Collection of RWE can be particularly useful to substantiate the continued acceptability of the benefit-risk profile throughout the lifetime of a medical device. For example, under the EU MDR there is a strict requirement to conduct a post-market clinical follow-up (PMCF) (e.g. with the collection of RWE) in order to proactively monitor, collect and analyse performance and safety of a device throughout its entire lifetime, regardless of device class, legacy and technology .
In the USA, medical devices are regulated by the FDA based on the Medical Device Amendment of 1976 . The vast majority of submissions are via the 510(k) pathway, with an average of 2,982 510(k) approvals between 2015–2020, compared to only 38 via Pre Market Approvals (PMA) . The difference is indicative of the requirements of the two regulatory pathways; whereas a 510(k) submission is based mainly on substantial equivalence of an already marketed device, with minimal additional pre-market clinical data generated for the target device, the PMA pathway is based on the generation of clinical data through clinal investigations. Although clinical data on the device in scope is rarely requested by the FDA for 510(k) submissions , the need for device-specific clinical data arises from PMS requirements, which dictate the monitoring and collection of RWE on the safety and performance of the device (Class I devices are exempt from this requirement) .
The collection of RWE has become of greater focus under the FDA due to the FDA RWE program. Within the drug sector, the FDA’s RWE program aims to optimize the use of RWE for regulatory decisions, specifically the use of RWE to support drug effectiveness and approval of a new indication for an already approved drug, or to support post-approval study requirements . In particular, the program aims to evaluate the use of RWE for supporting labelling changes on drug effectiveness, adding/modifying an indication, or changing dose/regimen/administration, amongst others . Although the RWE program is currently focused on drugs, recent guidance from the FDA on the use of RWE to support with regulatory decision-making within the medical devices sector has also been made available . For example, the FDA proposes that RWE may be able to provide information on a broader patient population compared to a clinical trial or investigation, and has potential implications for reimbursement decisions . As such, the collection of clinical data, particularly RWE, is at the forefront of medical device and drug regulations.
One way to aid with the need for collecting sufficient clinical data, particularly RWE, is through the use of digital health technologies (DHTs). DHTs can provide a more efficient and timely way of collecting numerous types of clinical data (e.g., physiological and behavioural data) and can also be beneficial with regards to participant recruitment, data management (e.g., data captured through DHTs can be sent directly to investigators and sponsors) and cost reduction (e.g., reduction in the burden and costs of traveling for monitoring staff and/or participants) . This article aims to highlight the value of DHTs in aiding with the collection of clinical data (particularly RWE). In addition, we discuss the considerations and challenges with regards to selecting and implementing DHTs in clinical research and outline the current guidelines and regulations that support this.
Digital health technology (DHT) includes a broad range of technologies, such as smartphone applications, wearables, ingestibles, implantable, cloud-based solutions, and other mobile platforms with sensors. The World Health Organisation (WHO) taxonomy of DHTs categorizes DHTs by the ways in which they can be used to support health care, and includes four overarching categories based upon the primary user: interventions for clients, interventions for healthcare providers, interventions for health system or resources managers, and interventions for data services .
In the last few years, rapidly expanding technology has enabled more powerful algorithms (software) to translate data detected by sensors (hardware) into clinically useful endpoints (health outcomes). An accelerometer, for example, might be included in wearable technology, and various algorithms can be used on the data to generate estimates of total sleep time, steps per day, and other endpoints that provide relevant data on health outcomes (such increased activity). Additionally, DHTs can be utilized to create new endpoints as well as digitizing existing ones. Real-time data capture and analytics, as well as the ability to capture day-to-day variability by collecting data continuously, are some of the potential benefits of DHTs .
The FDA’s recently published draft guidance on “Digital Health Technologies for Remote Data Acquisition in Clinical Investigations” defines DHT as “a system that uses computing platforms, connectivity, software, and/or sensors, for healthcare and related uses” . DHTs often contain sensor hardware that can capture physiological and/or behavioural data (e.g., blood pressure or physical activity). This data may be translated into useful endpoints (e.g., hypertensive event) that are of interest to clinical research. Table 1 provides some examples of different types of DHT devices that can be used in various clinical conditions. Cardiovascular disease (CVD), in particular, has been at the forefront of DHT advancements, and includes eHealth and mHealth therapies, as well as developing tools like big data analytics, AI, and machine learning . When compared to usual care, a systematic review and meta-analysis of 51 articles found that DHT significantly reduced CVD, with accompanying reductions in weight and body mass index .
|CLINICAL FIELD/CONDITION||TYPE OF DHT||FUNCTION/DATA OBTAINED|
|Allergic rhinitis||Smartphone/tablet/laptop with Allergy monitor app (see  for more details)||Questionnaire that records symptoms (eyes, nose, lungs), recording of medication intake, adherence to sublingual immunotherapy and any side effects. Data presented in graphs to show evolution of symptoms over time|
|Anxiety disorders||Virtual reality headset that projects a feared virtual environment/stimulus (see  for more details)||Enables virtual reality exposure therapy (technology-mediated form of exposure therapy)|
|Cardiovascular disease||SMART-TRIAL Electronic Case Report Form (eCRF) with API (data input and export from a study)||Platform used to collect and manage data from Cardiologs Holter platform (heart arrhythmia diagnostic software)|
|Central Nervous System Disorders||Smartphone/wearable device as part of the Remote Assessment of Disease and Relapse in Central Nervous System Disorders (RADAR-CNS) project (see )||Helps patients to continuously monitor moods and behaviours, symptoms, and daily function, to enable assessment and monitoring of changes in disease state over time.|
|Chronic obstructive pulmonary disease||Smartphone/tablet/laptop with Telehealth app (see  for more details)||Enables patients to self-manage their condition by monitoring of activity and symptoms, medication reminders etc|
|Diabetes||Smartphone/wearable sensor that serves as a hypoglycaemia alarm (see  for more details)||Detection of alterations in heart rate, electrocardiogram patterns, pulse-wave patterns, electroencephalogram patterns, galvanic skin response, skin temperature, and breath volatile organic compounds|
|Inflammatory bowel disease||Smartphone/tablet/laptop with Health PROMISE app (see  for more details)||Provides a representation of disease activity and QoL over time, tracking of vaccines, screenings, etc., patient access to current plan and ability to message care team|
|Kidney disease||Smartphone/tablet/laptop with Telehealth app (see  for more details)||Allows the patient to be seen/assessed remotely (via video) by the clinician|
|Neurological disease||Smartphone/wearable sensor with inertial measurement units to detect Parkinson’s disease symptoms (see  for more details)||Detection of tremor, motion fluctuations, gait freezing or impairments, limb motion, instabilities, stiffness etc|
|Oncology||Virtual reality computer games for paediatric patients (see  for more details)||Enables therapeutic play to reduce depressive symptoms|
|Orthopaedics||Virtual reality headset that projects patient-specific programs for rehab (see  for more details)||Enables surgeons and physical therapists to remotely deliver and monitor physical therapy programs that the patient can perform at home.|
|Rheumatoid arthritis||AI/ML (ML is a type of AI that learns from data as opposed to being programmed to follow rules) (see  for more details)||AI/ML model that uses data on demographic and clinical-related variables to predict all-cause mortality|
Software applications can also be classed as DHTs, such as software used to administer electronic data collection or applications that support triage decisions . An example of this is SMART-TRIAL by Greenlight Guru, which is an Electronic Data Capture (EDC) platform that aids with clinical data collection for PMCFs and CIs. The platform has multiple uses, including aiding with extracting and managing data from other connected devices or software. For instance, the SMART-TRIAL platform has been used to directly transfer data from a heart arrhythmia diagnostic software (Cardiologs, by Philips) to enable validation of the Cardiologs algorithm . The SMART-TRIAL EDC platform offers a unique and modern open Application Programming Interface (API) which can be used by DHTs and connected devices. Few EDC platforms support integration of continuous data through modern API interfaces like SMART-TRIAL. This is largely due to 1) continuous data can take up a lot of data storage space, which can be costly for vendors and 2) continuous data structure can vary in origin, type, and format, which requires a flexible API in order to support various DHTs and connected devices in modern clinical activities. The use of platforms/databases as tools to support other DHTs and/or clinical data collection is becoming increasingly common and can aid with reducing the time needed for clinical data collection and management .
Across the EU and US, recent initiates for DHT have been implemented (see Figure 5). Within the EU, an effort is being made to enhance the availability of DHTs to citizens. The European Commissions’ 2018 communication on the transformation of digital health and care  aims to address three key areas: 1) the secure access to and sharing of data; 2) a shared European data infrastructure to allow faster and efficient sharing of health data for research and diagnosis and, 3) the strengthening of citizen empowerment to enable citizens to take a greater role in the management of their own health .
In the US, both private and federal investment into digital health care (i.e. health care supported by DHTs), such as the Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009, is promoting the use of health information technology, such as electronic health record (EHR) systems . More recently, the FDA created the Digital Health Center of Excellence, which aims to “Empower stakeholders to advance health care by fostering responsible and high-quality digital health innovation” . The Clinical Trials Transformation Initiative (CTTI), a public-private collaboration created by Duke University and the FDA, has released four sets of guidelines and tools aimed at enhancing clinical trial quality and efficiency by using DHTs. The development of innovative endpoints, the design and execution of decentralized trials, the use of mobile technologies in clinical trials, and the optimization of mobile clinical trials through involving patients and sites are among the topics discussed. Investigators should conduct small feasibility/pilot studies before starting a clinical trial, with the overall goal of reducing risk by assessing sensor accuracy, developing and/or validating algorithms and optimizing data quality .
There are numerous beneficial uses of DHT in clinical research. In 2016, a think-tank involving various stakeholders (academia, industry, and regulatory bodies) identified numerous potential uses of DHT that span the entire process of a clinical study, from fundraising and recruitment to data management (see Figure 6) .
DHTs can aid substantially with the recruitment process, leading to a reduction in costs and reduced timeframes [42, 43, 44]. Challenges with recruitment include costs of advertising, lack of enrolment and difficult-to-reach populations (e.g. rare medical conditions) . Evidence suggests that social media platforms can help researchers reach segments of the population that may otherwise be difficult to access, or to identify participants from their desired demographic and eligibility criteria . Additionally, the use of DHT makes participation easier by removing potential issues such as travel, time restraints and access to the investigation site, as data can be collected remotely through a device or application [32, 42]. This can aid with recruitment, as shown by a recent survey that found participants preferred a mobile study compared to a conventional one due to negating the need to travel (amongst other reasons) . Furthermore, evidence suggests that a significant proportion of patients face substantial travel burdens with regards to cost and distance when taking part in clinical trials .
Recently, DHTs have also been used to gain informed consent from participants, for example via apps and video . The idea behind these methods is to make the consent procedure easier to understand so that participants are fully aware of the potential risks and benefits involved. However further development is needed to prevent potential issues with competency, identity (i.e., ensuring the participant is the one giving the consent) and consenting without actually reading the information (i.e. ticking the relevant boxes but not reading the information) .
The use of DHTs enables the remote collection of data, which not only aids with recruitment (as previously discussed) but also allows for continuous and more frequent data collection (e.g., during daily activity and even sleep) and from numerous environments (e.g., at home, work, outdoors) [32, 44]. Costs related to the investigational site will also be removed/reduced, as a physical investigational site may not be needed if all data can be collected remotely. More frequent data collection with increased participation means that potentially the duration of clinical studies can be reduced, as the required data can be collected over shorter periods of time. This reduction in duration may in turn lead to reduced costs, as the shorter the investigation the less operating costs involved. However, whether shorter durations are appropriate is entirely dependent upon the purpose of the clinical study, as if the aim is to assess the safety and performance of a device over a specific time-period, then the length of the clinical study cannot be reduced.
Another potential benefit of DHTs is that they enable the collection of data from novel endpoints and from participants that would otherwise not normally be able to provide data [32, 42]. For example, wearable devices can collect physiological data (such as heart rate) from pre-verbal infants and from those that may struggle to communicate or are cognitively impaired (subject to ethical considerations, such as informed consent) . This is particularly useful for devices that are intended to be used in these types of populations, as it provides external validity to the findings. It’s also possible to use multiple DHTs to collect a range of information, including clinical, physiological, psychological, behavioural, and functional data . This data may provide information on endpoints that traditionally are not captured. For example, a think-tank from 2016  identified a clinical trial called the NEATHFpEF (Nitrate’s Effect on Activity Tolerance in Heart Failure With Preserved Ejection Fraction) which evaluated patients with heart failure when using a 6-week dose-escalation regimen of isosorbide mononitrate or a placebo . Daily activity level was the primary endpoint, as measured by an activity monitor, and was found to be lower in the interventional group compared to the placebo. However, when activity was measured using the more traditional 6-min walking distance, no significant difference was found, suggesting that the endpoint obtained via the DHT (activity monitor) was more sensitive than that of the more traditional one . As such, endpoints that can be collected from DHTs may provide additional, and even superior, data when used alongside more conventional endpoints .
Regarding data management, all data captured from DHTs can be transmitted directly to the relevant authorized party (e.g., investigators, sponsors etc), making a more efficient way of sharing data. This can also be done in a way that keeps all parties involved blinded/masked to the clinical study conditions . This is important to ensure potential bias is minimized when interpreting the data (e.g., interpreting data in a more favourable way for the device under investigation). Data can also be shared directly with the participants, which may keep patients engaged and motivated in the study, particularly as recent research indicates that access to data is something that participants want .
As such, there are numerous beneficial uses for DHT within clinical research, that ultimately will aid with reducing the time and costs involved in clinical studies, whilst also removing potential barriers to participant participation.
As previously outlined, there are numerous types of DHTs that can be used to aid with the collection of clinical data, from software to hardware and combination devices. The choice of DHT will mostly depend on the characteristics of the clinical study to be conducted. In particular, the clinical characteristics of the disease/condition of interest, the population and the design of the clinical study will influence which DHT to use . For example, a clinical study that aims to assess the performance of a biopsy needle in paediatrics will need a DHT (e.g., digital microscopy) that is able to measure the appropriate endpoints of biopsy procedures (such as quality of specimen) in a paediatric population (which may entail a different DHT for use in an adult population). In some cases, it may be that a participant can use their own DHT (e.g., a commercial activity tracker) and/or general-purpose computing platform (e.g., mobile phone) to collect relevant data. The benefit of this is that the participant is already using the device and is familiar with how to use it and, as such, are more likely to use it consistently and correctly. However, they may not be appropriate for clinical studies that require highly specialized or customized measurements , moreover, the data collected from different devices may not be comparable . For example, a recent survey conducted with academic, government, regulatory, and health care experts with experience with RWE raised concerns about the credibility, accuracy and reliability of algorithms that are not clinically tested . Furthermore, algorithms that differ across vendors may not produce data that is comparable (i.e. algorithms across devices may measure endpoints differently) .
Regardless of the choice of DHT, all DHTs must be fit-for-purpose, that is, they should have sufficient validation and verification to support their use and interpretability of the data they collect [32, 49]. Verification is needed to ensure the DHT accurately measures what it intends to measure, whereas validation is needed to ensure it assesses the clinical event/characteristic in the proposed population . All DHTs should be able to yield accurate, consistent results and be reliable over time .
As well as being fit-for-purpose, an ideal DHT should consider various additional elements, such as design features, intended population, and risks. For example, the Clinical Trials Transformation Initiative (CTTI) have developed a framework of elements to consider when selecting mobile health technology , which includes measurement performance, data access, and human factor specifications, amongst others . Table 2 provides an overview of various elements that should be considered when selecting the most appropriate DHT.
|Privacy and data protection||
With regards to design and population considerations, usability studies can be used to obtain feedback on ease of use from populations similar to the intended population and subsequently be used to improve the design of the DHT and/or ensure that the most suitable DHT is used for the intended population . Additionally, feasibility studies can investigate the tolerability of the DHT with the intended population, which in turn will affect the willingness of the participant to use the DHT for the intended duration . For DHTs that are wearable, mobile or contain biosensors, specific issues regarding technological literacy, anxiety regarding the tracking of data and retention issues over the long-term require additional consideration [44, 51]. For example, evidence suggests that there is an educational, wealth and age division with regards to the use of digital technology, with younger, wealthier, and higher educated individuals more likely to use digital technologies [44, 52]. This may result in the exclusion of specific sub-populations, causing population bias and loss of data from potentially relevant participants .
A key aspect that must be considered for all DHTs is the ability to provide meaningful data that can be statistically analysed. This can be a challenge if a DHT is able to collect a large amount of various behavioural and physiological data. To this end, Taylor et al (2020)  have developed a framework for the development of digital endpoints based on measurements from DHTs. They propose two types of endpoints: data-centric endpoints and patient-centric endpoints. Data-centric endpoints are intended to measure the outcome of interest (e.g., disease progression), whereas patient-centric endpoints measure how the patient feels/is functioning. The framework for both types of endpoints is built upon ensuring statistical robustness and validity to ensure that meaningful data is measured (see  for details). Likewise, the Digital Medicine Society (DiMe) have also developed a framework to help with the development and selection of digital endpoints . For example, they provide guidance on developing meaningful and measurable endpoints using a three-step approach: 1) Meaningful Aspect of Health (MAH), 2) Concept of Interest (COI), and 3) Outcome to be measured. An example provided is that of Parkinson’s disease, in which MAH may be defined as the ability to walk, COI as an activity (e.g., walking ability) and the Outcome to be measured as the number of walking bouts per day. The endpoint would be the percentage of patients that had an increase in walking bouts from baseline .
Importantly, as with any medical device, potential risks to the patient/user using the DHT must be considered. The FDA emphasizes that DHT-related risks need to be reviewed in conjunction with risk management planning and to address any potential problems that may occur when using the DHT, and furthermore, address how these problems may be managed/resolved . Potential risks may be in relation to physical properties of the device (e.g., the material may cause injury/irritation) or to erroneous data (e.g., inflated readings). For instance, a DHT that measures glucose levels has the potential to give inaccurate readings, leading to the potential risk of hypoglycaemia events or inappropriate treatment. Other potential risks include breach of data, which is one of the biggest concerns surrounding DHT. Recent data hacking scandals have highlighted the issues with keeping data secure (see  for an overview) and as such, DHTs need to be able to protect patient’s data from cybersecurity threats.
Lastly, the costs of using the DHT should be assessed, which include not only the unit cost of the DHT (plus surplus devices in case of malfunctions), but training of both researchers and participants on how to use the device correctly [32, 51]. Appropriate training for both research personnel and participants on how to use the DHT is needed, as without a clear understanding of how to use the device, inaccurate data may be collected. Participants should also understand exactly what data will be collected and the relevant privacy and data protection processes that are in place and provide consent for the collection and analysis of their data. In addition, a plan should be in place to provide technical assistance when necessary to avoid data loss/error [32, 51].
To help address these practical considerations and challenges, various guidelines and regulations concerning the use of DHTs have recently been published. For example, the CTTI has published recommendations and various resources for those that wish to conduct digital health trials, with resources dedicated to developing novel endpoints that accurately reflect meaningful outcomes to patients  and testing a DHT via feasibility, verification and validation processes , amongst others. An overview of the current regulatory frameworks and guidelines surrounding DHT are presented below.
The most recent guidance to be published on DHT is the FDA draft guidance on “Digital Health Technologies for Remote Data Acquisition in Clinical Investigations” . This draft guidance provides recommendations for all relevant stakeholders about the potential use of DHTs for the remote collection of clinical data. The guidance covers the benefits of using DHT (as discussed in Section 3.1) as well as all aspects that should be considered, such as technology selection, potential risks, and how to choose an appropriate DHT (as discussed in Section 3.2) . Furthermore, examples of different types of DHT and their uses in different scenarios are provided, along with an example of how to select a DHT for use in a clinical investigation . As such, the guidance provides a comprehensive approach for stakeholders to assess the value that DHT can bring to data collection in CIs and the processes and considerations required to implement DHT.
In addition, the guidance touches upon the use of DHTS as Drug Development Tools (DDT) or Medical Device Development Tools (MDDT) . In the US, manufacturers can chose to get their DHTs qualified as a DDT or MDDT for use in a specific context . Qualified DHTs can then be used in multiple CIs with the same context of use (without needing to go through qualification each time), to support premarket submissions of drugs or medical devices (DDT or MDDR, respectively) . Of further note is that some DHTs themselves may qualify as a medical device under the FDA definition (section 201(h) of the FD&C Act) . Although the guidance does not cover whether a DHT meets the definition of a medical device, it does note that medical devices that are intended to be used in CIs are exempt from most of the requirements, provided that the CI itself complies with applicable requirements under 21 CFR part 812 .
Other FDA guidance of relevance to the use of DHT in CIs includes the Computerized Systems used in Clinical Investigations , the draft guidance on Electronic Records and Electronic Signatures in Clinical Investigations  and the Electronic Source Data in Clinical Investigations . For example, the later aims to help with ensuring reliability, quality, integrity, and traceability of electronic source data that is captured and submitted into an electronic case report form (eCRF) . This electronic source data may come from a DHT and can be directly transmitted to the eCRF.
Within Europe, the process of incorporating DHTs in RW clinical data acquisition is less advanced . Although the use of DHTs has been considered a strategic EU health priority already since 2012 , there is currently no harmonised regulatory framework for use of DHTs in clinical research. Furthermore, the EU Clinical Trials Regulation (EU 536/2014) does not provide any provision for clinical data acquisition via DHTs. In a recent Reflection paper by the European Federation of Pharmaceutical Industries and Associations (EFPIA), it is recognised that a manufacturer/sponsor will need to navigate through complex regulatory pathways in order to use DHTs in clinical development or as a combination product with (investigational) medicinal products . In this Reflection paper, EFPIA suggests a number of initiatives to support the routine use of DHTs, which are expected to “help to improve patient care” . Additionally, the European Commission is planning the establishment of a European Health Data Space to “promote better exchange and access to different types of health data” . Despite this, a central European regulatory harmonisation for DHT application in healthcare is still in its infancy, with individual countries, such as Germany, already making progress.
Within Germany, the use of DHTs (specifically, digital health applications) is part of the 2019 Digital Healthcare Act (Digitale-Versorgung-Gesetz, DVG). This act introduced the “app on prescription”, which entails that all individuals on the German statutory health insurance have access to digital health applications (DiGa) . The DiGa directory provides a list of all digital health applications that are reimbursable. Any DiGa application is subject to a “Fast-Track” process in which the German regulatory authority (BfArM) has 3-months to assess the application in terms of its positive healthcare effect,2 security, functionality, quality, data protection and usability . To enter the Fast-Track pathway, a digital health application should meet certain provisions, including being classified as low-risk device (class I or IIa under MDR) that is used primarily by the patient to support collection of either patient-reported outcome measures (PROMs) or patient-reported experience measures (PREMs) .
The Fast-Track pathway supports generation of clinical evidence to support DHT approval by other approaches (e.g., collection of RWE). For example, the reSET from Pear Therapeutics was granted marketing clearance with use of RWE (collected from observational studies) on the device’s performance and usability . However regulatory uncertainty precludes manufacturers from using this approach, with RCTs still considered the “safest” approach for regulatory approval . Nonetheless, experience from Germany’s Fast-Track program suggests that further study and ongoing discussions should focus on reaching a consensus for best practices in the following areas: missing data, study endpoints, selection of a comparator group, multimodal interventions, hypothesis testing standardisation, equity in the included population(s), generalisability, confounders and Fit for purpose use of RWE .
In the UK, the NICE Evidence standards framework for DHTs describes the type and level of evidence needed to show the value that a DHT brings to the UK health and social care system . The standard of evidence required is based upon the tier of the DHT (Tier A: system impact; Tier B: Understanding and communicating; Tier C: Interventions), which specifies the function and level of risk. Each tier requires different types of evidence, with both minimum evidence standards and best practice standards presented . This framework thus ensures that all DHTs used with the UK system are sufficiently supported by appropriate evidence. Figure 7 provides an overview of the current implementation of DHTs by country.
An important consideration for use of DHTs in healthcare applications is the protection of sensitive, patient information. To this end, regulatory frameworks and acts have been developed to ensure (health) data protection. In Europe, the General Data Protection Regulation (GDPR)  was introduced to ensure that everyone has a right to privacy and security of personal data. It applies to anyone that processes personal data from EU citizens/residents, no matter whether the person processing it is in the EU or not. According to the GDPR, there are 7 main protection and accountability principles that must be followed if processing data: Lawfulness, fairness and transparency, Purpose limitation, Data minimization, Accuracy, Storage limitation, Integrity and confidentiality, and Accountability. In addition, there are principles with regard to data security, data protection, when you are allowed to process data, consent, and privacy rights [68, 69]. According to the EU GDPR, healthcare data falls under the category of ‘sensitive data’, which requires extensive additional technical and contractual safeguards between data processors and data controllers. Failure to conform to the GDPR can result in large fines.
In the US, the Health Insurance Portability and Accountability Act of 1996 (HIPAA) resulted in the creation of standards to protect the disclosure of sensitive patient health information without prior consent/knowledge of the patient [70, 71]. The privacy rule aims to protect patient’s health data by limiting the circumstances in which a patient’s data can be disclosed, whilst also allowing for the data to be used to inform and promote high quality health care and protect public well-being [70, 71]. It is applicable to healthcare providers, health plans, healthcare clearing houses, and business associates [70, 71]. The security rule is only appliable to “electronic protected health information”, which covers all health information that is created, received, maintained, or transmitted in electronic form [70, 72]. The security rule ensures that the confidentiality, integrity, and availability of electronic health information is protected .
Clearly it is of utmost importance to ensure sensitive, patient data is protected and secure. However, it is also important to obtain a balance between the protection and privacy of sensitive data and enabling the use of sensitive data to help improve understanding and provide better healthcare for patients [71, 73]. Methods such as pseudonymization, defined in the GDPR as “The processing of personal data in such a way that the data can no longer be attributed to a specific data subject without the use of additional information”, are methods that can be used to help ensure privacy of sensitive data .
As previously outlined, clinical research designs are evolving to incorporate the use of DHTs (Table 1), which in turn is enhancing the way data is collected, analysed and shared, from the use of remote data collection in real-world settings, to the generation of novel and patient-centred endpoints (as discussed in Section 3.1).
The recent regulations and guidance focusing on RWE collection via DHTs (see section 3.3) paves the way for the wider implementation of DHTs in clinical research, and subsequently, within regulatory decision-making processes. For example, RWE is of particular value for reimbursement decisions in Asia, due to a low number of clinical trials being conducted as a result of the financial, ethical and operational barriers, amongst others . This means that the Asian population may be under-represented in clinical trials, resulting in potential differences between populations (e.g., Caucasians and Asians) being overlooked, such as biological (e.g., genetics) and societal differences (e.g. practice guidelines) . RWE collected via DHTs could help inform reimbursement decisions where clinical trial data is lacking, and moreover, provide data on the long-term safety and performance which cannot be addressed in the limited duration of a clinical trial.  Indeed, a survey by Lou et al (2020) established that Health Technology Assessment (HTA) agencies in Asia already accept RWE to inform reimbursement, both as standalone evidence or supplementary , thus the groundwork for DHTs to aid with this is already laid.
More generally, RWE can provide data in situations where it may not be ethical or practical to conduct a clinical trial or investigation, such as for rare or life-threatening diseases, or for situations where recruitment or financial costs prevent the ability to run a clinical trial . Moreover, RWE can provide information for HTA beyond that which clinical trials or investigations can provide. The strict and controlled environment in which clinical trials are conducted means that data obtained likely does not reflect the real-world use of the device. RWE, in theory, is thus more applicable to HTA agencies for reimbursement decisions, but in practice, is limited in use by potential bias introduced with the lack of experimental controls . However, the recent guidelines on RWE and the use of DHTs may help to reduce potential bias in RWE collection, translating the use of RWE from theory to practice.
The need for sufficient clinical evidence (particularly RWE) to support the performance and safety of a medical device is at the forefront of medical device regulation. Recent guidelines and regulations on the collection of RWE for use in regulatory decision-making processes, alongside advances in DHT for clinical data collection, opens the door for the wider implementation of DHTs in clinical research, which in turn will aid with the often challenging process of collecting clinical data (See Figure 8 for an overview of the key elements of this review). Although challenges and concerns remain regarding the use of DHT, it presents a promising opportunity for providing meaningful and patient-centred data to aid with regulatory decisions and to aid with the process of obtaining sufficient clinical evidence.
1The International Medical Device Regulators Forum (IMDRF) is an international voluntary group of medical devices regulators that aim to accelerate harmonisation and convergence of medical device regulation. They create internationally agreed upon documents that cover a broad range of medical device topics.
2An additional 12-months can be given to the manufacturer to conduct a comparative study if they cannot provide sufficient evidence of the positive healthcare effect.
Jon I. Bergsteinsson is an employee at SMART-TRIAL.
All other authors declare no competing interests.
Union, T.E.P.A.T.C.O.T.E. Regulation (Eu) 2017/745 Of The European Parliament And Of The Council; 2017.
FDA. Study Data for Submission to CDER and CBER. https://www.fda.gov/industry/study-data-standards-resources/study-data-submission-cder-and-cber; 2022.
Administration, T.U.S.D.o.H.a.H.S.F.a.D. E6(R2) Good Clinical Practice: Integrated Addendum to ICH E6(R1). 2018.
Wealth Sciences Library. Data Resources in the Health Sciences. 2021; Available from: https://guides.lib.uw.edu/hsl/data/findclin.
Saczynski JS, McManus DD, Goldberg RJ. Commonly used data-collection approaches in clinical research. Am J Med. 2013; 126(11): 946–50. DOI: https://doi.org/10.1016/j.amjmed.2013.04.016
Jansen AC, et al. Guidelines were developed for data collection from medical records for use in retrospective analyses. J Clin Epidemiol. 2005; 58(3): 269–74. DOI: https://doi.org/10.1016/j.jclinepi.2004.07.006
Unim B, et al. Health data collection methods and procedures across EU member states: findings from the InfAct Joint Action on health information. Arch Public Health. 2022; 80(1): 17. DOI: https://doi.org/10.1186/s13690-021-00780-4
Ahalt SC, et al. Clinical Data: Sources and Types, Regulatory Constraints, Applications. Clin Transl Sci. 2019; 12(4): 329–333. DOI: https://doi.org/10.1111/cts.12638
MDCG. MDCG 2020–6 Regulation (EU) 2017/745: Clinical evidence needed for medical devices previously CE marked under Directives 93/42/EEC or 90/385/EEC; 2020.
ClinSkill. Clinical Data Collection Methods. 2018; Available from: https://www.clinskill.com/clinical-data-collection-methods/.
EUPATI. Data Collection in Clinical Trials. 2022; Available from: https://toolbox.eupati.eu/resources/data-collection-in-clinical-trials/.
European Union. Clinical trials on medicinal products for human use; 2014.
World Health Organization. WHO good manufacturing practices for pharmaceutical products: main principles, Annex3; 2011.
Krishnankutty B, et al. Data management in clinical research: An overview. Indian journal of pharmacology. 2012; 44(2): 168–172. DOI: https://doi.org/10.4103/0253-7613.93842
Kessel KA, Combs SE. Review of Developments in Electronic, Clinical Data Collection, and Documentation Systems over the Last Decade – Are We Ready for Big Data in Routine Health Care? Front Oncol. 2016; 6: 75. DOI: https://doi.org/10.3389/fonc.2016.00075
Sertkaya A, et al. Key cost drivers of pharmaceutical clinical trials in the United States. Clinical Trials. 2016; 13(2): 117–126. DOI: https://doi.org/10.1177/1740774515625964
Lambert PC, et al. Estimating the cost-effectiveness of an intervention in a clinical trial when partial cost information is available: a Bayesian approach. Health Econ. 2008; 17(1): 67–81. DOI: https://doi.org/10.1002/hec.1243
Adams CP, Brantner VV. Estimating the cost of new drug development: is it really 802 million dollars? Health Aff (Millwood). 2006; 25(2): 420–8. DOI: https://doi.org/10.1377/hlthaff.25.2.420
Hahn S, Whitehead A. An illustration of the modelling of cost and efficacy data from a clinical trial. Stat Med. 2003; 22(6): 1009–24. DOI: https://doi.org/10.1002/sim.1459
Kermani F, Narayan-Dubois C. Thinking ahead for effective clinical trials. Bioentrepreneur; 2005.
06 MDCG 2020–6 Guidance on Sufficient Clinical Evidence for Legacy Devices.
MDCG. MDCG 2019–11 Guidance on Qualification and Classification of Software in Regulation (EU) 2017/745; 2019.
FDA. Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices; 2017.
FDA. A History of Medical Device Regulation & Oversight in the United States. 2019; Available from: https://www.fda.gov/medical-devices/overview-device-regulation/history-medical-device-regulation-oversight-united-states.
Darrow JJ, Avorn J, Kesselheim AS. FDA regulation and approval of medical devices: 1976–2020. JAMA. 2021; 326(5): 420–432. DOI: https://doi.org/10.1001/jama.2021.11171
FDA. The 510(k) Program: Evaluating Substantial Equivalence in Premarket Notifications 510(k); 2014.
FDA. Postmarket Surveillance Under Section 522 of the Federal Food, Drug, and Cosmetic Act; 2016.
FDA. Examples of Real-World Evidence (RWE) Used in Medical Device Regulatory Decisions; 2022.
FDA. Digital Health Technologies for Remote Data Acquisition in Clinical Investigations, Draft Guidance; 2021.
Organisation WH. Classification of Digital Health Interventions v 1.0. 2018; Available from: https://apps.who.int/iris/bitstream/handle/10665/260480/WHO-RHR-18.06-eng.pdf#:~:text=The%20classification%20of%20digital%20health%20interventions%20%28DHIs%29%20categorizes,the%20digital%20technology%20to%20achieve%20health%20sector%20objectives.
Bakker JP, et al. A systematic review of feasibility studies promoting the use of mobile technologies in clinical research. NPJ digital medicine. 2019; 2(1): 1–7. DOI: https://doi.org/10.1038/s41746-019-0125-x
Santo K, Redfern J. Digital Health Innovations to Improve Cardiovascular Disease Care. Current Atherosclerosis Reports. 2020; 22(12): 71. DOI: https://doi.org/10.1007/s11883-020-00889-x
Islam SMS, Maddison R. Digital health approaches for cardiovascular diseases prevention and management: lessons from preliminary studies. mHealth. 2021; 7: 41–41. DOI: https://doi.org/10.21037/mHealth-2020-6
SMART-TRIAL. Clinical Trials on AI-Assessed QTc Measurement Directly from Consumer Smartwatches. May 2022; Available from: https://www.smart-trial.com/cases/clinical-trial-case-study-cardiologs.
Lu Z, Su J. Clinical data management: Current status, challenges, and future directions from industry perspectives. Open Access Journal of Clinical Trials. 2010; 2: 93–105. DOI: https://doi.org/10.2147/OAJCT.S8172
Commission E. Communication From The Commission To The European Parliament, The Council, The European Economic And Social Committee And The Committee Of The Regions; 2018.
Services, U.S.D.o.H.H. HITECH Act Enforcement Interim Final Rule; 2009.
Sharma A, et al. Using Digital Health Technology to Better Generate Evidence and Deliver Evidence-Based Care. Journal of the American College of Cardiology. 2018; 71(23): 2680–2690. DOI: https://doi.org/10.1016/j.jacc.2018.03.523
Gelinas L, et al. Using Social Media as a Research Recruitment Tool: Ethical Issues and Recommendations. The American journal of bioethics: AJOB. 2017; 17(3): 3–14. DOI: https://doi.org/10.1080/15265161.2016.1276644
Marquis-Gravel G, et al. Technology-Enabled Clinical Trials. Circulation. 2019; 140(17): 1426–1436. DOI: https://doi.org/10.1161/CIRCULATIONAHA.119.040798
Perry B, et al. Patient preferences for using mobile technologies in clinical trials. Contemporary Clinical Trials Communications. 2019; 15: 100399. DOI: https://doi.org/10.1016/j.conctc.2019.100399
Borno HT, et al. At What Cost to Clinical Trial Enrollment? A Retrospective Study of Patient Travel Burden in Cancer Clinical Trials. The Oncologist. 2018; 23(10): 1242–1249. DOI: https://doi.org/10.1634/theoncologist.2017-0628
Redfield MM, et al. Isosorbide Mononitrate in Heart Failure with Preserved Ejection Fraction. The New England journal of medicine. 2015; 373(24): 2314–2324. DOI: https://doi.org/10.1056/NEJMoa1510774
Facile R, et al. Use of Clinical Data Interchange Standards Consortium (CDISC) Standards for Real-world Data: Expert Perspectives From a Qualitative Delphi Survey. JMIR medical informatics. 2022; 10(1): e30363. DOI: https://doi.org/10.2196/30363
CTTI. Framework of Specifications to Consider During Mobile Technology Selection. 2021; Available from: https://ctti-clinicaltrials.org/wp-content/uploads/2021/06/CTTI_Digital_Health_Technologies_Technology_Selection_-Framework.pdf.
Dashiell-Aje E, Kovacs S, Sacks L. Regulatory Perspective: Digital Health Technology Tools Use in Clinical Investigations to Evaluate Clinical Benefit in Patients; 2019.
McKenna KC, et al. Investigator Experiences Using Mobile Technologies in Clinical Research: Qualitative Descriptive Study. JMIR mHealth and uHealth. 2021; 9(2): e19242. DOI: https://doi.org/10.2196/19242
Mahajan S, et al. Trends and predictors of use of digital health technology in the United States. The American journal of medicine. 2021; 134(1): 129–134. DOI: https://doi.org/10.1016/j.amjmed.2020.06.033
Taylor KI, et al. Outcome measures based on digital health technology sensor data: data-and patient-centric approaches. NPJ digital medicine. 2020; 3(1): 1–8. DOI: https://doi.org/10.1038/s41746-020-0305-8
Society DM. The Playbook – Digital Clinical Measures. July 2022; Available from: https://playbook.dimesociety.org/.
FDA. Guidance for Industry. Computerized Systems Used in Clinical Investigations; 2007.
FDA. Use of Electronic Records and Electronic Signatures in Clinical Investigations Under 21 CFR Part 11 – Questions and Answers; 2017.
FDA. Guidance for Industry. Electronic Source Data in Clinical Investigations; 2013.
Stern AD, et al. Advancing digital health applications: priorities for innovation in real-world evidence generation. The Lancet Digital Health. 2022; 4(3): e200–e206. DOI: https://doi.org/10.1016/S2589-7500(21)00292-2
Parliament E. eHealth – Technology for health, in Briefing; 2015.
EFPIA. EFPIA Reflection Paper on the Need for Better Defined Regulatory Pathways in the EU for Digital Health Technologies used concomitantly with Medicinal Products or as drug development tools during Clinical Development; 2021.
Commission E. 15 April 2022; Available from: https://ec.europa.eu/health/ehealth-digital-health-and-care/european-health-data-space_en.
BfArM. The Fast-Track Process for Digital Health Applications (DiGA) according to Section 139e SGB V.
NICE. Evidence standards framework for digital health technologies; 2018.
Lovell T. France to enable rapid market access for digital therapeutics. 2021; Available from: https://www.healthcareitnews.com/news/emea/france-enable-rapid-market-access-digital-therapeutics.
Kanta. Wht are the Kanta Services? Available from: https://www.kanta.fi/en/what-are-kanta-services.
Regulation P. Regulation (EU) 2016/679 of the European Parliament and of the Council. Regulation (eu). 2016; 679
GDPR.EU. What is GDPR, the EU’s new data protection law? May 2022; Available from: https://gdpr.eu/what-is-gdpr/.
CDC. Health Insurance Portability and Accountability Act of 1996 (HIPAA). [May 2022] Available from: https://www.cdc.gov/phlp/publications/topic/hipaa.html.
HHS. Summary of the HIPAA Privacy Rule. [May 2022] Available from: https://www.hhs.gov/hipaa/for-professionals/privacy/laws-regulations/index.html.
The Security Rule. HHS [May 2022] Available from: https://www.hhs.gov/hipaa/for-professionals/security/index.html.
Kolasa K, et al. Future of Data Analytics in the Era of the General Data Protection Regulation in Europe. PharmacoEconomics. 2020; 38(10): 1021–1029. DOI: https://doi.org/10.1007/s40273-020-00927-1
Lou J, et al. Real-world data for health technology assessment for reimbursement decisions in Asia: current landscape and a way forward. International journal of technology assessment in health care. 2020; 36(5): 474–480. DOI: https://doi.org/10.1017/S0266462320000628
Rudrapatna VA, Butte AJ. Opportunities and challenges in using real-world data for health care. The Journal of Clinical Investigation. 2020; 130(2): 565–574. DOI: https://doi.org/10.1172/JCI129197
Simpson A, Ramagopalan SV. R WE ready for reimbursement? A round up of developments in real-world evidence relating to health technology assessment. Journal of Comparative Effectiveness Research. 2021; 10(10): 797–799. DOI: https://doi.org/10.2217/cer-2021-0098
Tripodi S, et al. Digital technologies for an improved management of respiratory allergic diseases: 10 years of clinical studies using an online platform for patients and physicians. Italian Journal of Pediatrics. 2020; 46(1): 105. DOI: https://doi.org/10.1186/s13052-020-00870-z
Kothgassner OD, Felnhofer A. Lack of research on efficacy of virtual reality exposure therapy (VRET) for anxiety disorders in children and adolescents: A systematic review. Neuropsychiatrie: Klinik, Diagnostik, Therapie und Rehabilitation: Organ der Gesellschaft Osterreichischer Nervenarzte und Psychiater. 2021; 35(2): 68–75. DOI: https://doi.org/10.1007/s40211-020-00349-7
RADAR-CNS. Remote Assessment of Disease and Relapse in Central Nervous System Disorders. [July 2022]; Available from: https://www.radar-cns.org/about.
Ding H, et al. Digital health for COPD care: the current state of play. Journal of thoracic disease. 2019; 11(Suppl 17): S2210–S2220. DOI: https://doi.org/10.21037/jtd.2019.10.17
Klonoff AN, et al. Six Digital Health Technologies That Will Transform Diabetes. Journal of diabetes science and technology; 2021. DOI: https://doi.org/10.1177/19322968211043498
Yin AL, et al. Digital Health Apps in the Clinical Care of Inflammatory Bowel Disease: Scoping Review. Journal of medical Internet research. 2019; 21(8): e14630–e14630. DOI: https://doi.org/10.2196/14630
Lew SQ, Sikka N. Telehealth and Kidney Disease Care. Role after the Public Health Emergency? 2021; 16(12): 1784–1786. DOI: https://doi.org/10.2215/CJN.13651021
Sacks L, Kunkoski E. Digital Health Technology to Measure Drug Efficacy in Clinical Trials for Parkinson’s Disease: A Regulatory Perspective. Journal of Parkinson’s Disease. 2021; 11(s1): S111–S115. DOI: https://doi.org/10.3233/JPD-202416
Li WH, Chung JO, Ho EK. The effectiveness of therapeutic play, using virtual reality computer games, in promoting the psychological well-being of children hospitalised with cancer. Journal of Clinical Nursing. 2011; 20(15–16): 2135–2143. DOI: https://doi.org/10.1111/j.1365-2702.2011.03733.x
Bini SA, et al. Digital Orthopaedics: A Glimpse Into the Future in the Midst of a Pandemic. The Journal of arthroplasty. 2020; 35(7S): S68–S73. DOI: https://doi.org/10.1016/j.arth.2020.04.048
Lezcano-Valverde JM, et al. Development and validation of a multivariate predictive model for rheumatoid arthritis mortality using a machine learning approach. Scientific Reports. 2017; 7(1): 10189. DOI: https://doi.org/10.1038/s41598-017-10558-w