Evaluating the Totality of the Evidence: Why Real-World Data Matters forMedical Devices

By: Dr. Steven Farmer - Senior Partner, ABIG Health

The term “totality of evidence” refers to the concept that all available information and evidence related to a particular matter should be considered. A single, or even a few pieces of evidence are not enough.

When determining the safety, efficacy, and cost effectiveness of a medical device, totality of evidence is essential. While a medical device may be universally safe, it may not work for certain patients or in certain settings.

Premarket trials, which progress from initial development and testing to lab and animal testing and finally human testing, often do not represent these real-world patients and usage scenarios. Real-World Data (RWD) studies are necessary to demonstrate to clinicians, patients, and payers that a product will work for certain populations and merit coverage. As such, RWD studies are a valuable complement to — though not a replacement for — traditional clinical trials.

What Are the Limitations to Premarket Clinical Trials

While rigorous and essential (I do not mean to call into question either the need for or value of premarket trials), device developers and manufacturers are leaving a lot of valuable, and potentially life-altering, information on the table if they stop with these trials.

Why?

Premarket studies often rely on small samples, make evaluations based on short durations, and use surrogate or mechanistic endpoints to make assessments. They also often exclude older adults, women, racial and ethnic minorities, and people with disabilities and comorbidities. In other words: they do not reflect the world’s population as it actually exists. These limitations restrict the generalizability of findings and may overlook differences in device performance or safety across subgroups.

Despite long-standing efforts to improve representation, lack of diversity is still a problem. And it harms Americans.  As physicians at the University of Southern California’s Leonard D. Schaeffer Institute for Public Policy and Government Service wrote in 2022, “The lack of diversity in clinical trials unfairly skews American medicine and costs billions of dollars a year in early deaths and poor health.”

The Value of Real-World Data in Device Evaluation

Real-world data (RWD), as defined by the U.S. Food and Drug Administration (FDA), is information related to patient health status or the delivery of healthcare that is collected outside a controlled clinical trial setting. More simply, it is data that reflects how a device is used in real life, whether that is an operating room in an urban center or at a small academic institution in a rural area.

Real-World Data can be collected from all sorts of sources, including electronic health records, insurance claims, healthcare registries, and even wearables like a smartwatch.

Medical device developers and manufacturers should embrace RWD because it:

• Enables large-scale analyses, facilitating subgroup evaluations;

• Makes it economically feasible to study more diverse populations across broader geographies and healthcare settings; and

• Captures longer-term outcomes and usage patterns.

Real-World Data studies also help identify variations in outcomes based on patient characteristics like age, sex, race and ethnicity; in different care settings (whether something works only in an academic setting or whether it is feasible in a rural emergency room); and provider experience of institutional volume.

RWD Versus RCT: A Complement, Not a Competition

Once more, clinical trials are essential. I am not advising that they be replaced by RWD collection. Rather, they should be complemented by it.

That is because randomized control trials (RCTs) offer internal validity and controlled conditions even if they lack generalizability. One the other hand, RWD offers external validity and real-world relevance, but may lack certain controls or data precision. Additionally, RCTs can define efficacy under ideal conditions while RWD can assess effectiveness across varied, real-life conditions. They work together, not in silos.

Device developers and manufacturers must take and “both-and” not an “either-or” approach to evidence gathering.

But they are not the only ones who need to reform their thinking. Regulators, payers, and clinicians to adopt a more holistic view of evidence that integrates both RCT and RWD.

Using Real-World Data to Inform Equity and Access

Improving access and equity depends on understanding how devices work for everyone, not just the limited populations that are traditionally included in traditional trials.

As alluded to above, incorporating RWD offers the healthcare community the chance to improve equity and access. By incorporating RWD, we can close data gaps for historically excluded populations like women, racial and ethnic minorities, and people living with disability. RWD allows for proactive identification of differential outcomes or safety signals and it supports more inclusive, patient-centered innovation and policy decisions.

Federal regulators and government and private payers are starting to change their way thinking by embracing RWD. By gathering RWD, device developers and manufacturers also can ensure lifesaving interventions are affordable to the populations that need them.

No single type of evidence is sufficient on its own; RWD is an essential piece of the total product lifecycle, especially for devices where there is a wide range of efficacy based on a patient’s own characteristics and medical history, and based on the care setting. Developers and manufacturers should more broadly use RWD to complement premarket studies, improve representation, and evaluate performance in real-world conditions.

The future of device evaluation must be inclusive, comprehensive, and grounded in “totality of evidence.”