THE PROBLEM

Randomized Controlled Trials (RCTs)—a methodological example of the frequentist approach— dominate the hierarchy of evidence generation due to their stable, systematic approach to clinical experimentation. However, over reliance on one methodology in the treatment development system can result in impractical, costly, and inflexible trials.

There are severe system-wide impacts of generating evidence using predominantly one methodology: today, our RCT-dominated treatment development system is consistent, but inflexible to the evolving needs of human biology. Further, if one methodology dominates evidence generation, other more relevant techniques are less utilized or underdeveloped.

IMPRACTICAL

Due to restrictive eligibility criteria and lack of external validity, therapies derived from RCTs don’t function in real world as they do in controlled environments. This methodological impasse is so striking that there is a lack of evidence in what works for whom.

COSTLY

RCTs are time and resource intensive. Statistical power must be achieved through larger sample size or repeated, costly tests and procedures.

INFLEXIBLE

Our RCT-reliant system is unable to adapt to small sample sizes, which creates methodological problems for rare diseases and new-omics discoveries, as medicine shifts towards personalized care.

AS DESCRIBED BY ANGUS DEATON1,
A RECENT NOBEL PRIZE WINNER IN ECONOMICS:

“Each study has to be considered on its own. RCTs are fine, but they are just one of the techniques in the armory that one would use to try to discover things. Gold standard thinking is magical thinking.”

VISION

Medicine needs more tools in its toolbox. Twinning the medical and mathematical communities— clinical trialists and pure mathematicians — will help practitioners within the treatment development system not only imagine ways of better matching treatments to patients, but also of responding to heavy volumes of clinical decisions seen in the world today.

The flexibility of new techniques will bring their own advantages to the regulatory approval process, by being better suited to regulatory decision-making than RCTs. Having a flexible and robust regulatory approval process built on strong evidence generation methods will lead to better outcomes for the general population.

SOLUTIONS

Building a methodological toolbox for the treatment development system will require two fundamental shifts.

In the short-term, the medical community must find and make the most of existing — but less used — evidence generation techniques. These include familiar tools such as centralized registries, case studies, and case series, but also less familiar approaches such as causal inference (an active topic in statistics, but often overlooked in mathematics). By bringing together mathematicians, statisticians, and medical researchers we can determine the viability and application of existing methods and apply

finding to spaces where RCTs are impractical, costly, and inflexible.

In the long-term, disrupting the current way of doing things will necessitate generation and regeneration of new and different evidence generation methodologies, complete with new depths and breadths of data, specifically necessary for clinical research. We must not only search for new technologies, but embolden medicine to continuously develop its own new methods for evidence generation.

BUILDING

Building codes have been highly dependent on experiments, but this traditional approach was difficult to apply to natural disasters (high impact, low probability, non-Gaussian events)

Engineers added a tool to their toolbox called fuzzy set theory to model low probability events.

ENERGY

When renewable energy started penetrating the electricity grid, engineers were not able to use approximated models of direct current flows to see how consumers would sell electricity back to the grid.

Nonlinear optimization was repurposed and revolutionized to meet the needs of a 21st century electric grid.

OMICS

The explosion of research in genomics, epigenomics, transcriptomics, and other -omics presents researchers with a variety of challenges, including how to integrate and analyze data from multiple -omics in order to capture the complexity of biological systems.

A variety of new multi-omics methodologies have been created to meet this ongoing challenge.

CONCLUSION
Modern evidence generation need not be a one- method show. Moving to a treatment development system that employs multiple evidence generation techniques will require both the use of existing forms of evidence generation, but also an appetite for continually reimagining new ones. Such a regenerating methodological ‘toolbox’ will result in approaches that are both automatically relevant to the world of clinical research and also ultimately flexible to change.

CITATIONS

  1. Ogden,T.N.(Ed.).(2017).Experimental Conversations: Perspectives on Randomized Trials in Development Economics. MIT Press.

Co-Research Leads:

Sauleh Siddiqui
siddiqui@jhu.edu
Jen Bernstein
jberns12@jhu.edu

@JHUctsp