Pipeline Physics

Pipeline Physics produces profit
Gary Summers, PhD	1700 University Blvd, #936
President, Pipeline Physics LLC	Round Rock, TX 78665-8016
gary.summers@PipelinePhysics.com	503-332-4095

---

Clinical Trial Error Rates

What are some big costs in drug development? How about:

• Advancing unsafe and ineffective compounds, so they become expensive downstream failures
• The opportunities lost by mistakenly canceling safe and effective compounds

To minimize these costs, statisticians design clinical trials to achieve target false-positive and false-negative rates, with common targets for phase III being a false-postive rate of 2.5% (alpha, one-tailed test) and a false-negative rate of 10% (1 - power).

Do trials achieve these targets? From historical data, industry-wide, phase III attrition is 40%, and 54% of these compounds fail for showing insufficient efficacy. Thus, 22% of phase III compounds fail for lacking efficacy, or equivalently, if executives advanced or canceled compounds based solely on efficacy, 78% of phase III compounds would advance to the NDA. Meanwhile, about 17% of NDA applications fail, of which 48% fail for insufficient efficacy. Thus, 8% fail for lacking efficacy, or equivalently, 92% of NDA compounds are effective.

Multiplying the above percentages, if drugs advanced or failed based on efficacy only, 72% of phase III compounds would advance to the NDA and be approved, 6% would advance and be rejected, and 22% would not advance. Table 1 presents these data.

Table 1: Phase III's efficacy-only success and failure rates.

		NDA Result
		Approve	Reject
Phase III results (efficacy only)	Success	72%	6%
	Failure	22%

Unfortunately, we do not know whether the canceled compounds were effective or ineffective, but even without this knowledge, we can calculate phase III's minimum false-positive rate. The minimum false-positive rate occurs if all the canceled compounds were ineffective, which yields a false-positive rate of 23% = 6% / (6% + 22%). The minimum false-positive rate for phase III nearly ten times the rate that clinical trials were designed to achieve.

Possibly, phase III trials produce higher false-positive and false-negative rates then their designs predict, making development more expensive and reducing productivity. Phase II trials could suffer more since they face greater uncertainty.

The above analysis calculates the minimum false-positive rate. What are the actual false-positive and false-negative rates? What fraction of a phase's compounds are effective? Using a data science technique, you can estimate these qualities for each therapeutic area, large v. small compounds, NMEs v. nonNMEs, and trials managed by pharma companies v. those managed by CROs. With this data, you can make trials more effective and development more productive.

* The historical data is from: Hay et al. (2014), "Clinical development success rates for investigational drugs," Nature Biotechnology, 32(1): 40-51.