What CMOs Get Wrong about Statistics and Trial Design

Randall Moreadith, MD, PhD, is the CEO of Serina Therapeutics. 

What do CMOs get wrong about biostatistics and trial design?

My experience has taught me, over the 25 years that I've been doing this, that the sooner you engage with a really good biostatistician to help you develop your drugs, the better. Bringing them in late can be a challenge. I’ve been very fortunate in my career to work with people who are not only really good at biostatistics, but are also great friends. I don't typically go to a CRO to get that expertise now because I can reach out to those friends. I think there are a lot of companies and a lot of CMOs who do that because most biotech companies cannot afford to have a biostatistics group as part of their headcount, nor do they need it. But you need biostatisticians you can call on, particularly if you are approaching  the FDA with a trial design that requires input from a biostatistician. I would never imagine going to the FDA with a phase two study without a biostatistician.

How can someone who doesn't necessarily have that statistical expertise evaluate who would be considered a good external statistical consultant or CRO with that capability?

There are a lot of CROs that offer this expertise as part of the package of services for your clinical studies. You often can get that expertise just by being associated with the CRO that is going to run or help you design your clinical study. Years ago, I was running the clinical operation at a gene therapy company. We had licensed in the technology from a company where the two phase three trials had failed. We were convinced from the animal data that there was a real signal there but that they were unable to see that signal in their studies. We hired an independent statistical group to help us do the post-mortem on those phase three studies and what we found was quite astonishing. It turned out that there was a rather robust treatment effect, but it only occurred in women.

There may be times in your career where you are going to do the post-mortem analysis on a clinical trial, and you wonder why you hit it or why you didn't hit it. Those are two very important questions that you need to answer. How is it that you actually achieve statistical significance and possibly even clinical significance? And if you missed it, why did you miss it? There are a lot of statistical CROs that specialize in that industry, and you can hire just that expertise. You don't have to hire a large CRO to do that. You should call your colleagues in the industry, particularly folks who attend the CMO Summit, and ask who they work with when they need statistical input on an issue. Then you begin to develop a relationship with those folks. These are people that you can trust to go with you to the FDA. When you are looking at those groups, you are interviewing them as somebody who you are likely going to take with you to the FDA at some point in the future. You develop a relationship with them and rely on that expertise.

In that interview process, are there specific questions you should be asking?

My first question to them is, “How many times have you been in front of the FDA?” And if they've never been in front of the FDA, that is a cautionary note. They may be really good at statistics, but if they haven't helped a biotech prepare a background package and a presentation to the FDA, they may not be the best CRO for you to work with. That is ultimately what you are anticipating doing. You have an idea, you have a trial design in mind, you are going to hire a group, and you want that group to have had that experience being in front of the FDA. It’s invaluable. 

When do you switch from outsourcing to insourcing?

So far, I haven't worked at a company that had an in-house statistical expert. We did have a statistician at one of the companies where I was CDO, but that person came in late to help us interpret one of the studies and only then did that person become a full-time report directly to the CDO. If you go to any of the big pharma companies, they have this expertise in-house. Most biotech companies are not going to need in-house expertise for this. It is too expensive, and frankly, a lot of the time, they will just be sitting around twiddling their thumbs, wondering what to do next. If you don't have a lot of clinical trials at various stages of development, you certainly don't need to invest in that headcount. There are plenty of CROs that can provide that expertise.

Are there any quirks that people should be thinking about for clinical trial design?

The big transition for a lot of biotech companies is that transition from phase one to phase two. In particular, if your phase two trial is going to be a randomized control trial, it will need a control arm. In many instances, you know what the approvable endpoint is for the disease that you are going after. We are at the stage at Serina where we are thinking about going into phase one in the population of patients we ultimately want to study. We are hopeful we will see a signal there and can immediately interpret that data to help us design the phase two study. But it is rare in many instances to go directly into the population that you intend to study in phase one. The take home message is: If the FDA is okay with you working in the intended patient population in phase one, that is really helpful, because you are going to get data for safety and tolerability, as well as pharmacokinetics and pharmacodynamics, which will help for your phase two trial.

When it gets to the phase two study, that is probably the time to bring in a statistician and subject matter experts and ask, “What treatment effect are we looking for here so that we can adequately power this study and detect that treatment effect versus the control arm?” Fortunately for us, there is a wealth of data on phase two studies in Parkinson's disease that can help us design and power that study. If you are going after a rare indication, that might not be the case. You are probably going to have to be a lot more creative about your phase two trial design. In those instances, I would simply go to the FDA and request information. Show them your phase one data and request an end-of-phase-one meeting and ask them to advise you on what to do.

“Don’t cut the budget and cut the number of patients because you will ultimately pay a price for that. Power the study adequately, even if you have to power it at a statistical power that you wouldn’t normally power it at.” 

How do you balance the speed, cost and strength of a trial?

If you are a small biotech company with limited resources, your CEO might be telling you to have the minimal number of patients that will give you a clear endpoint. The biostatistician will issue a very cautionary note about that. How do you know that six patients is going to be enough? I have been pushed by CEOs who wanted to do that and I have issued a very cautionary note on that. You are setting yourself up for the potential to fail if you underpower your study, especially for a phase two study, which is an important readout for any biotech company. 

I've learned my lesson there. Don’t cut the budget and cut the number of patients because you will ultimately pay a price for that. Power the study adequately, even if you have to power it at a statistical power that you wouldn't normally power it at. Rely on your biostatistician to help you.

How can CMOs effectively communicate with statisticians?

Any CMO needs to be reasonably well versed in statistics going into any conversation with a biostatistician because they will throw language at you that you probably haven't heard before. Statisticians do speak with a particular nuance about their ideas about trial design. I hear a lot of CMOs say, “but the p-value was 0.03,” and try to convince you that the drug works based on that p-value. Your biostatistician is not going to like that comment because they understand the p-value is simply a number that tells you how likely it is your observation could have occurred under the null hypothesis. It does not mean the drug works. In fact, your clinical readout might be clinically meaningless according to your key opinion leaders, and yet your p-value can suggest statistical significance. I can assure you, if you stay in this business long enough as a CMO, you are going to  have the experience where you hit it in the first phase three trial, and you are reasonably convinced that it was adequately powered and the treatment effect is real, and completely miss it in the next phase three. It's not a particularly good experience because it's obvious the first phase three was a statistical fluke.

How should CMOs think about integrating interim analyses into their trials?

This question usually comes up in the middle of your initial large phase three trial and is a responsibility delegated to your Data Safety and Monitoring Board. Are they looking for a safety signal because they are concerned that the drug they are investigating might not be safe? That is different than looking at the efficacy. If they want to determine efficacy through an early readout, they might suggest a futility analysis. If it is likely that your trial is never going to reach an efficacious endpoint, you need to build an analysis plan that will look into that. 

In many instances, you are going to  have to discuss this with the FDA, who will help determine if you are going to have to pay a statistical penalty for taking that look. But these are pretty well vetted charters. Your biostatistician and the head of your DSMB will help you understand whether an interim analysis is advisable. This is something you need to discuss with the FDA at your end-of-phase two meeting.

How did your experience as CMO prepare you for the CEO role?

Being a CMO uniquely prepared me for the role of being a CEO. If you are running a biotech company, you are likely involved in running clinical trials that are developing drugs. It is really important to have that background to rely on, particularly if you are also acting as CMO at that company. I've learned that being a CEO of a small company really stands for “Chief Everything Officer.” On any given day, you have to wear different hats. Whether it's business development or finance or strategy or medical officer, having that broad based experience is critically important to a small biotech company.

There are a lot of other responsibilities that come with being a CEO. You now have responsibility not only for making certain that the company is going to conduct its science and clinical trials properly. You've got to make sure the company is financed. You've got to make sure that you can retain outstanding employees. You've got to be able to set a vision for where the company is going to be in one year and five years. You need a lot of experience to step into that role, and I don’t believe being a CMO is a necessary prerequisite for becoming a CEO

Anything else?

This is something CMOs think a lot about. They want to make sure their trials are going to succeed. They want to make certain that they understand what the likelihood of success is going to be once they pull the trigger on what is likely going to be a very expensive study. My experience has taught me to bring in statistical expertise sooner rather than later. You need someone who you can trust, and you need that person to go to the FDA with you. Establishing that rapport with someone that you trust and who also trusts you as a company is critically important.