Realizations in Biostatistics

MOOCs–a low-risk way to explore outside your field

noreply@blogger.com (John Johnson) — Mon, 29 Apr 2013 03:21:00 +0000

One of the things I'm realizing from Massively Open Online Courses (MOOCs) -- those online free classes from universities that have seem to sprung up from almost nowhere in the last year and a half -- is that they offer a perfect opportunity to explore outside my field. At first (and this was even before the term MOOC was coined), I took classes there were just outside my field. For instance, I've been in clinical and postmarketing pharmaceutical statistics for over 10 years, and my first two classes were in databases and machine learning. I did this because I was aching to learn something new, but I figured that with a class in databases I could make our database guys in IT sweat a bit just by dropping some terms and showing some understanding of the basics. It worked. In addition, I wanted to understand what this machine learning field was all about, and how it was different from statistics. I accomplished that goal, too.

Since then, I have taken courses in the area of artificial intelligence/machine learning, sociology and networks, scientific computing (separately from statistical computing), and even entrepreneurship. I have also encouraged others to take part in MOOCs, though I don't know the result of that. Finally, I have come back to some classes I've already taken as a community TA, or former student who actively takes part in discussions to help new students take the class.

This is all valuable experience, and I could write several blog entries on the benefits. The main one I'm feeling right now is the feeling that I'm coming up for air, and taking a sampling of other points of view in a low-risk way. For example, though I don't actively use Fourier analysis in my own work, one recent class and one current class both use it to do different things (solve differential equations and process signals). Because these classes involve programming assignments, I've now deepened my understanding of the spectral theorem, which I only studied from a theoretical point of view in graduate school. I'm also thinking about this work from the point of view of time series analysis, which is helping me think about some problems involving longitudinal data at work.

From a completely different standpoint, another class helped me think about salary negotiations in terms of expected payoff (i.e. combination of probability of an offer being accepted vs. salary). This sort of analysis invited further analysis of the value of that job vs. what I would be paid and the insecurity of moving to a different job. In the end, I turned down what would have been a pretty good offer, because I decided it did not compensate for the risks I was incurring. The cool thing is that these were all applying concepts I already understood (expected value, expected payoff), but applied in a different way from what I was already doing.

The best thing about MOOCs is that the risk is low. All that is required is an internet connection and a decent computer. Some math courses may require a better computer to do high-powered math, but I've seen few that require expensive textbooks or expensive software. Even Mathworks is now offering Matlab at student pricing to people who are taking some classes, and Octave remains a free option for people unable to take advantage of it. And, if you are unable to keep up the work, there is now downside. You can simply unenroll.

RStudio is reminding me of the older Macs

noreply@blogger.com (John Johnson) — Tue, 16 Apr 2013 02:33:00 +0000

The only thing missing is the cryptic ID number.

Well, the only bad thing is that I am trying to run a probabilistic graphical model on some real data, and having a crash like this will definitely slow things down.

Presenting without slides

noreply@blogger.com (John Johnson) — Sat, 30 Mar 2013 18:21:00 +0000

Tired of slides, I’ve been experimenting with different ways of presenting. At the recent Conference on Statistical Practice, I decided only to use slides for an outline and references. As it turns out, the most critical feedback I got had to do with the fact that the audience couldn’t follow the organization because I had no slides.

I tried presenting without slides because, well, I started to use them as a crutch. I also saw a lot of people presenting essentially by putting together slides and reading from them. So I figured I would expand my horizons.

Next time I present, I’ll do slides, I guess, but I may try something a bit different.

Last session of Caltech's Learning from Data course starts April 2

noreply@blogger.com (John Johnson) — Wed, 27 Mar 2013 20:49:00 +0000

I just received this email:

Caltech's Machine Learning MOOC is coming to an end this spring, with the final session starting on April 2. There will be no future sessions. The course has attracted more than 200,000 participants since its launch last year, and has gained wide acclaim. This is the last chance for anyone who wishes to take the course (http://work.caltech.edu/telecourse).
Best.
The Caltech Team

I strongly recommend this course if you can take it, even if you have taken other machine learning classes. It lays a great theoretical foundation for machine learning, sets it off nicely from classical statistics, and gives you some experience working with data as well.

If you were for some reason waiting for the right time, it looks to be now or never.

Review of Caltech's Learning from Data e-course

noreply@blogger.com (John Johnson) — Thu, 21 Mar 2013 02:47:00 +0000

Caltech has an online course Learning from Data, taught by Professor Yaser Abu-Mostafa, that seeks to make the course material accessible to everybody. Unlike most of the online courses I've taken, this one is independently offered through a platform created just for the class. I took the course for its second offering in Jan-March 2013.

The platform on which the course is offered isn't as slick as Coursera. The lectures are offered through a Youtube playlist, and the homeworks are graded through multiple choice. That's perhaps a weakness of the class, but somehow the course faculty made it work.

The class's content was its strong point. Abu-Mostafa weaved theory and pragmatic concerns throughout the class, and invited students to write code in just about any platform (I, of course, chose R) to explore the theoretical ideas in a practical setting. Between this class and Andrew Ng's Machine Learning class on the Coursera platform, a student will have a very strong foundation to apply these techniques to a real-world setting.

I have only one objection to the content, which came in the last lecture. In his description of Bayesian techniques, he claimed that in most circumstances you could only model a parameter with a delta function. This, of course, falls in line with the frequentist notion that you have a constant, but unknowable "state of nature." I felt this way for a long time, but don't really believe it any more in a variety of contexts. I think he played up the Bayesian v. frequentist squabble a bit much, which may have been appropriate 20 years ago but is not so much an issue now.

Otherwise, I found the perspective from the course extremely valuable, especially in the context of supervised learning.

If you plan on taking the course, I recommend leaving a lot of time for it or having a very strong statistical background.

Distrust of R

noreply@blogger.com (John Johnson) — Tue, 12 Mar 2013 23:12:00 +0000

I guess I've been living in a bubble for a bit, but apparently there are a lot of people who still mistrust R. I got asked this week why I used R (and, specifically, the package rpart) to generate classification and regression trees instead of SAS Enterprise Miner. Never mind the fact that rpart code has been around a very long time, and probably has been subject to more scrutiny than any other decision tree code. (And never mind the fact that I really don't like classification and regression trees in general because of their limitations.)

At any rate, if someone wants to pay the big bucks for me to use SAS Enterprise Miner just on their project, they can go right ahead. Otherwise, I have got a bit of convincing to do.

Bad statistics in high impact journals

noreply@blogger.com (John Johnson) — Fri, 01 Mar 2013 01:49:00 +0000

Better Journals… Worse Statistics? : Neuroskeptic

In the linked blog entry, Neuroskeptic notes that high impact journals often have fewer statistical details than other journals. The research reported in these journals is often heavily amended, if not outright contradicted, by later research. I don't think this is nefarious, though, nor is it worthless. The kind of work reported in Science and Nature, for instance, generates interest and, therefore, more scrutiny (funding, studies, theses, etc.).

But as with all other research, if statistical details are included it might direct subsequent research in these topics a bit better.

The burst of the Big Data bubble, and do we need the hype, anyway?

noreply@blogger.com (John Johnson) — Thu, 21 Feb 2013 01:34:00 +0000

So, now I'm seeing some buzz over Twitter that the Big Data disillusionment is starting now. Frankly, I've been wondering when this would happen. Of course, the next stage involves making strategic investments in Big Data resources, and having these resources quietly being used effectively, at least if Big Data follows technologies such as neural networks, Java, etc. So the theory goes, all surviving technologies follow a pattern of hype, disillusionment, and then quiet acceptance.

Did we really need this period of hype? I can understand companies hype up a technology to maintain interest while they try to make their offerings mature, and overhyping usually leads to disillusionment, but I wonder if there is a different path. R, Python, and some other open projects seem to have flattened the hype hill and disillusionment valley, probably because the larger number of people hacking the inside generates its own interest and maturity mechanism.

Anyway, I look forward to the maturing of big data at least until the privacy concerns generate widespread panic.

Sloppy journalism with interactive graphics is still sloppy journalism

noreply@blogger.com (John Johnson) — Fri, 15 Feb 2013 13:30:00 +0000

The Guardian recently discussed the "declining linguistic standards" in State of the Union addresses. I thought this was an interesting exercise, but something seemed wrong about the article, and it turns out this is one case where the data do not really speak for themselves. There's a lot of interpretation and understanding behind cultural trends in the use of the English language in America, as well as the evolution of the presidents' intentions behind the address. There are a few important points:

The author correctly points out that Woodrow Wilson essentially changed the format of the address through precedent from written document to speech. Right after Wilson's first speech there is a huge drop in the "education level" (hang on for a discussion of this terminology) of these addresses. As I recall, Wilson is the only American president with a Ph.D.
The index used - Flesch-Kincaid (FK), is questionable. Good on The Guardian to use a single measure for all speeches, but I have to wonder if it is wise to use the same measure for speeches and written addresses. Furthermore, FK is very sensitive to the placement of punctuation (it weights sentence length heavily). For instance, as a friend pointed out, one of Wilson's speeches has a FK grade level of over 17, but if you replace one of the semi-colons in the speech with a period, the FK grade drops to 12. This subtlety is lost in speech format, giving FK an extremely high uncertainty (this same friend calls FK "utterly useless" for speeches).
The audience of the SOTU address has changed. Though it's a constitutional duty of the president, the delivery as a speech is not, and it only has to be delivered to Congress. However, most modern addresses have been in the form of televised speeches, and have to be understood by a wider and less politically savvy audience.
Cultural trends in the use of spoken and written English in America involve shorter sentences over time in general.
In this case, a more sophisticated natural language processing analysis might reveal some interesting trends. For instance, how do wartime speeches compare to times of peace? Are there any natural categories of speeches that fall out? What are the outliers? How does this compare to polls?

In short, we have some interesting data that needs heavy qualification and critical analysis, that is just presented on a page and capped with a headline that gives an overly simplistic interpretation.

Operational details can be pesky

noreply@blogger.com (John Johnson) — Mon, 11 Feb 2013 22:49:00 +0000

Recently, I was working with a team to finalize a clinical trial protocol. I raised some concerns about their strategic matters, and my concerns were dismissed as "operational details."

The thing about those pesky operational details is that, if something doesn't work due to an operational detail, you might have to modify your strategy. And if enough of these pesky operational details get in the way, you may have to rethink your strategy.

MOOCs have exploded!

noreply@blogger.com (John Johnson) — Sun, 09 Dec 2012 15:15:00 +0000

About a year and two months ago, Stanford University taught three classes online: Intro to Databases, Machine Learning, and Artificial Intelligence. I took two of those classes (I did not feel I had time to take Artificial Intelligence), and found them very valuable. The success of those programs led to the development of at least two companies in a new area of online education: Coursera and Udacity. In the meantime, other efforts have been started (I’m thinking mainly edX, but there are others as well), and now many universities are scrambling to take advantage of either the framework of these companies or other platforms.

Put simply, if you have not already, then you need to make the time to do some of these classes. Education is the most important investment you can make in yourself, and at this point there are hundreds of free online university-level classes in everything from the arts to statistics. If ever you wanted to expand your horizons, now’s the time.

I’ve personally taken 7 online classes now, and earned certificates in all of them. I use the material in many of these classes in my work, and I even have used two (Machine Learning and Probabilistic Graphical Models) to expand my company’s capabilities. I am far more secure in my job because of what I’ve learned. In addition, I had the honor of trying out the Probabilistic Graphical Model Community TA program, and my only regret is that I couldn’t put more time into it. To the extent that I took advantage of it, I got a lot out of the experience.

Now, the hard part. These classes require self-discipline. Like universities, there are some duds as well. At least you can add and drop at will, not worrying about prerequisites. You have to take responsibility for your own education and your own motivation.

In all, I’m very grateful that there are these pioneers Andrew Ng, Daphne Koller, Sebastian Thrun, and others who saw this need and had the knowledge and motivation to fill it. They are now moving in the direction of accreditation, and both free and premium models (probably for some kind of licensing or degree, which I don’t care about right now). For now, you can sign up and take classes at will.

Happy MOOCing!

Rare things happen all the time

noreply@blogger.com (John Johnson) — Wed, 14 Nov 2012 17:25:00 +0000

John Cook reports on the probability of long runs. This is a very useful reality check.

I think there is a larger principle here, though, that rare things happen all the time.

Analysis of the statistics blogosphere

noreply@blogger.com (John Johnson) — Sun, 11 Nov 2012 05:16:00 +0000

My analysis of the statistics blogosphere for the Coursera Social Networking Analysis class is up. The Python code and the data are up at my github repository. Enjoy!

Included are most of the Python code I used to obtain blog content, some of my attempts to automate the building of the network (I ended up using a manual process in the end), and my analysis. I also included the data. (You can probably see some of your own content.)

Here's what I learned/got reminded of the most:

Doing projects like this is hard when you have other responsibilities, and you usually end up paring down your ambitions toward the end
Data collection and curation was, as usual, the most difficult process
Network analysis is fun, but I have a ways to go to know where to start first, what questions to ask, and so forth (these are the things you learn with experience)
The measures that seem to be the most revealing are not always obvious -- in this network, it was the number of shortest paths compared to a random graph
Andrew Gelman's blog is central (but you probably don't need a formal analysis to tell you that)
There's a lot of great content about statistics, data analysis, data science, and statistical computing out there. I've relied on blog posts for a lot of my work, and I've found even more great stuff. It's a firehose of information.

Snapshot of the statistics blogosphere

noreply@blogger.com (John Johnson) — Mon, 05 Nov 2012 13:45:00 +0000

This was generated during my social network analysis project. I haven’t finished yet, but I did want to show the cute picture. The statistics blogosphere is like a school of jellyfish.

Sometimes, saving CPU time is worth it for small data jobs

noreply@blogger.com (John Johnson) — Sun, 04 Nov 2012 20:37:00 +0000

There appears to be a conventional wisdom, one that I myself have espoused on several occasions, that for “most” statistical computing jobs that developer time is more precious than CPU time. (The reason I write “most” in quotes is that there are some people who work in environments where Big Data or large jobs is the norm, or they are developing high performance computing libraries, and they have to squeeze every last bit of performance out of the CPU.)

However, sometimes it can be worth it to save a few extra minutes small jobs, especially if they are run over and over. At one point today, I had an algorithm that I wrote inefficiently using Python’s built-in lists. I decided to stop the job and rewrite using the NumPy libraries, which took me an extra half hour. At first, I thought the time was wasted, but I have ended up running the code several times for various reasons. Those save minutes have now, a couple of hours later, saved me more time than I spent rewriting.

Politics vs. science and the Nate Silver controversy

noreply@blogger.com (John Johnson) — Fri, 02 Nov 2012 23:35:00 +0000

I’ll take a small departure from the narrow world of biostatistics and comment on a wider matter.

Nate Silver of FiveThirtyEight has really kicked the hornet’s nest. This is a nest that really needed stirring, but I do not envy him for being the focus of attention.

This all started, I think, when he released his book and basically called political pundits out for a business model of generating drama rather than making good predictions. This wouldn’t be a huge deal, except that he has developed a statistical model that combines data from national and state polls with demographic data to project outcomes of presidential and senatorial elections. This model, as of this writing, has President Obama at close to an 81% probability of re-election, given the current state of things. As it turns out, there are a lot of people that don’t like this, and they generally fall into two camps:

1. People who would rather see President Obama defeated in the election, and

2. Pundits who have a vested interest in a dramatic “horse-race” election

I’ll add a third:

3. Pundits who want to remain relevant (whether to keep their jobs or reputations).

Frankly, I don’t think that pundits will have to worry about #3. There’s an allergy to fact in this country, a large group of people who would rather ignore established fact and cling to a fantasy. (You can find a sampling of these people over at the intelligent design blogosphere, for instance.) I think the demand for compelling stories over dry facts will remain.

I’ve run into people of the first type, when I’ve published some armchair statistician analyses based on Twitter sentiment, for instance. The responses weren’t critiques of the method, but rather, “who cares, Republicans rule!” Even more dangerous, I’ve run into similar responses to negative clinical study results in cases where sponsors have a vested interest in positive outcomes. (There was at least one case I remember a sponsor moved forward with an expensive study to follow on, and some where I was asked to reanalyze a zillion times.)

Nate write The Signal and the Noise where he, among a lot of explanation, points out that there is a whole cottage industry of people getting paid to BS about politics. So I think that some in the second category are starting to face an existential crisis, and that makes them dangerous.

Ultimately, we have to understand where Nate is coming from to understand his prediction. His money is (literally – He made a bet[1] on Twitter with “Morning Joe” Scarborough of NBC) on Obama’s victory in the election, not necessarily because he wants Obama to win, but because he has confidence in his prediction. When he made the bet, he made the controversy more than just trading words, but he called Joe’s bluff (Joe had said that anyone not calling the race a tossup is an ideologue). We can now call him The Statistician Who Kicked the Hornet’s Nest – the punditry, including the public editor of the New York Times that hosts his blog, is collectively attacking him.

Unfortunately, the punditry has the upper hand, because people are more interested in the narrative than the science.

[1] The bet originally consisted of the loser donating $1000 to charity. Nate subsequently donated $2538 to the Red Cross before the election.

Willful statistical illiteracy

noreply@blogger.com (John Johnson) — Wed, 31 Oct 2012 12:45:00 +0000

The fine folks over at Simply Statistics have a very good educational article about the difference between the probability of winning an election and vote share. This article stems from a controversial column over at Politico criticizing Nate Silver and his election forecasts.

Twitter responses are even worse. Conservative filmmaker John Ziegler calls Nate Silver a “hyper-partisan fraud” who is “not an expert on polls.”

Glenn Thrush mentions a “conservative 538:”

And it’s not hard to find other examples.

I’ve run into this reaction a bit, especially when it comes to politics. There are a large group of people, who will dismiss any evidence going against their beliefs. I guess the punditry wasn’t so dismissive of Silver in 2010.

At any rate, I give a recommendation I rarely give: read this Politico article and the comments (ignore the “conservatives aren’t bright” nonsense, which is the same stuff coming from the left).

And let’s thank Nate Silver, RealClearPolitics, and all the honest pollsters who try to shine some data on this election.

The most valuable thing about my little stat blog network project

noreply@blogger.com (John Johnson) — Tue, 30 Oct 2012 01:45:00 +0000

So, I decided to construct the linking graph through blogrolls, and finally settled on using a manual process. The best part of this project is really finding out for myself all the great content out there!

SNA class proposal

noreply@blogger.com (John Johnson) — Mon, 22 Oct 2012 12:45:00 +0000

I’ve been taking several classes through Coursera (nothing against the other platforms; I took two of the original three classes via Stanford and just stuck with the platform). The latest one is Social Network Analysis, which has a programming project. Here is what I have posted as a proposal:

Ok, I've been thinking about the programming project idea some, and at first I was thinking of analyzing the statistics blogging community, mostly because I belong to it and I wanted to see what comes out. The analysis below can be done for any sort of community. I've developed this idea a little further and wanted to record it here for two reasons. First, I simply need to write it down to get it out of my head and in such a way that the public can understand it. Second, I'd like feedback.
As it turns out, I took the NLP class in the spring and think there's some overlap that can be exploited. (This comes up nicely in the Mining the Social Web and Programming Collective Intelligence books.) There are measures of content similarity, such as cosine similarity, which are simple to compute and reasonably work well to see how similar content is. Content can then be clustered based on similarity. So, then, I have the following questions:

What are the communities, and do they relate to clusters of content similarity?
If so, who are the "brokers" between different communities, and what do they blog about? There are a couple of aggregators, such as StatBlogs and R-Bloggers, that I imagine would glue together several communities (that's their purpose and value), but I imagine there are a few others that are aggregator-like + commentary as well. Original content generators, like mine, will probably be on the edges.
Is it better to threshold edges based on a number of mentions, or use an edge weight based on the number of mentions?
If I have time, I may try to do some sort of topic or named entity extraction, and get an automated way of seeing what these different communities are talking about.

Nate Silver on The Daily Show

noreply@blogger.com (John Johnson) — Sat, 20 Oct 2012 12:15:00 +0000

Watch it!

There’s an interesting conversation about how the campaigns use analytics in get out the vote efforts. It doesn’t go a lot in depth, but I think this is an important aspect of campaigns that will come out into public view in the next couple of election cycles.

Of course, you can find his blog at http://fivethirtyeight.blogs.nytimes.com/.

Data cleaning is harder than statistical analysis

noreply@blogger.com (John Johnson) — Wed, 19 Sep 2012 12:45:00 +0000

Statistical analysis is relatively hard, but it is a piece of cake compared to data collection, cleaning, and manipulation. In fact, in clinical trials research, we spend millions of dollars to develop and advance the capability to effectively manage data. Just about any clinical research organization worth the price has a strong data management department that they’ve spent a lot of time cultivating.

It’s time to take this a step further. In my workplace, we have a very close integration of the statistics group (consisting of statisticians and statistical programmers) and the data management group. In the latest issue of their newsletter, the Society for Clinical Data Management has included an article for the optimal collaboration between statisticians and data managers. (I take this a step further and include the medical writer.) This collaboration takes a lot of time – time I could be spending doing statistical analysis. However, if the statistical analysis involves working around fewer data issues, it’s all worth it.

Exercise helps statisticians

noreply@blogger.com (John Johnson) — Mon, 10 Sep 2012 12:45:00 +0000

Statistics is a rather sedentary job, and, over the years, I found my effectiveness decreasing as I found fewer “peak” hours in the day. I also gained a lot of weight. The number of migraines I experienced went from about two a year to about once a month.

In the last two or three years, I’ve been getting out of my chair to go for runs, I’ve taken up taekwondo, and also I a small gym that provides small-group personal training. In addition to adding who knows how many years to my life, they’ve really helped my focus and concentration when I’m doing statistics. I’ve also decided to take once a week or so off of thinking about statistics, which I’m finding helpful.

I only wish I had established these habits years ago.

Integrating R into a SAS shop

noreply@blogger.com (John Johnson) — Wed, 29 Aug 2012 12:45:00 +0000

I work in an environment dominated by SAS, and I am looking to integrate R into our environment.

Why would I want to do such a thing? First, I do not want to get rid of SAS. That would not only take away most of our investment in SAS training and hiring good quality SAS programmers, but it would also remove the advantages of SAS from our environment. These advantages include the following:

Many years of collective experience in pharmaceutical data management, analysis, and reporting
Workflow that is second to none (with the exception of reproducible research, where R excels)
Reporting tools based on ODS that are second to none
SAS has much better validation tools than R, unless you get a commercial version of R (which makes IT folks happy)
SAS automatically does parallel processing for several common functions

So, if SAS is so great, why do I want R?

SAS’s pricing model makes it so that if I get a package that does everything I want, I pay thousands of dollars per year more than the basic package and end up with a system that does way more than I need. For example, if I want to do a CART analysis, I have to buy Enterprise Miner, which does way more than I would need.
R is more agile and flexible than SAS
R more easily integrates with Fortran and C++ than SAS (I’ve tried the SAS integration with DLLs, and it’s doable, but hard)
R is better at custom algorithms than SAS, unless you delve into the world of IML (which is sometimes a good solution).

I’m still looking at ways to do it, although the integration with IML/IML studio is promising.

Romney’s “secretive data mining”–could the same techniques be used for clinical trial enrollment?

noreply@blogger.com (John Johnson) — Mon, 27 Aug 2012 12:45:00 +0000

Romney has been “exposed” as using “secretive data mining techniques” to find donors to his campaign in traditional Democratic strongholds. (These techniques can be learned in any of these free online courses offered through Coursera and Udacity along with the massive databases collected by the different parties.)

Of course, my thought is, can we use these techniques to find potential participants in clinical trials? I think that if we can work out the privacy issues, this represents a useful tool for clinicians to find not just trial participants, but patients who need to be treated, but for some reason are not being treated. This could be a win for everybody.

Other ideas:

using Google trends, much like Google uses to identify flu outbreaks
mining discussion boards
identifying need through blog networks

I’ll be taking the Web Intelligence and Big Data class through Coursera, so maybe I’ll get more ideas.

Clinical trials: enrollment targets vs. valid hypothesis testing

noreply@blogger.com (John Johnson) — Mon, 20 Aug 2012 12:45:00 +0000

The questions raised in this Scientific American article ought to concern all of us, and I want to take some of these questions further. But let me first explain the problem.

Clinical trials and observational studies of drugs, biologics, and medical devices are a huge logistical challenge, not the least of which is finding physicians and patients to participate. The thesis of the article is that the classical methods of finding participants – mostly compensation – lead to perverse incentives to lie about one’s medical condition.

I think there is a more subtle issue, and it struck me when one of our clinical people expressed a desire not to put enrollment caps on large hospitals for the sake of a fast enrollment. In our race to finish the trial and collect data, we are biasing our studies toward larger centers where there may be better care. This effect is exactly the opposite of that posited in the article, where treatment effect is biased downward. Here, treatment effect is biased upward, with doctors more familiar with best delivery practices (many of the drugs I study are IV or hospital-based), best treatment practices, and more efficient care.

We statisticians can start to characterize the problem by looking at treatment effect by different sites, or using hierarchical models to separate out center effect from drug. But this isn’t always a great solution, because low-enrolling sites, by definition, have a lot fewer people, and pooling is problematic because low-enrolling centers tend to have way more variation in level and quality of care than high-enrolling centers.

We can get creative on the statistical analysis end of studies, but I think the best solution is going to involve stepping back at the clinical trial logistics planning stage and recasting the recruitment problem in terms of a generalizability/speed tradeoff.