Should randomised trials be the only type of evidence accepted for rolling out drug treatments?

If so, then two researchers wrote in the Lancet this week that that we face a problem:

The evidence we have might not be the evidence we need, and the evidence that we need may never become available.

They are writing in response to the publication of a trial of a new combination drug treatment for stomach cancer that seems to extend people’s lives by nearly 3 months.

According to these guys, there’s probably lots of effective combinations of current cheap drug treatments to treat cancer, but we’re never going to get the evidence to prove it.

The new trial must have been crazy expensive: It consisted of patients from 122 different institutions in 24 counties on 4 continents and, as the editorialists point out, it would never have been paid for had it not promised huge returns for the pharmaceutical company that funded the trial, Roche.

This raises a sticky issue. Drug companies like Roche produce treatments that save lives. But they don’t do so because they save lives. Rather, they produce the drugs because they can make the company money.

So what happens when these two motivations come apart — when drugs that could save lives don’t make drug companies money? That’s the problem that the Lancet editorialists are referring to.

If we presuppose that only randomised trials produce evidence of sufficient quality to support decisions about the allocation of scarce resources, there is a problem. There is a lot of evidence on the effects of adding expensive new drugs to conventional therapies, but little evidence for when older, less expensive interventions are combined.

And boy are these drugs expensive. In the same comment piece, the authors calculated that the cost of each year of life gained by this new treatment, is about $100,000 AUD ($85,000 USD).

The authors of the study argue that this should be rolled out as a “new standard option” in the treatment of gastric cancer — and that’s not surprising given that the manufacturer of the drug not only funded the trial, but was involved in the data analysis and editing of the report.

So how are we supposed to get evidence about cheap drugs that will help millions of people but not make money for drug companies?

I don’t know. The authors mention the model in physics where large projects that don’t have obvious practical benefits are funded by governments and research institutions. And they seem to imply that there might be a way of not relying solely on randomised trials — but they don’t say what that is.

Whatever the way forward, it seems crazy that in the mean time, we can only get evidence about drugs which, for the majority of people around the world, are prohibitively expensive when there’s almost certainly some great cheap alternatives right under our noses.

Munro, A., & Niblock, P. (2010). Cancer research in the global village The Lancet DOI: 10.1016/S0140-6736(10)61022-7

Bang, Y., Van Cutsem, E., Feyereislova, A., Chung, H., Shen, L., Sawaki, A., Lordick, F., Ohtsu, A., Omuro, Y., & Satoh, T. (2010). Trastuzumab in combination with chemotherapy versus chemotherapy alone for treatment of HER2-positive advanced gastric or gastro-oesophageal junction cancer (ToGA): a phase 3, open-label, randomised controlled trial The Lancet DOI: 10.1016/S0140-6736(10)61121-X

This week we got a quick and entertaining lesson in the way science sometimes works… or doesn’t work.

Just last week, someone asked me how to tell good science from bogus science. My first, not entirely adequate response, was that you should check whether it’s been published in a peer-reviewed journal.

There’s lots of reasons why that response is not completely adequate, and one of those reasons was highlighted recently by an article being published in Virology Journal about how Jesus cured a woman of the flu.

Within three weeks the article was retracted but the fact that it got published in the first place shows that peer-review does not qualify something as good science.

I won’t say much more about this, but instead direct you to some good discussions of the debacle.

- Ivan Oransky, editor of Reuters Health, has a good overview of the debacle on his new and noteworthy blog Retraction Watch.

- Bob O’Hara has a funny discussion of the original article in This Scientific Life, written before the retraction. The editor of the journal posts his apology and retraction in the comments to the post.

Is it possible that if you have schizophrenia, smoking marijuana will actually improve your cognitive performance?

Since this blog is often concerned with the relationship between science and its communication, something which has come up once or twice here is the way drug and alcohol research is reported in the media.

Very often, it is reported that marijuana use causes schizophrenia and psychosis. This is despite the fact that there is quite strong evidence that it does not do so.

We also see government campaigns presenting very misleading information about marijuana and its effects.  Consider this one stating that “Cannabis can leave you permanently out of it”.

So it was not surprising this week, when we did not see headlines such as “Marijuana makes schizophrenics smarter,” even though there was some interesting research possibly suggesting exactly that.

But what’s really interesting about this research is that it seems to suggest that not only does smoking marijuana make people with schizophrenia smarter, it seems to show that the more often you smoke, and the younger you start, the smarter you get. (Yes, there are also other interpretations of the data. Continue reading.)

The researchers did two things: a meta-analysis and an experimental study. They found 10 studies that looked at the cognitive performance of people with schizophrenia who smoked marijuana and found a remarkable homogeneity between the studies.

Together, these 11 studies (the 10 in the analysis and the 1 they performed) found that people who had schizophrenia, and had a history of smoking marijuana, had a better visual memory as well as better planning and reasoning than their non-using peers. For several domains, there were no differences between the groups but for no domain, were the non-using groups better than the using groups.

Additionally, they found that a higher frequency of smoking was associated with higher cognitive performance, as was earlier age of smoking onset. (In their own study, the association was only present for people who started smoking before they were 17.)

How could these findings possibly be explained?

Well, of course one option is that smoking marijuana causes the better performance. This is plausible since it is known that people with schizophrenia and psychosis tend to have poorer cognitive performance and so something like marijuana might perform what the authors call “a neuroprotective role”. That is, the cognitive deficits that become apparent in people with schizophrenia at around puberty might be avoided if they smoke marijuana before, or at, that age.

However, there is another explanation of the data suggested by the authors that turns on the question raised earlier: does marijuana cause psychosis? If it does cause psychosis, the researchers suggested that there might be a group of people who would not have developed schizophrenia if they had not smoked marijuana. And that group of people, as a group less prone to schizophrenia, might also be less cognitively impaired. So people who have a history of marijuana would be more likely be a group that is less prone to schizophrenia and therefore perhaps have better cognitive performance.

So should people who think they might be schizophrenic go out and smoke lots of dope? Clearly not. While it seems to me that the evidence stands against the hypothesis that pot causes schizophrenia, the matter is far from closed and a lot of researchers do think there is such a link. Moreover, this latest research is not based on an awful lot of data and really needs to be replicated in larger studies.

What’s the bet, however, that the government website “PermanentlyOutOfIt.com.au” won’t mention this new and interesting research? Much better, they think, to stick with skewed, misleading messages rather than provide believable, balanced evidence from which people can make informed decisions.

[Via me at Psychiatry Update]

Yücel M, Bora E, Lubman DI, Solowij N, Brewer WJ, Cotton SM, Conus P, Takagi MJ, Fornito A, Wood SJ, McGorry PD, & Pantelis C (2010). The Impact of Cannabis Use on Cognitive Functioning in Patients With Schizophrenia: A Meta-analysis of Existing Findings and New Data in a First-Episode Sample. Schizophrenia bulletin PMID: 20660494

Frisher, M., Crome, I., Martino, O., & Croft, P. (2009). Assessing the impact of cannabis use on trends in diagnosed schizophrenia in the United Kingdom from 1996 to 2005 Schizophrenia Research, 113 (2-3), 123-128 DOI: 10.1016/j.schres.2009.05.031

You’ll often see loony zealots refer you to a study showing how effective their preferred treatment is — there usually is some small study supporting the use of almost any treatment.

You’ll also often hear people reply that the study was only small, so shouldn’t be trusted. But why shouldn’t you trust small studies? Sure, they won’t provide quite as much statistical power as larger ones, but surely they can still be useful.

And that’s true. They can be useful, and they do provide important information. But a meta-epidemiological study in the British Medical Journal recently showed a really interesting fact about small studies.

The researchers highlight what is known as the “small study effect”: a very particular bias that small studies introduce into systematic reviews.

It turns out that small studies are systematically biassed towards the effectiveness of the intervention they are testing.

Systematic reviews pool the results of all the relevant studies on a particular issue and usually provide the very best evidence. Before major decisions are made about some particular treatment, we usually wait for a big systematic review of the literature to be published.

But if there have been lots of small studies done, then when researchers conduct a systematic review, it turns out they might end up being slanted.

These particular researchers looked at studies that tested various treatments for osteoarthritis and they plotted all the studies for each treatment on a graph with the larger trials near the top and the smallest ones near the bottom.

If the study showed the treatment was very effective, they plotted it further to the left and if it showed it was ineffective (or had a negative effect) they plotted it further the the right.

They call them funnel plots because if small trials are not biassed, the graphs will resemble funnels. The large studies will group together at the top and the small studies will scatter evenly on either side.

The results are visually striking. Far from resembling funnels, they resemble toppling towers — with small studies heavily drawing the plots to the left, towards the treatment being more effective.

The funnel plots for the 13 meta-analyses studied.

You can see just by looking at these plots that if systematic reviews are conducted, pooling all these data, the final analysis will generally be skewed towards supporting the treatment.

So why are small studies biassed in this way? The study authors suggest that there might be several factors at play. For one thing, there might be a selection bias: Small studies that show less effect might be less likely to be published. They also suggest a number of other explanations including a problem with participants being excluded from the analysis after being randomised into one of the arms.

They urge that authors of systematic reviews include the above funnel plots in all systematic reviews, and if a “small study effect” is observed, the reviewers should include a separate analysis that excludes all the small studies.

Update: The researchers defined “small” studies in this paper as ones with less than an average of 100 participants in each arm. Thanks, Simon, for pointing this out in the comments.

So next time someone points you to a small study showing how effective acupuncture is, how reflexology relieves depression or how fish oil cures everything, you can rest easy knowing that you’re under no obligation to accept the study’s conclusions. It’s much better to wait for a larger study, a meta-analysis, or even better, a meta-analysis that controls for the small study effect.

Nüesch E, Trelle S, Reichenbach S, Rutjes AW, Tschannen B, Altman DG, Egger M, & Jüni P (2010). Small study effects in meta-analyses of osteoarthritis trials: meta-epidemiological study. BMJ, 341 PMID: 20639294

Reports of a physicist “taking on gravity” have recieved a bit of attention recently, with a New York Times article outlining Erik Verlinde’s idea that gravity is an emergent property of thermodynamics.

I think it’s great that the piece was written — even though apparently it hasn’t excited any physicists since the start of the year. Regardless of how current or accepted the idea is, the topic is interesting enough that it’s making lots of people read about some pretty complex scientific ideas. And that’s always good!

But one thing that does kind of erk me about the story is the seeming philosophical niavity of both the journalist, and the physicist (assuming the physicist hasn’t been completely misquoted). And working through the philosophical problem reveals some really interesting issues, I think. Let me explain.

The piece starts off as follows:

It’s hard to imagine a more fundamental and ubiquitous aspect of life on the Earth than gravity, from the moment you first took a step and fell on your diapered bottom to the slow terminal sagging of flesh and dreams.

But what if it’s all an illusion, a sort of cosmic frill, or a side effect of something else going on at deeper levels of reality?

And then it continues later, quoting Verlinde:

“For me gravity doesn’t exist,” said Dr. Verlinde, who was recently in the United States to explain himself.

And once more:

“We’ve known for a long time gravity doesn’t exist,” Dr. Verlinde said, “It’s time to yell it.”

‘Not fundamental’ is not the same as ‘doesn’t exist’

Let’s get something straight: saying that something is an emergent property, or not a fundamental property of the world, is not the same as saying that the property does not exist! Not even close!

If those two things were the same, then most philosophers and scientists wouldn’t think that very much existed at all. Perhaps all that they would think exists is energy or information or whatever they think is the most fundamental stuff in the universe.

Now, there is a philosophical position that you can take about certain things (properties, objects etc) called “eliminativism”. To be an eliminativist about, say, unicorns, is to say that they don’t exist. You are “eliminating” them from your ontology (that is, the list of things you think exist).

There have been some interesting eliminativist ideas in philosophy of science. Bertrand Russell (political dissident, mathematician, philosopher, total hero) once argued that causality doesn’t exist — he was an eliminativist about causation, arguing that the word should stop being used in serious, scientific discussions. (He was wrong, by the way.)

But there’s a problem…

But hang on. Verlinde might think he’s an eliminativist for the following reason: Everyone thinks that gravity is this fundamental force in the world. But they’re wrong! What THEY THINK GRAVITY IS, doesn’t exist!

If what we meant originally by “gravity” is “a fundamental force”, then this physicist’s theory that says that it’s an emergent force is in fact saying that gravity doesn’t exist.

So what’s the answer?

At this point then, it might seem like a definitional problem. We should just decide what we meant by “gravity” to start with and move one.

And that’s almost right. But there is an answer: Verlinde is wrong to say he thinks gravity doesn’t exist. (And it’s worth noting that he doesn’t say it in the paper, he only says it to the journalists.)

If Verlinde were right, and his poisition was an eliminativist one, then the following absurd conclusion is true: every theory that has ever changed the way we think about anything was an eliminativist one. But that’s obviously false.

Consider the Copernican revelation: the earth goes around the sun, and not vice versa. Copernicus did not wake up one morning and announce “My god! The earth does not exist!” Instead, he woke up one morning and said “My god! The earth is not the centre of the universe!”

Of course, you could insist that what you meant by “earth” was the thing we live on, which is in the centre of the universe. And then, Coperincus certainly did show that what you’re calling “earth” does not exist.

But a much more helpful way of viewing the matter is that Copernicus showed us that the earth is not exactly what we thought it was.

Similarly, Verlinde’s claim that gravity is an emergent property is not the claim that gravity does not exist. Rather, if he’s right, he’s shown us that gravity isn’t exactly what we used to think it was.

[For those particularly interested in this issue, there are some fascinating borderline cases of theories that may have shown us new facts about old things, or shown us that the old things don't exist. The most famous of these is phlogiston.]

Erik P. Verlinde (2010). On the Origin of Gravity and the Laws of Newton arxiv.org arXiv: 1001.0785v1

Bertrand Russell (1912). On the notion of cause Proceedings of the Aristotelian Society

I’m often frustrated by the pseudoscientific rubbish people around here believe in. (By around here I mean places like Australia, the US and the UK.) But thank god we don’t believe in quite as much rubbish as some places.

According to Associated Press, an acupuncturist has convinced people she can detect a man’s virginity by looking behind his ear. And doing so, she contributed to the reopening of the cases of thee convicted rapists, who were eventually freed. Apparently the convicted rapists were actually virgins.

Nuff said.

This is the second part in a series about the myths and realities of alcohol consumption.

I pray thee let me and my fellow have
A hair of the dog that bit us last night—
And bitten were we both to the brain aright
-  John Heywood

The idea that alcohol may itself be a cure for alcohol hangovers is something that most people would have heard. “Hair of the dog”. Could it have any truth to it?

It’s certainly popular: A glance at twitter shows that people all around the world are talking about needing some “hair of the dog” every minute of the day and a survey of college students revealed that 25% reported using alcohol as a hangover cure.

A google search reveals a lot of sites recommending “hair of the dog” — many of them supplying less than plausible explanations of how it might work.

Chiff very unconvincingly suggests that “adding more alcohol (ethanol) makes your body stop and concentrate on the new alcohol coming in.” (I find this kind of explanation interesting: It’s very common for people to psychologise their bodies, giving psychological explanations for non-psychological processes.)

Wikipedia explains that the newly consumed ethanol “dilutes” the metabolites (the things your body breaks the alcohol down to) of the earlier alcohol. The idea behind this is that the metabolites of alcohol are what give you the hangover. But how diluting them with more alcohol could make you feel better is not explained. Although wikipedia cites a research paper for this explanation, no such suggestion appears in that paper. (Someone should consider fixing the wikipedia page to reflect this.)

What the research says
The first thing worth mentioning here is that there hasn’t been an enormous amount of research into hangovers.

An editorial in the journal Current Drug Abuse Reviews outlined the state of the literature as follows.

Although the vast majority of excessive drinkers do experience alcohol hangovers, the scientific community has paid little attention to its pathology and possible treatment. This can be explained by the fact that physicians do not view hangover as a disease, but as a consequence of unwanted behavior. In fact, alcohol hangover can be easily prevented by moderating alcohol consumption. Moreover, finding a cure for hangovers might result in more excessive drinking episodes. This makes treating alcohol hangover a controversial topic.

That said, there has been some relevant work.

A recent review in a prominent journal considered the possibility that the hangover is an early stage of alcohol withdrawal. If that were the case, then hair of the dog would make sense. Consuming more alcohol would eleviate withdrawal the way that small doses of opiates eleviate opiate withdrawal, or having a cigarette relieves nicotine withdrawal.

Borrowed from The Annals of Internal Medicine.

But according to the reviewers this theory doesn’t seem to stack-up. Alcohol withdrawal and hangovers seem to differ in many ways, with some symptoms being quite the opposite in the two conditions. (See table, right.)

As far as I can tell, that’s the extent of the research directly relevant to the matter.

Systematic reviews have shown that no hangover cures have been proven to work — but that doesn’t mean they don’t. As I’ve said, very little research has been done and they usually focus on obviously bogus commercial “miracle” cures.

A priori, some explanations seem to make some sense. One simple one is that perhaps getting drunk simply makes you forget that you’ve got a hangover. The well-known analgesic effects of alcohol might help with some of the symptoms and the cognitive impairment might make you notice the other symptoms less.

Of course, this is a completely unsubstantiated claim. It’s also very plausible that hair of the dog simply doesn’t work and, as one editorialist suggested, is simply an excuse for early stage alcoholism.

What do the rest of you think?

(Stay tuned for the next installment in the series on the myths and realities of alcohol consumption.)

Jeffrey G. Wiese, MD; Michael G. Shlipak, MD, MPH; and Warren S. Browner, MD, MPH (2000). The Alcohol Hangover Annals of Internal Medicine, 152 (12), 897-902

Verster JC (2009). The “hair of the dog”: a useful hangover remedy or a predictor of future problem drinking? Current drug abuse reviews, 2 (1), 1-4 PMID: 19630732

Wiese JG, Shlipak MG, & Browner WS (2000). The alcohol hangover. Annals of internal medicine, 132 (11), 897-902 PMID: 10836917

This is the first part of a series that aims to bust some myths about booze.

Hangovers suck and they’re probably best avoided. But once you’ve got one, can you get rid of it? People swear by their favourite hangover cures — insisting that if you just follow their advice, you’ll free yourself of the post-intoxicated state.

Can they be avoided by drinking only expensive wines? Will your hangover be worse if you mix spirits and beer? Do some people never get hangovers? If you get drunk easily, will your hangovers be less severe? Does alcohol kill brain cells that never grow back? Is alcohol more or less addictive than heroin?

These are just some of the questions I’ll be exploring through a series of posts on booze-related myths and realities.

I was moved to look into this when I was faced with the seemingly very widely held belief that cheap wines give you worse hangovers than expensive wines. While I know very little about the biology of alcohol intoxication, this just seemed to me to be utterly implausible. If it is true, then it stands to reason that what really gives you the hangover is something other than the alcohol. And if that is true, then presumably you could get a hangover by drinking something other than alcohol. But this seems absurd. So (by what philosophers call reductio ad absurdum) our initial premise must be wrong (or one of our logical steps was wrong).

So, somewhere along the way, I’ll try to find the answer to that, and many other alcohol related questions.

For the first post in the series I want your help. I want you to tell me all your alcohol-related questions that you were too afraid to ask. Throughout the series, I’ll try to answer them by sifting through published papers and speaking to people who’s job it is to know (scientists, I mean). Do you have tried-and-true hangover cure? Have you heard about a cure that you’d like to know more about? Are you worried about some putative effect of alcohol consumption? Leave a comment below or email me.

One thing that is of central concern to this series is the question of what a hangover is. In a paper called “The alcohol hangover – A puzzling phenomenon”, one researcher put the question as follows.

The alcohol hangover is an intriguing issue since it is unknown why these symptoms are present after alcohol and its metabolites are eliminated from the body.

The paper continues:

Although numerous scientific papers cover the acute effects of alcohol consumption, researchers largely neglected the issue of alcohol hangover. This lack of scientific interest is remarkable, since almost everybody is familiar with the unpleasant hangover effects that may arise the day after an evening of excessive drinking, and with the ways these symptoms may affect performance of planned activities.

This researcher explains that the popular belief that dehydration is the cause of hangover symptoms is almost certainly false. More likely, he thinks, are the hormonal changes and changes to the immune system that occur following a heavy drinking session. As we will see, not everyone agrees.

The plausibility of many of the putative myths and realities I consider will no doubt depend on the precise nature hangovers and alcohol intoxication — something I hope to shed some light on over the next few posts.

So stay tuned for more blogging about the myths and realities of alcohol…

Verster JC (2008). The alcohol hangover–a puzzling phenomenon. Alcohol and alcoholism (Oxford, Oxfordshire), 43 (2), 124-6 PMID: 18182417

Martin Gardner — the science writer, maths writer, skeptic and the man who’s book I plagiarised the title of this blog from — has died at the age of 95.

I didn’t know his work as intimately as many others did, so will not say much here. Instead, I suggest you read one of the many wonderful pieces that have been written about him, his life and his work.

• Scientific American has a tribute page to him containing some previously published profiles as well as some obituaries.

• The Guardian has an obit.

• Writers at Scienceblogs.com have written lots of stuff.

-
Ahead of a landmark study, newspapers all around the world have carried news reports warning of the terrible risk faced by mobile phone users.

The story behind the story is worth a few words. Firstly, the study seems to say the opposite to what all these reports claim. Secondly, the study was under embargo until the time this post was published (9.30am, Sydney time), a good 48 hours after the publication of some of these new reports.

The UK media — broadsheet and tabloid alike — have led the sensationalist charge.

The TimesOnline says:

PEOPLE who use their mobile phones for at least 30 minutes a day for 10 years have a greater risk of developing brain cancer, a landmark study has found.

Here in Australia, the national broadsheet, The Australian, led their story with:

A LONG-awaited international study of the health risks of mobile phones has linked extended mobile phone use to an increased risk of developing brain tumours.

All this is a little surprising since when you actually look at the study, it says that “overall, no increase in risk of glioma or meningioma was observed with use of mobile phones“.

Professor Bernard Stewart, the Scientific Advisor to Cancer Council Australia, commenting on the research says that the study “has found no evidence that mobile phones cause brain cancer, consistent with previousresearch in this area”. He continues:

We already know that mobile phones are not able to damage genetic material (DNA) in cells directly and so cannot produce cancer-causing mutations. However, there have been concerns that electromagnetic fields from mobile phones may be able to increase the rate of cancer development by influencing cancer promotion or progression. The Interphone study has found no evidence supporting this theory.

Some questions do still remain. As Professor Stewart says:

However, it did find that in patients with glioma, based on heavy phone use, the tumour was likely to be on the same side of the head as the mobile phone. While this does not prove a link between mobile phones and cancer, it does merit further research.

Also, other experts have pointed out that the study did not have much data on the use of mobile phones by children or on very heavy phone users. Nevertheless, stories saying that the study established a link are far from justified.

Indeed, it’s hard to imagine what more the researchers could possibly have said to avoid these kind of headlines: Not finding evidence of a link is as good as it gets insofar as establishing that there is no link. With epidemiological studies, you can’t prove that there is no link other than by looking for evidence of one, and not finding it.

In addition to the questionable reporting of the results, news outlets were reporting on the findings a good 48 hours ahead of the lifting of the embargo. I’m not quite sure what to think about this.

I decided not to post info about the article, even though it was freely available [pdf] to anyone with an internet connection. A number of news outlets initially weaseled their way out of actually breaking the embargo by reporting on reports of the study… But once they opened the lid, every other news outlet assumed it was ok for them to report on the study itself.

The whole notion of embargos makes me feel a little uneasy. I feel that it drives a wedge between two principles of journalism… But perhaps that whole issue is best saved for another post.

Elisabeth Cardis (2010). Brain tumour risk in relation to mobile telephone use: results of the INTERPHONE international case–control study International Journal of Epidemiology, 1-20 : 10.1093/ije/dyq079

.