I caught up with Keith for a Curbside Consult video conversation at healthinsurance.org. We touched on the politics of Medicare mental health parity, how ACA will change America’s treatment system, alcohol as the substance which sends more Americans to prison than any other, whether improved policies could reduce crime by individuals with severe mental illness. I hope that TIE readers will enjoy it.

@haroldpollack

The satire is worth a read. Despite its age it holds up well. I promise a smile, if not a laugh out loud. Hat tip to Harold Pollack on Twitter. Tyler Cowen blogged on it in 2007. Click through for a few quotes.

@afrakt

This is why I blog and in the style I do so. After some back-and-forth in the comments to my post on bias yesterday (go read those comments), Matt offers more precise terminology — distinguishing bias from internal and external validity — within a differently organized discussion. I like what he’s done. My comments and questions are interleaved with his. Let’s keep discussing this!

(1) What internally valid estimates can we obtain from the Oregon Study?

We can obtain the effect of winning the lottery (ITT) and the effect on the population that gained insurance due to winning the lottery (LATE).

We cannot obtain the ATE [average treatment effect] or the TOT [effect of treatment on the treated]; the seemingly natural estimators of these quantities are biased since the populations we are comparing differ due to self-selection. The ITT and LATE avoid this problem because they scrupulously _solely_ compare the full group of lottery winners to the full group of lottery losers.

I agree that LATE exploits the lottery, but does it really compare the full groups of winners to losers? My understanding is it compares the two groups of compliers, as I wrote. That’s the difference between ITT and LATE.

(2) What internally valid estimates can we obtain from alternative study designs? How do they differ?

From a perfect compliance RCT, we can estimate the ATE for the study population. Relative to the group covered by the LATE, the group covered by the ATE also includes: (A) the types of people who still enroll if they lose; and (B) the people who will not enroll even if they win.

Point of clarification: With perfect compliance, there are no people who still enroll if they lose. There are no people who do not enroll even if they win. However, those groups exist in an RCT without full compliance and, as I wrote above, LATE filters our their effect. Under full compliance LATE is the same as ATE is the same as ITT. The way I’d put this is not that ATE includes these noncompliant groups. I’d say that the ITT and LATE estimates are the ATE in a fully-compliant RCT. They are not in an RCT without full compliance. Continuing with Matt’s section (2):

From a Oregon-like study in which we forbid enrollment by lottery losers, we can obtain the TOT. Relative to the group covered by the LATE, the group covered by the TOT adds (A) from above but not (B).

TOT compares treated with untreated. I can think of three ways to do this, and I’m not certain which one we call TOT. Way 1 compares all treated to untreated, regardless of random assignment. Way 2 does so only for those assigned to treatment. Way 3 does so only for those assigned to control. Which one is TOT? Why does it incorporate a group like (A) but not (B)? The only version of the three versions of TOT I suggested that is consistent with that is way 3. But way 3 is the one that sounds least likely to be what one means by TOT. (I’d order it as way 1 > way 2 > way 3.) Still continuing under his section (2):

The LATE/ATE/TOT difference is not about bias. Each average treatment effect is perfectly valid for the population it pertains to; those populations are just different.

I agree that the differences among these types of estimates have nothing to do with bias. However, Matt wrote early in his comment (way above), “the seemingly natural estimators of these quantities are biased since the populations we are comparing differ due to self-selection.” I think we need to explore this more. What does “seemingly natural” mean? It appears to be doing a lot of work here. When do we say we have obtained a biased estimate? Can we give a precise example? Does it merely mean we haven’t really computed one of them properly?

(3) Which estimates have the greatest external validity for the policy questions of current interest?

This is a hard question. The answer depends on whether the group affected by our proposed policy looks more like the group included in the LATE or the full population. How do we think about that?

Insert your existing discussion of this point.

Matt is either referring to my comments to my post or the post itself. Either way, I presume you’ve read them.

@afrakt

Our FY 2013… budget is approximately $1.395 billion. This includes a 5.0% sequestration reduction, an across-the-board 0.2% rescission, and a 0.6% DHHS Secretary’s discretionary transfer reduction. Overall, relative to FY2012, our budget is reduced by $84 million, or 5.7%. For anyone who tracks trends in funding, our FY 2013 budget is roughly equivalent, in absolute dollars, to our FY 2004 budget. Corrected for inflation, we are nearly back to 1999.

What is the effect of those cuts on NIMH research?

…we anticipate awarding 529 new Research Project Grants (RPGs) in FY 2013, which will reflect a reduction of 55 awards below the 584 new awards issued in FY 2012.

So, a 9.4% reduction in the number of new projects. How important is that?

Medical research is like buying lottery tickets. The great majority of projects make small contributions to knowledge or human well-being, but every so often there is a transcendent victory. It’s impossible to know in advance which project will be a triumph. If you want to find one, you need to buy a lot of tickets.

Mental health really really really needs some triumphs. Mental illness kills thousands of Americans and disables millions. The available medications and psychotherapies have only limited benefits. To my knowledge, there are no fundamentally new medications or psychotherapies on the horizon. If it was up to me, I would buy a lot more tickets.

@Bill_Gardner

For many goods and services, relying on patents and market prices might be a great outcome. For prescription drugs, however, the “market” rarely sets prices, at least in countries with government-funded reimbursement systems. Access is another salient issue. We might be fine with patents raising the price of an iPhone 5 to be beyond the reach of the lowest income quartile (for example), but that result seems unacceptable when the drug is lifesaving and is priced beyond the reach of several billion people. Differential pricing theoretically addresses some of these concerns, but has been very challenging in practice.

One delinkage proposal is the R&D Treat proposal floated at WHO during the last few years, coming from the CEWG process. The Atlantic article describes the strident opposition to the proposal from the Obama Administration, which seems surprising. Some major pharmaceutical companies (such as GSK) publicly support delinkage, while most do not.

Several other delinkage proposals were discussed in late February at an FDA conference at the Brookings Institute, all focused on antibiotics. As I’ve written with Aaron Kesselheim (in Health Affairs and the Yale JHPLE), antibiotics might be a particularly apt drug class to test the delinkage concept. One model (with Thomas Pogge & Aidan Hollis) is the antibiotic health impact fund, paying annual prizes to the patent holder over 10 years for the actual health impact of the drug around the world. For the first time, companies would have a financial incentive to get the drug to the sickest people able to benefit the most at an affordable price instead of overmarketing a precious exhaustible resource. Another model is the Strategic Antimicrobial Reserve, paying a company NOT to market a particularly valuable antibiotic, saving it for a time of greater need. A third might be to modify antibiotic reimbursement away from unit sales (which drive resistance) through payer-based models.

Prior TIE posts on antibiotics here.

@koutterson

Importance  Randomized trials of implantable cardioverter-defibrillators (ICDs) for primary prevention predominantly used single-chamber devices. In clinical practice, patients often receive dual-chamber ICDs, even without clear indications for pacing. The outcomes of dual- vs single-chamber devices are uncertain.

Objective  To compare outcomes of single- and dual-chamber ICDs for primary prevention of sudden cardiac death.

Design, Setting, and Participants  Retrospective cohort study of admissions in the National Cardiovascular Data Registry’s (NCDR) ICD registry from 2006-2009 that could be linked to Centers for Medicare & Medicaid Services fee-for-service Medicare claims data. Patients were included if they received an ICD for primary prevention and did not have a documented indication for pacing.

Main Outcomes and Measures  Adjusted risks of 1-year mortality, all-cause readmission, heart failure readmission, and device-related complications within 90 days were estimated with propensity-score matching based on patient, clinician, and hospital factors.

Here’s the deal. Most of the research on implanted defibrillators has been conducted on what are called single-chamber devices. But then came along dual chamber devices, which have some theoretical benefits. However, most of those are for a select population. But this being the US and all, soon two-thirds of all ICDs being put in were dual-chambered, and about 60% of those lacked any documented reason for using the more complicated device. Oh – the dual chambered devices cost more, too.

This was a comparative effectiveness study of people who received a single or dual chamber ICD from 2006-2009. What did they find? Of the 32,034 patients, 38% got a single-chamber device and 62%  got a dual-chamber device. After controlling for come clinical issues, those who got a single-chamber ICD had fewer complications than those who got a dual-chamber ICD (3.51% vs 4.72%). In contrast, those with the single-chamber device did not see higher 1-year mortality (9.85% vs 9.77%), 1-year all-cause hospitalization (43.86% vs 44.83%), or hospitalization for heart failure ( 14.73% vs 15.38%).

In other words, the dual-chamber device offers no improvements in mortality, all-cause hospitalization, or heart failure hospitalization. This held true even when looking at sub-groups of age, sex, or the presence of renal dysfunction. BUT, dual-chamber devices did have a significantly higher risk of complications.

And they cost more. This is how comparative effectiveness can help. But only if we act on it.

@aaronecarroll

Sidebar: I quote from the methods: “Power calculations were not performed because the study was retrospective with a fixed sample size. The precision of the estimates for the hazard ratios between single- and dual-chamber ICDs is reflected in the confidence intervals for these estimates.” Please recognize, however, that this was a study of over 32,000 patients – all of whom were appropriately analyzed. All got one of the therapies, and all could have had the outcomes. They were able, therefore, to detect a clinical difference in complications of 3.51% and 4.72% as statistically significant. The difference in hospitalization for heart failure (14.73% vs. 15.38%) was not statistically significant.

Yet in Oregon, we are expected to believe that the application of insurance to 56 or 57 people with documented hypertension should tell us something significant.

There’s a precise way in which the study is not biased. By design it estimated the effect of Medicaid on those who won the lottery and enrolled, relative to those who lost the lottery and did not. This estimate is unbiased for the contrast between precisely these two groups, but not necessarily for others. In econometric jargon, this is known as the “local average treatment effect” (LATE). The “treatment effect” part of “LATE” is clear, but what’s this “local average” business?

Sigh. I hate this terminology. It’s supposed to evoke the idea that the instrument (the lottery in this case) doesn’t have a “global” effect on study participants, causing all randomized to Medicaid (lottery winners) to be on and all those randomized to control (lottery losers) to not be. It has a more modest, “localized” effect. The other jargon used for this is that the LATE estimate is an estimate of the effect of treatment on “compliers.” That’s a more meaningful term to me. The compliers are those that do what randomization “tells” them to do, they enroll in Medicaid if randomized to do so and they don’t if not.

Of course, you can’t expect full compliance in this study (or many other RCTs) because some lottery winners turned out to be ineligible for Medicaid by the time they were permitted to enroll. Some had too high income. Some moved out of state. Some may have found other sources of coverage. (You had to have income below 100% FPL, live in state, and uninsured for 6 months to be permitted to enroll.) Also, enrollment wasn’t mandatory. So, if you just decided it wasn’t worth the trouble or didn’t receive or notice the letter inviting enrollment, you might have missed the window (45 days is all they gave you).

On the flip side, nobody was preventing lottery losers from enrolling on Medicaid if they became eligible in another way. The study pertained only to the expansion of Medicaid beyond the statutory requirements. If people ended up in one of the eligible categories (aged, blind, disabled, pregnant) they could get on Medicaid.

So, there was considerable “crossover” (lottery losers enrolling in Medicaid, lottery winners not) or “contamination” or “noncompliance,” all jargon for the same thing. This was not a perfect RCT. Few are.

What to do? The investigators did two things. First, they considered an “intent-to-treat” (ITT) approach, comparing lottery winners to losers no matter whether they enrolled in Medicaid or not. These results are in their first year paper. I’ve forgotten what they say specifically, though in general they’re much smaller effects than the LATE results. The concern with ITT is that all this crossover biases the results toward zero. There isn’t as much contrast between study arms due to noncompliance.

Next, the investigators provided LATE estimates, about which I wrote above. These are unbiased for contrast among compliers. In this study, they’re about four times the size of the ITT estimates by virtue of the mathematics (“instrumental variables“) of LATE. But they need not be the same as one would find in the absence of noncompliance. There may be bias in that sense. Why?

• Hypothesis 1: Those who took the trouble to enroll in Medicaid were sicker than those who didn’t. After all, why enroll if you don’t need it? Remember, even some lottery losers (18.5% of them) enrolled in Medicaid. The LATE estimate removes the effect of them since they are noncompliers. Also, some lottery winners didn’t enroll (most of them didn’t) and the LATE estimate removes their effect too. What’s left under this hypothesis is a comparison of relatively sicker people who did enroll in Medicaid with relatively healthier people who didn’t. The investigators actually found some evidence to suggest that Medicaid enrollees are sicker. Many other studies find that Medicaid enrollees are sicker to the point that some studies find an association of Medicaid with increased mortality. Under hypothesis 1, results are biased downward relative to what they would be under full compliance. Medicaid looks less effective than it might otherwise be. 
• Hypothesis 2: Those who are more organized, better planners, with higher cognitive function and literacy (including health) skills enroll. It takes some awareness and planning to enroll, so there is some face validity to this argument. I’m aware of no evidence to support it though. (Got any?) Under this hypothesis Medicaid enrollees would do a better job of getting and staying healthy even apart from whatever Medicaid does for them. This would bias results toward showing a larger Medicaid effect than would be true in general (under full compliance).

There may be other hypothetical sources of bias. The point I’d make about all of them is that we don’t know whether any of these biases actually exist and, if they do, how big an effect they have. It’s all speculation. Still, LATE is an unbiased (and causal) estimate of the effect of Medicaid on compliers. It does filter out some who want to be on Medicaid and can’t enroll (lost lottery, no other route) and filters out some who enroll but weren’t invited (lost lottery but became eligible another way). Some of these noncompliers could be unusually sick. Some noncompliers could be unusually organized and aware. LATE filters some of them out.

Some might wonder about another type of estimate one could do, the effect of “treatment on the treated.” Here one just compares Medicaid enrollees to non-enrollees, ignoring the lottery draw. Unfortunately, this just exacerbates whatever bias might exist. There is no random assignment at play here. There’s no filtering for selection at all. You get an association, not a causal estimate. This is the problem with many studies of Medicaid and insurance. Randomness is key. The lottery should be exploited in some fashion (either ITT or LATE).

Lastly, notice how complicated RCT interpretation is? Yes, it’s the gold standard, but it still has issues. Using an IV approach for a LATE estimate is, in my view, about the best you can do. But there may be bias when considering generalizing the findings outside the “local” effect of the instrument (lottery or random assignment). These concerns arise with any IV study. In this sense, IV and RCT are much closer cousins than one tends to think. Disparage one and you disparage the other.

Not all that’s gold glitters, but it is still valuable.

@afrakt

I am one of those who are very willing to be refuted if I say anything which is not true, and very willing to refute any one else who says what is not true, and quite as ready to be refuted as to refute-for I hold that this is the greater gain of the two, just as the gain is greater of being cured of a very great evil than of curing another. For I imagine that there is no evil which a man can endure so great as an erroneous opinion about the matters of which we are speaking and if you claim to be one of my sort, let us have the discussion out, but if you would rather have done, no matter-let us make an end of it.

-Socrates (h/t Plato)

@afrakt

I discuss this, and how it’s a very personal decision over at CNN.com.

Go read!

@aaronecarroll

From the supplementary materials posted by the authors of the Oregon study:

29,664 households were selected by lottery. If individuals in a selected household submitted the appropriate paperwork within 45 days… and demonstrated that they met the eligibility requirements, they were enrolled… About 30% of selected individuals successfully enrolled. There were two main sources of slippage: only about 60% of those selected sent back applications… (emphasis added).

Jim Manzi (via Megan McArdle) comments

If your mental model of the uninsured is a poor family huddled outside of a hospital unable to find any way to pay for a doctor to give antibiotics to their coughing child, then this result doesn’t make a lot of sense.

It doesn’t make a lot of sense because health insurance is costly and yet most people who have it are very reluctant to give it up. Nearly everyone seems to think health insurance is important: liberals passed the Affordable Care Act to extend health insurance to more people and Republicans criticized the Democrats for cutting Medicare. So it was a surprise that given the chance, 40% of Oregonians who had entered themselves in the lottery did not apply after they won.

Why is this? Manzi believes that

Either: (1) a rational analysis indicated that the expected gain from the coverage being offered didn’t justify the time and effort of filling out a form and submitting it; or (2) the winner acted irrationally about long-term benefits versus immediate inconvenience.

Manzi leans toward (2) and with significant qualifications I agree with him. Some Oregonians may have found it easy to complete the forms (tens of millions of Americans are functionally illiterate). Nevertheless, it is likely that many simply failed to complete the form.

Of course, the poor have no monopoly on irrational imprudence. Ask me about the time that I procrastinated about renewing the registration of my car. Or ask the former employees of Lehman Brothers about whether they should have looked more closely at the details of some real-estate-backed securities. But Manzi’s right: humans, including poor humans, are often negligent and irresponsible. So what follows from this?

I think the failure of so many Oregonians to complete their applications illustrates the need for universal coverage. Here is how it works in Canada: you don’t apply for health insurance. You just get it. There may be only a few goods or services that should be supplied universally and paternalistically. However, precisely because of our irrational imprudence, if health insurance is important we shouldn’t make it hard for poor people to get it. (Whether health insurance in general or Medicaid in particular are worth having are important but separate arguments.)

Manzi also believes that the finding that so many people failed to follow through on their Medicaid applications suggests that giving them Medicaid will not improve their health. This is because staying healthy requires

consistent compliance over months and years with many of the therapeutic regimes necessary to achieve improvement on the physical health outcomes measured in the experiment – blood pressure, blood sugar and cholesterol.

If you can’t get it together to submit some forms, are you really going to forgo all those cookies and Big Macs?

Manzi is pointing at a real and daunting problem but a health insurance scheme can do only so much to get people to take better care of their health. The responsibility for health promotion for the healthy and chronic disease care for the ill falls on us, individually and collectively: health care providers, schools, employers, family, and friends. Insurance can and perhaps should be re-engineered to provide supporting incentives, but it is a very limited tool for health behaviour change.

(Thanks to Adriana McIntyre [@onceuponA] for pointers to information about the Oregon study.)

@Bill_Gardner