Today in the Proceedings of the National Academy of Sciences, Nosek and an international team of researchers present a tool for doing that — betting. They found that compared to simply asking experts to predict the likelihood that studies will be reproduced, asking them to bet money on the outcomes improved the accuracy of the guesses.

The researchers began by selecting some studies slated for replication in the Reproducibility Project: Psychology — a project that aimed to reproduce 100 studies published in three high-profile psychology journals in 2008. They then recruited psychology researchers to take part in two prediction markets. These are the same types of markets that people use to bet on who’s going to be president. In this case, though, researchers were betting on whether a study would replicate or not.

Before each prediction market began, participants (47 actively took part in the first market, 45 traded in the second) were asked two questions: How likely do you think it is that each hypothesis in this market will be replicated, and how well do you know this topic?

They were then given points worth a total of $100 to bet on whether the studies in their prediction market would replicate. A replication was considered successful if it produced a result, with a p-value of less than 0.05, in the same direction as the original result. Players entered the market with 10,000 points each and could buy and sell contracts for each hypothesis. If a replication succeeded, then its share paid 100, but if the replication failed, then it paid nothing. “If you believe the result will be replicated, you buy the contract, which increases the price,” said the study’s lead author, Anna Dreber, an economist at the Stockholm School of Economics. “If you don’t believe in a study, then you can short-sell it.”

….The prediction market correctly called nearly three-quarters (71 percent) of the attempted replications, 39 percent of which succeeded in the reproducibility project. By comparison, the survey conducted before the market began correctly predicted the result of only 58 percent of the replication studies. The prediction market anticipated a finding’s reproducibility better than asking the same bunch of experts to put their best guesses in a hat.

Our team has sometimes done that at Bridge. For example, we’ve had a pool for over a year to predict our kids’ KCPE scores. Not sure yet how we did!

“The beauty of the market is that we allow people to be Bayesian,” Dreber said. People come in with some prior belief, but they can also follow prices to see what other people believe and may update their beliefs accordingly. While the survey required everyone to provide an estimate for every study, participants in the market could focus their bets on the studies they felt most sure of, and as a result, rough guesses didn’t skew the averages as much.

Finally, putting money at stake motivated people to try harder to find the right answer and reveal what they really think. “It’s really putting your money where your mouth is,” Dreber said. “You want to see what people do, not what they say.”

I’m not sure if our “winner gets to pick restaurant, everyone else takes him/her out” is particularly motivating. But I do think we changed our picks to a degree as we noticed what others said, and were allowed to ask their rationale.

Doug Lemov profiles a teacher who is particularly good at turn-and-talk. Erin Krafft in Houston.

He writes:

The first thing we noticed was how beautifully she built a positive, productive and efficient procedure for Turn and Talk. In this clip you can see her essentially installing the Turn and Talk system she will use for the duration of the year. It’s a brilliant system that we hope you’ll steal! The system included guidance on how to agree or disagree within one’s partner within the Turn and Talk, and it fostered an incredibly powerful Culture of Error (discussed later) that enables more rigorous and productive student-to-student discussion.

Some key points:

Erin’s Turn and Talk began with clear “Managed Turns.” “Door partners” were instructed to speak first, and Erin checked with a show of hands to be sure that students were clear on which partner is “door partner.” Then it would be the “window partner’s turn.” This small detail is critical since the goal of Turn and Talk is to involve all students. In many classrooms students are sent to a series of Turn and Talks where some kids talk constantly and others talk not at all. Managing turns makes sure everyone talks.

Erin was also clear about how the first partner should start by providing a sentence starter: “My answer is _____ because…” This sentence starter is important for ensuring that students get right to work productively and that they get to discussing their process not just their answer.

The directions for the second partner to agree/disagree were equally clear. She taught them how to agree or disagree directly and respectfully. One person on our team noted after watching Erin’s class, “The simplicity of her directions is elegant because it places the emphasis on student conversation.”

Despite the clarity of her directions, Erin checked in one last time (with hand raising for responsibilities and Call and Response for sentence starters) to be sure students were clear on the directions—because this part of the system was new, and because it’s so important to her overall culture.

Erin brought her class back together with a simple, 3-2-1. She respectfully allowed students to wrap up their conversations, and then efficiently began her whole group review.

Something else struck us just as forcefully, though, when she called on a student to report back on what she and her partner talked about. The student, Alexis, raised her hand to share the content of her discussion: “I realize that I got it wrong because my neighbor told me that I’d missed a zero [in evaluating the problem].” With a chuckle, she happily admitted– to her teacher and all of her peers– that she’d gotten the answer wrong at first, and it wasn’t until her partner pointed out her error that she could understand where she’d made her mistake.

You can read the whole thing here.

• The number of extremely poor people (defined as those earning less than $1 or $1.25 a day, depending on who’s counting) rose inexorably until the middle of the 20th century, then roughly stabilized for a few decades. Since the 1990s, the number of poor has plummeted.

• In 1990, more than 12 million children died before the age of 5; this toll has since dropped by more than half.

• More kids than ever are becoming educated, especially girls. In the 1980s, only half of girls in developing countries completed elementary school; now, 80 percent do.

The most important thing going on in the world today is something we almost never cover: a rapid decline in poverty, illiteracy and disease.

Granted, some 16,000 children still die unnecessarily each day. It’s maddening in my travels to watch children dying simply because they were born in the wrong place at the wrong time.

But one reason for our current complacency is a feeling that poverty is inevitable — and that’s unwarranted.

The world’s best-kept secret is that we live at a historic inflection point when extreme poverty is retreating.

Read the whole thing here.

Cato Institute adds:

But, where did all the progress that Kristof talks about come from?

The Homo sapiens has been on this earth for 200,000 years. For 99.9% of that time, we lived in ignorance, poverty and misery. What has changed? Reading the NYT, the reader is left with the impression that “good stuff,” like manna from heaven, suddenly was conjured up out of thin air.

Not so. The key to the improvements in the lives of ordinary people over the last 200 years were industrialization and trade, which generated historically unprecedented rates of growth. And the importance of growth cannot be overemphasized. There is not a single example of a country emerging from widespread poverty without sustained economic growth. As University of Oxford Professor Paul Collier writes, “Growth is not a cure-all, but lack of growth is a kill-all.”

As a reader of Kristof’s column, I was not left with the impression that this progress was conjured up out of thin air.

But I agree with Cato that industrialization and trade are important drivers here.

I think private philanthropy has helped, particularly in health, and it would seem that Cato and Kristof agree here too.

I think getting more girls to attend school is perhaps less driven by industrialization. Some progress on supply-side issues and demand-side issues (like social norms). World Bank, maybe even the oft-maligned (by me included) UN, has likely had some role here influencing policy.

Unlike progress in school enrollment, however, I don’t think there is much good news on actual learning. I.e., more kids are spending more days in school (particularly girls) — good! But grimmer news on how much they learn (boys and girls) once in school (bad).

Just like in the USA, where policy can’t easily pierce schools to the day-to-day classroom reality of kids and teachers, so too in poor nations.

The 3 key limiting factors from inside a typical school in the developing world:

a. Strong strong strong pedagogical preference for “teacher lectures and writes some stuff on the board, while kids sit still, and copy notes.”

We’ve made progress on changing this at Bridge.

b. Virtually no reading of good fiction (by any definition) or non-textbook non-fiction. Related: If they do read, they typically can’t read and summarize the key ideas in their own words…it’s not even what they’re instinctively trying to do (to form meaning as they read, which until 2013 I thought was an intuitive thing to try to do, but I no longer think that).

We’re in early phases of trying to create some breakthroughs here.

c. Kids frequently don’t know how to form meaningful questions of their own, let alone ask them out loud.

This puzzle needs a lot more work. I’m interested in anyone who has seen breakthroughs here. Our early efforts have “bounced off” (unlike “a” above, where we’ve had success). We’re gearing up (resources permitting) for a new dive here in 2016.

In Delhi, Teach For India fellow Pooja Chopra taught 30 fourth-grade girls in a poor settlement. As she visited with her students and their families, she learned that, far too often, girls in the community were not expected to reach the highest levels of education.

Pooja oriented her teaching toward helping her students realize their dreams through education. She set out to foster their personal leadership and to ensure they would always be able to learn, regardless of circumstances. Pooja broke the class up into groups, allowing each group to choose its leader and set its rules. Within each group, every girl was assigned a buddy, whose learning she was also responsible for. Pooja guided the students through their lessons, made sure that no group or pair fell behind, and gave them the agency to debate questions and determine their answers independently.

As Pooja developed relationships with the girls’ parents, she discovered that many of the mothers had no financial means to support their daughters’ educations. To enable them to partner in their daughters’ schooling, she helped them launch a small tailoring business. Inside and outside of her classroom, Pooja’s leadership is setting her students on new life paths. By the end of the school year, her students showed an incredible 1.7 years of growth and were exhibiting new confidence, initiative, and independence.

You can read the whole thing here.

Teach For India, just like Teach For America, has at least 2 big effects:

a. The direct effect on kids when a teacher like Pooja helps 30 kids.

b. The larger effect when lots of Teach For India alumni create an education reform movement that is focused on school quality — the outcomes (what kids learn), rather than the inputs (what is spent, or what philosophy is used).

With Bridge now recruiting for a team to launch our first schools in India, it’s easy to see what a powerful force the Teach For India alumni are — they’re results-driven, humble.

Meanwhile, it’s rainy season here in Nairobi. But I didn’t expect the hailstones we got yesterday!

Kids were preparing for midterm exams. Geordie (who took the photos) and I spent some time visiting with teachers, trying to gauge how it was going. Came away with a long list of notes on little things we can do better in terms of curriculum.

Little things can add up to big things!

Omitted photo: Back at the house, my colleague Carol had left out 2 bananas by the window sill, to ripen. They were stolen by a passing monkey — no joke. Meanwhile, I stole some bottled water from another colleague. Monkey see, monkey do.

A few news stories about Bridge.

1. CNBC video here about our co-founders.

Accompanying article

2. Fast Company here about:

The Girl Effect Accelerator, a partnership between the Nike Foundation and Unreasonable Group, has a singular mission: to scale up projects in the developing world that benefit girls in poverty. Launched in November 2014, the two-week program offered mentorship, financing, and networking to 10 for-profit projects, with the ultimate goal of rapidly scaling them up.

The Girl Effect Accelerator is an investor in Bridge.

3. Last, the Bill and Melinda Gates 2015 Annual Letter has a photo of Bridge (they’re investors). It’s worth reading for a broad view of progress and challenges in the developing world (as well as USA ed reform).

I noticed today she co-wrote an Op-Ed for the Indian Express, with colleague Rukmini Banerji. It’s called Let’s Remake The Classroom.

The basic idea?

Take all the kids in Grades 3, 4, 5; reorganize them into reading groups and math groups for some of the school day; teach lessons slated to the “right level” of the kids.

That way, a top 3rd grader is with other strong pupils getting “the right level” of challenge. Similarly, a struggling 5th grader would have an hour to be with other strugglers, and therefore also “at the right level.”

In the US context, this is called tracking, and much ink has been spilled (including by me) to argue the merits and challenges with this approach.

In the Poverty Action Lab, Duflo and her colleagues don’t debate things so much as they empirically measure them.

From the Indian Express:

Two recently concluded randomised evaluations of “teaching at the right level” in Haryana and Uttar Pradesh provide promising insights into how this model can be scaled up successfully within a government school system.

In Haryana, 400 schools participated in an intervention implemented by the government and supported by Pratham in 2012-13. Students enrolled in Classes III, IV and V in 200 of these schools were randomly assigned to this model. Students in the selected schools were re-organised by their reading level and taught for one hour each day using level-appropriate materials and methods. The programme was implemented by trained government teachers during school hours, while block- and cluster-level education officials monitored implementation and provided ongoing academic support to teachers. At the start of this programme, only 34 per cent of students were able to read a simple paragraph or story. Upon completion, this number jumped to 53 per cent for students in schools that participated as compared to only 48 per cent for students in schools that did not participate.

Bridge team — thoughts on this? Something we might experiment with to see if we can reproduce those gains for our pupils?

Title? “How standard is a standard deviation? A cautionary note on using SDs to compare across impact evaluations in education.”

C’mon. If that doesn’t get you excited, then you’re not geeky enough.

Singh writes:

In this post, I explore some of the reasons why such comparisons might be flawed and what we might do to move towards less fragile metrics.

Dispersion is not constant across populations

The most fundamental issue with such comparisons is that standard deviations are merely a measure of dispersion – and this is not constant across samples. So an intervention delivering the same absolute increment in learning would look less effective in a context with high variance in test scores than in another with low variance. These differences can be important. In the 2012 PISA assessment in math, the SD of test scores was 75% lower in Costa Rica than in Taiwan. On the same test and at the same age. So, like David’s hypothetical example in his post, the impact of an identically-effective (in absolute terms) intervention in Costa Rica will look 75% larger than in Taiwan.

More worryingly, it is possible that…

Read the whole thing here.

Also an idea from Justin Sandefur in the comments:

A modest proposal:

All we need to do is gather up all the disparate and uncoordinated exams being administered by researchers around the world. Get a bunch of kids, ideally from diverse backgrounds, and stick ’em in a room. Make them take ALL the tests. Then link them up. That way we can compare PISA with SAT with SACMEQ with anything. Simples.

Only half joking.

They just released a report. It’s about Assessment in India. Good stuff.

MSDF has donated more than $130 million in India. But do their investments pay off for

children? That’s what they wanted to better understand.

Specifically: How can they measure student growth? Because growth is what matters. See here.

In the USA, over the past few years, state governments have taken this on. They set up good measurements of student growth. That data has been a boon for all sorts of analysts, to judge “what works and what doesn’t.” For example, the average charter school does not generate much student growth compared to traditional schools. However, there is a particular segment of charters that generates huge growth for kids, often called “No Excuses” schools.

Anyway, growth metrics in the USA face political opposition. For one thing, annual growth measurements allows individual teachers, strong and weak, to be identified. The status quo (99%+ of US teachers rated “satisfactory”) has appeal in certain quarters. So there is an unusual alliance brewing in the US, where traditional political enemies — teachers unions and Republicans — may be joining forces in 2015 to block the collection of annual growth data. But I digress. Back to India.

So a foundation like MSDF, which in its USA reform work does not have to invent its own set of assessments, does have to in the India context. Or they can’t easily learn as an organization.

They set up a common test — for Grade 3 5 7 9 in English and Math, with a baseline and endline each year. If you’re a grantee, you must participate (if I’m reading the report correctly).

As a result, they learn about “What works.”

For example, imagine Grantee 7 and Grantee 1. In absolute terms, Grantee 7 kids score 545 on an exam, and Grantee 1 scores 515. Grantee 7 is better. Duh.

Wait a minute.

With the new “Growth” data, MSDF learns Grantee 1 is making large gains. 460 to 515. Grantee 7 very small gains. 535 to 545.

Kaboom! Measured by Growth, Grantee 1 is doing some great things. Grantee 7 is not. Grantee 7 needs to step it up, or (presumably) MSDF would withdraw support after some years (getting some sort of chance to change its approach).

At Bridge, we see things the same way as MSDF. With our “Academic” hats on, we care only about Growth.

Now that’s not the only “hat” we have. Our parents care about Absolute Scores. That is true everywhere, from Kenya to India to Americans in wealthy suburbs. Parents all mistake “high absolute scores” as a proxy for “good schools.” The whole notion of Student Growth is a pretty new concept. It will take time for this concept to become sticky with parents, if it ever does. And since we care about our parents at Bridge, we care about what they care about.

Now back to MSDF. A couple anecdotes:

*They learned their particular grantees doing Academic Coaching to teachers (after school) seem to have high gains for math (for kids). I had the opportunity to talk some months ago to one of their staffers about this work, and I’d like to learn even more.

*One grantee, KEF which works on leadership training, was getting low growth. One cause: these schools weren’t particularly good at teaching basic skills. Moreover, their previous math test didn’t assess these “simple skills” very well. So they got a new test; then they changed leadership training to help school headmasters push a focus on these skills, and how to use data to improve this type of instruction. Evidently, growth went up. That’s a victory for the kids they serve.

The KEF example reminds me of issues I faced at Match Charter Schools back in Boston. The Massachusetts state exam doesn’t test simple fact recall like “4 times 7” for 3rd graders (the first age at which kids are tested). They prefer to measure “higher order skill.” Of course “higher order skill” is tough to do if you can’t do the easy stuff. See here.

By contrast, at Bridge we use a test called EGMA that was commissioned by the US Government (USAID). It does start with the (appropriate) easy stuff, like counting, and moves up from there. I actually think EGMA would be valuable in the USA context. Oh well.

My only quibble with the MSDF report was this. (But I may be reading it wrong).

If the dip from 500 to 405 is actually anywhere close to one standard deviation, and then the rebound looks to be something like 405 to 455, that’s still very concerning! A half standard deviation worse off in language is quite large. But who knows what the full story is here; there’s nothing in the report besides that graphic.

Anyway, if you dig ed reform in the developing world, it’s worth reading the report.

The director, Janet Mountain, writes:

We’ve learned again and again and in all the fields we’ve worked in that you can’t manage (much less improve upon) what you can’t accurately measure. No one can. So, for the last seven years, our India team has worked to devise a reliable measurement system that helps us better manage our work.

We hope it also explains both why such a framework matters and how it can enable transformation within the classroom, throughout schools, and across cities and even states.

I 100% agree. My fondest technocratic wish for Kenyan (and soon Ugandan and Nigerian, and Indian in 2016) education policy is that they’d follow the recommendation of Peter Musyoka and measure student growth. It’s the only way to discover “what works.”