Far less well-researched, however, is the issue of sorting within schools – for example, whether teachers with certain characteristics are assigned to classes with different students than their colleagues in the same school. In addition to the obvious fact that which teachers are in front of which students every day is important, this question bears on a few major issues in education policy today. For example, there is evidence that teacher turnover is influenced by the characteristics of the students teachers teach, which means that classroom assignments might either exacerbate or mitigate mobility and attrition. In addition, teacher productivity measures such as value-added may be affected by the sorting of students into classes based on characteristics for which the models do not account, and a better understanding of the teacher/student matching process could help inform this issue.

A recent article, which was published in the journal Sociology of Education, sheds light on these topics with a very interesting look at the distribution of students across teachers’ classrooms in Miami-Dade between 2003-04 and 2010-11. The authors’ primary question is: Are certain characteristics, most notably race/ethnicity, gender, experience, or pre-service qualifications (e.g., SAT scores), associated with assignment to higher or lower-scoring students among teachers in the same school, grade, and year? Although this analysis covers just one district, and focuses on a specific set of student and teacher characteristics, it’s a big step forward.

Needless to say, one should not examine the association between student performance and any of these traits in isolation. For one thing, they are all interrelated– e.g., if male teachers, on average, are more experienced than their female colleagues, differences between male and female teachers in the performance of their assigned students may be partially due to the underlying differences in experience (see this 2006 analysis of within-school sorting in North Carolina). In addition, looking at school- and classroom-level factors, such as student characteristics or accountability pressure, may provide additional insight regarding contextual variation in assignment processes.

Accordingly, the authors of this paper fit multiple models using both administrative and survey data to try and unpack some of these relationships. Theirs is probably the most detailed analysis available on this topic.

In the models using only the administrative dataset (i.e., just race, gender, experience, and degree), they find that less experienced, female and minority teachers are more likely to be assigned to classes with lower-scoring students vis-à-vis more senior, white male colleagues, all else being equal. These results are substantively important and interesting, and have been covered elsewhere (see Stephen Sawchuk’s well-done article). But there’s quite a bit more to these relationships than the “overall” findings.

With regard to the “gender gap” – i.e., the difference in test-based performance between students assigned to male versus female teachers – the estimated discrepancy is really quite small, and basically “disappears” when one accounts for the greater presence of female teachers in special education. In other words, special education students tend to score a bit lower on tests, and female teachers are more likely to specialize in special education; so, when the authors control for this, the small estimated gap is erased (actually, reversed).

Discrepancies between white and minority teachers and between more and less experienced teachers are substantially larger. For instance, on average, black and Hispanic teachers are assigned to classes with more lower-scoring students than are white colleagues teaching in the same school and grade. The magnitude of this difference is educationally meaningful. A large portion of this gap, however, is “explained” by the fact that minority teachers tend to be assigned more poor and minority students, and these students tend to score lower than white students from higher-income families (see this paper for more on this phenomenon). Controlling for classroom composition reduces the estimated gaps to the point where they’re rather modest.

And the differences are further reduced when additional teacher characteristics are added to the model (using the survey dataset), including SAT scores, the selectivity of their undergraduate institutions, and whether teachers ever served in leadership roles. When all these factors are included in the model, assignment differences by teachers’ race/ethnicity are mostly explained away.

Finally, it’s definitely worth noting that these gaps are larger in schools with white (versus minority) principals, and that lower-scoring students are more likely to be assigned to black teachers who work in schools with a larger proportion of white teachers.

The findings for experience require similar disentangling. The relationship between years of service and student performance is positive – i.e., on average, higher-scoring students are assigned to more senior teachers. Some experience-based gap persists in basically every specification the authors report, but it does vary in magnitude.

Teachers with between 2-7 years of experience in the district are assigned students who, on the whole, do not score discernibly higher than those assigned to first-year teachers. By contrast, the scores of students assigned to teachers with between 10-20 years on the job tend to be substantially higher vis-à-vis the students of first-year teachers, even when the relationship is modeled as non-linear, and also when the researchers follow the same teachers over time to see if the associations are “real” and not just a result of differential attrition. This is consistent with the limited prior research on within-school sorting – e.g., this analysis of data from Florida and this study, which uses a nationally-representative dataset.

There are three interesting notes about these experience-based gaps. First, they are not larger in middle and high schools compared with elementary schools, which is counterintuitive insofar as “tracking” is generally seen as more prevalent in the former versus the latter. Second, the relationship between experience and student performance is significantly stronger in schools with more senior teachers, but weaker in schools where larger proportions of students are not proficient (i.e., those that may be experiencing accountability pressure, which may influence classroom assignments). Third, there is some indication in the results that experience in the district is less important than how long teachers have served in their specific school.

So, overall, there does appear to be systematic sorting of teachers within schools by measurable characteristics such as gender, race/ethnicity and experience (as well as, by the way, whether or not teachers have ever served as a professional development instructor and the selectivity of their undergraduate colleges).

This analysis is descriptive, and there are a variety of possible ways to explain or interpret its many findings. On the whole, there appears to be a rather complicated web of factors underlying teacher assignment, including teacher, principal and school characteristics. In other words, teachers and principals (and, in many cases, perhaps parents as well) consider a bunch of different factors when sorting students into classes, and this process also varies by context (e.g., by school).

For instance, as the authors note, the fact that more senior teachers are given classes with higher-scoring students may indicate that advanced students are being assigned to teachers with the stronger content knowledge necessary to teach advanced classes, whereas, in other cases, it may be a result of teachers accruing influence and building relationships as they spend more years in the same school.

In any case, efforts to understand what’s going on with classroom assignments bear directly on issues, such as teacher turnover and the use of value-added estimates in evaluations and other personnel policies. As just one example, the fact that novice teachers tend to be assigned lower-scoring students may exacerbate new teacher attrition and mobility. Analyses such as this one are therefore important, and will hopefully spur further examination, especially given the fact that new evaluations and other policies (e.g., ramping up school accountability systems) may influence the teacher/student matching process going forward.

- Matt Di Carlo

A couple of months ago, I warned, “We cannot possibly continue to move solely in the direction of ‘college and career readiness’ in History & Social Studies education without ensuring that ‘civic’ readiness is valued equally.” While our struggle continues in New York State, the Council of Chief State School Officers (CCSSO) took an extremely promising first step towards assuaging my fears with the release of The College, Career, and Civic Life (C3) Framework for Social Studies State Standards. This document was intended for a targeted review by certain groups last month. Copies, however, are not difficult to find by searching.

Simply put, the proposed C3 Framework is brilliant. It is exactly what our nation needs to ensure civic life and participation is properly valued, and it is what the Social Studies teaching profession needs to ensure our discipline retains its unique and essential role within our education system. It is brilliant in its conception, its modesty and its usefulness as a document to inform policy and practice.

The C3 describes itself as a document that provides “states with voluntary guidance for upgrading existing social studies standards… [which] aims to support states in creating standards that prepare young people for effective and successful participation in college, careers, and civic life.” It is not another set of Common Core standards, but rather a framework to guide states as they seek to upgrade existing state Social Studies standards to align them with the Common Core. The work to create the document was led by Kathy Swan, a professor at the University of Kentucky, and involved representatives from state education agencies and from various social studies organizations, overseen by the CCSSO.

The primary brilliance of the the C3 is that it moves beyond the over-simplified debates of content vs. skills (the discourse in which the Core Knowledge Blog made a misguided attempt to engage with my previous piece for the Shanker Blog), and instead recognizes Civics as more than the sum of those two parts. Rather, it recognizes that Civics is about dispositions that lead to action. For students, this means that they will not only learn the story of history or explanations of economics and government. Nor does it mean, as many fear, that social studies courses will just become another site of reading and writing instruction, void of content or understanding, to improve students’ scores on Common Core-aligned exams. It means that as students’ develop skills, gain knowledge, and uncover understandings, that these gains will come from a position of inquiry, and will be applied, in hopefully meaningful ways, to the issues and audiences students’ face in their lives.

According to the Framework, to act with civic responsibility:

students need the intellectual power to recognize societal problems; ask good questions and develop robust investigations into them; consider possible solutions and consequences; separate evidence-based claims from parochial opinions; and communicate and act upon what they learn. And most importantly, they must possess the capability and commitment to repeat that process as long as is necessary.

The C3 framework begins with the act of questioning, and ends with the act of sharing knowledge and taking informed action. In between, it offers clear guidelines for disciplinary thinking and research. It is particularly strong in its definition of historical thinking and recognition of history as “interpretive.” The Framework describes these four stages as the “arc” of social studies. By repeating this arc of inquiry multiple times in a grade, year after year, it will create more than civic knowledge and capacities; it will create a civic disposition.

The C3 is also intelligent for what it is not: It’s not a list of content that students should learn. Those decisions are smartly left to the states, and I hope that states will leave those decisions to school communities. Any attempt to do otherwise would lead to a lowest common denominator approach, which, in the end, would not serve anyone.

Finally, the C3 Framework is very useful document. It clearly shows its own connections with the Common Core. It lucidly articulates expectations of performance for each part of each dimension at different levels. And it documents the research behind its work and recommendations. Kathy Swan and her team deserve high praise for this contribution to our field.

However, the framework is not perfect, which is not at all surprising, as it is just a draft. Its description of Civics and Political Institutions focuses too much on government, and would improve with additional emphasis on civil society and types of non-governmental civic institutions – e.g., churches, advocacy groups, non-profits, etc.

In addition, the draft’s expectations for Economic Decision Making would be stronger if it portrayed risks and rewards as dynamic rather than static considerations – that is, as factors that change over time. I hope the process of targeted review will strengthen the document in other small ways, without comprising its high quality and overall vision.

The C3 Framework represents a significant step forward for Social Studies teaching and our country’s civic life. Teachers and caring citizens should support its implementation.  I will be doing what I can to push New York to adopt it.

- Stephen Lazar

Every year, a few major media outlets publish high school rankings. Most recently, Newsweek (in partnership with The Daily Beast) issued its annual list of the “nation’s best high schools.” Their general approach to this task seems quite defensible: To find the high schools that “best prepare students for college.”

The rankings are calculated using six measures: graduation rate (25 percent); college acceptance rate (25); AP/IB/AICE tests taken per student (25); average SAT/ACT score (10); average AP/IB/AICE score (10); and the percentage of students enrolled in at least one AP/IB/AICE course (5).

Needless to say, even the most rigorous, sophisticated measures of school performance will be imperfect at best, and the methods behind these lists have been subject to endless scrutiny. However, let’s take a quick look at three potentially problematic issues with the Newsweek rankings, how the results might be interpreted, and how the system compares with that published by U.S. News and World Report.

Self-reported data. The data for Newsweek’s rankings come from a survey, in which high schools report their results on the six measures above (as well as, presumably, some other basic information, such as enrollment). Self-reported data almost always entail comparability and consistency issues. The methodology document notes that the submissions were “screened to ensure that the data met several parameters of logic and consistency,” and that anomalies were identified and the schools contacted for verification. So, this is probably not a big deal, but it’s worth mentioning briefly.

Partial, self-selected sample. Newsweek sent their survey to roughly 5,000 schools (I couldn’t find any description of how they were chosen). Around 2,500 responded, and these are the schools included in the rankings. There are over 20,000 public high schools in the U.S. (any one them, surveyed or not, can submit their data, but it’s not clear how many of the 2,500 were unsolicited). It’s therefore more than a little strange to call this list “the nation’s best high schools” when it seems to have considered only a small portion of the nation’s high schools, and a non-random portion at that – the schools that were surveyed and/or responded may differ in “performance” from those that weren’t surveyed or didn’t respond. 

Inappropriate interpretation of measures. The six indicators that comprise the rankings are all measures of student performance, not school performance (a distinction we’ve discussed here many times). Schools vary widely in the students they enroll. Every year, some high schools enroll incoming cohorts that are way ahead, whereas other schools must continually catch their students up. To be clear, schools can have a substantial impact on these outcomes, but the raw statistics, such as graduation and college acceptance rates, are predominantly a function of student background and prior schooling inputs (e.g., elementary/middle school effectiveness). Thus, the Newsweek rankings tell you far more about the students enrolled in a high school than the school’s actual impact on those results.*

In other words, Newsweek is not really ranking the high schools that “best prepare their students for college” as much as they’re ranking the high schools whose students are best prepared for college (or, more accurately, the high schools whose students are best prepared among the 2,500 or so that responded to the survey). This is one big reason why the top of Newsweek’s list is dominated by high schools that are either selective (e.g., magnets) or located in more affluent neighborhoods (i.e., have low free/reduced-price lunch eligibility rates).

To reiterate, even the best measures are highly imprecise and subject to all sorts of bias; isolating schools’ contribution to measured outcomes is extremely difficult. In addition, the utility of school performance measures can vary quite a bit depending on who’s using them and for what purpose. Rankings, including Newsweek’s, that are not quite appropriate for use in a formal accountability system might still be useful to, say, a parent choosing schools for his or her children (i.e., parents have an interest in their children being surrounded by high-performing peers).

That said, some publishers of these “best high schools” take a different, in many respects more thorough approach. Most notably, the U.S. News and World Report rankings, the analysis for which is done by the American Institutes for Research, include over 18,000 high schools. They also rely on publicly-available data rather than self-reporting (which is easier given their choice of measures), and part of the process of ranking schools entails assessment of outcomes versus statistical expectations that roughly account for subsidized lunch eligibility (i.e., residual analysis), as well as consideration of the performance of “disadvantaged subgroups.” This represents a rough attempt to address differences in the students these schools serve (at least in the first two stages of the three-stage process).

To put it mildly, the U.S. News rankings still require very cautious interpretation (for several reasons, a couple of which pertain to Newsweek’s as well). Nevertheless, given the considerable constraints, it is a fairly well-designed system (and one that required a lot of groundwork). None of these “best high school” lists is anywhere near perfect, but some are arguably better than others.

Overall, though, the most important point to bear in mind is that all of these rankings are potentially useful so long as they are presented and interpreted properly. First and foremost, in the case of the Newsweek or similarly constituted lists, there should be much stronger warnings about the sample, which includes only a fraction of the nation’s high schools (and a self-selected fraction at that). It’s not enough to simply report how many schools were surveyed and how many responded; that should be put in context.

In addition, the results of these systems, like all performance metrics, are to varying degrees confounded by the observed and unobserved characteristics of the students who attend the schools (as well as other factors). Depending on how one conceptualizes school effectiveness, this might be considered very serious bias, and so, at the very least, a more prominent discussion of these issues — one that is accessible to and likely to be seen by the average reader — might go a long way toward ensuring better interpretation of the rankings.

- Matt Di Carlo

*****

* To their credit, Newsweek did at least standardize the measures before combining them, which is more than one can say for most states’ school rating systems (and for NCLB).

NCTQ looks at 41 of the 50 largest districts in the U.S. (i.e., all but nine responded to the survey). Between 2008-09 and 2011-12, four out of five of these districts froze pay at least once. As would be expected, districts did so in different ways – sometimes by freezing step increases (or awarding them without associated raises), sometimes via lower (or no) cost of living adjustments, etc. It’s compelling evidence that public school teachers, like most U.S. workers, have felt the pain from the recession. This is useful information (also check out NCTQ’s TR3 database, a terrific resource).

There are, however, a couple of points worth mentioning about salary schedules, which may seem picky (or even obvious), but they do bear on the data presented in this report.

NCTQ (properly) calculates within-lane schedule raises as a function of two “components”: A salary increase from experience “increments” (teachers move up a step on the scale); and cost of living adjustments, or COLAs (typically, the entire salary schedule increases by a percentage every year).

In the report, these raises seem to be averaged across all the steps on the schedule for each district, which NCTQ calls the “average annual teacher raise.” They even compare the “average teacher raise” with the increase in median earnings among other occupations during this time period.*

There is, however, a very important distinction between the “average teacher raise” and the average raise on a given salary schedule. The basic issue here is that teachers are not distributed equally across the steps on the salary schedule. Or, put differently, each step does not “contain” the same number of teachers.**

For example, most basically, it is not uncommon for districts to have a fairly large proportion of teachers on the “top step” of the schedule (for their particular degree, or “lane”), particularly in districts in which there are fewer steps on the salary schedule – i.e., teachers reach the top step more quickly. These teachers do not receive step increases anymore, only COLAs (though some districts also grant “longevity” raises at points after teachers reach the top step). In these districts, the “average raise” may to some extent overstate how much of an increase teachers are actually getting.

Similarly, the raises for experience might be small at first and get progressively larger as one moves up the steps, or vice-versa. To the degree teachers are clustered in one “end” of the schedule rather than the other (or in one lane, if the lanes vary as well), the “average annual teacher raise” may be sending misleading information about what’s going on with the typical teacher’s pay in any given district. In other words, if the biggest raises occur in the later steps, and most teachers are located in the later steps, they are actually getting larger raises than the schedule average would suggest.

Finally, one should use caution in making inter-district comparisons. For instance, the 41 districts are ranked in terms of the “average teacher raise” they gave between 2008-09 and 2011-12. Some of them were negative, a few close to zero, while most were positive but rather small.

Even if one ignores the aforementioned issue of the distribution of teachers by experience (and it’s of course relevant here too), schedules vary widely in the number of steps they contain. In some districts, teachers reach the top step after just 7-10 years, whereas in others, it can take as long as 25 years or more. As a rule, schedules with fewer steps will tend to give larger proportional raises. It is therefore problematic to compare districts in terms of the “average schedule raise” without accounting for differences in the number of steps.

Overall, salary schedules transmit a great deal of information of how much teachers can earn in a given district, but you need to be a little more careful about conclusions regarding how much they do earn, on average. To be clear, there’s not much NCTQ could have done to address the issues discussed above, since it’s difficult to get data on the distribution of teachers across steps and lanes. But they are important to keep in mind.

That said, this report is useful, and it’s more than appropriate to conclude, based on the data, that teachers’ raises slowed down during these years, at least in these 41 districts.

- Matt Di Carlo

*****

* This general interpretation of the results was also repeated in the New York Times, among other places.

** As another way to think about this, let’s say you calculated the average of all the amounts, or “cells” (step/lane combinations), in a salary schedule. You wouldn’t call this the “average teacher salary” in the district, because some of those cells contain many teachers (e.g., top step of MA lane), whereas a bunch contain few or even none.

Now, I personally am not opposed to using these estimates in evaluations and other personnel policies, but I certainly understand opponents’ skepticism. For one thing, there are some states and districts in which design and implementation has been somewhat careless, and, in these situations, I very much share the skepticism. Moreover, the common argument that evaluations, in order to be “meaningful,” must consist of value-added measures in a heavily-weighted role (e.g., 45-50 percent) is, in my view, unsupportable.

All that said, calling value-added “junk science” completely obscures the important issues. The real questions here are less about the merits of the models per se than how they’re being used.

If value-added is “junk science” regardless of how it’s employed, then a fairly large chunk of social scientific research is “junk science.” If that’s your opinion, then okay – you’re entitled to it – but it’s not very compelling, at least in my (admittedly biased) view.

And those who hold this opinion will find that their options for using evidence to support their policy views are extremely limited. They should, for instance, cease citing the CREDO charter school study, which uses a somewhat similar approach – i.e., put simply, judging effectiveness by statistical comparison with schools serving similar students. In this sense, CREDO (and most of the charter school literature) must also be called “junk science.”

Furthermore, what is the case against calling classroom observations “junk science” too? Even when done properly — by well-trained observers observing multiple times throughout the year, observation scores also fluctuate over time and between raters, and they are subject to systematic bias (e.g., poorly-trained or vindictive principals).

You might believe that human judgment is a better way to assess performance than analyzing large-scale test score datasets, and you might be correct, but that’s just an opinion, and it hardly means that all alternative measures are “junk” no matter their policy deployment.

It’s also important to bear in mind the obvious fact that value-added has a wide range of research-related uses outside of high-stakes personnel decisions (including program evaluation). In fact, many of the conclusions from this literature are things with which few teachers would disagree – e.g., teachers vary widely in their measured performance, they improve a great deal during their first few years, etc.

In short, value-added models are what they are – sophisticated but imperfect tools that must be used properly. We can and should disagree about their proper uses, but calling the models “junk science” adds almost nothing of substance to that debate.

- Matt Di Carlo

Two events last week which might seem worlds apart provide evidence that working people around the globe are indeed tied together in King’s “single garment of destiny.”

In Texas, fourteen people died and up to 180 were injured in an explosion that obliterated a fertilizer factory and leveled the surrounding town. In Bangladesh, over 400 garment workers died when a factory building collapsed with thousands inside. Rescue and recovery operations continue to find additional Bangladeshi dead, with hundreds still missing. The human toll makes this the deadliest accident in the history of the garment industry worldwide, even before the terrible final count is known.

Neither of these terrible events was “an accident.” In both cases, factory management engaged in dangerous and reprehensible conduct, creating entirely avoidable conditions that made these events possible, even predictable.

There was a history of environmental safety violations at the Texas factory (see here,  here and here). It warehoused large amounts of ammonium nitrate fertilizer — the main ingredient in the 1995 Oklahoma federal building bombing by Timothy McVeigh — which the owners never reported this to the Department of Homeland Security, as required by law. (In 2012 alone, the plant processed 270 tons of the explosive fertilizer.) Complaints by surrounding townspeople of noxious ammonia smells seem to have been largely disregarded, even though the factory had on hand large amounts of an extremely hazardous gaseous chemical, anhydrous ammonia.

In Bangladesh, large cracks appeared in the foundation of the illegally constructed factory building the day before its collapse, leading local authorities to ask for an evacuation (see here, here  and here). The owner of the building and owners of the garment factories on the building’s upper floors refused to comply. Since the garment workers knew they would be penalized three days of unpaid labor for every missed day of work, most felt they had little choice but to enter the building and go to work. The building owner and factory owners have now been arrested.

Clearly, the managements in both settings were greedy and unscrupulous. But it would be a mistake to write off these events as that and nothing more; both reflect larger economic and political realities.

The context for the Texas factory disaster is a decades-long corporate assault on workplace safety and health in the U.S., a campaign which has grown in strength with the decline in the American labor movement. After years of deregulation, U.S. occupational safety and health laws are weak and poorly enforced; oversight relies in significant measure upon corporate self-reporting. The factory owners had told the federal Environmental Protection Agency (EPA) that there was no ”worst case scenario” risk for fire or explosion at the factory, and that proffer alone was sufficient to make it a low priority for inspections: the last federal OSHA (Occupational Safety and Health Administration) inspection was nearly thirty years ago, in 1985. After decades of defunding, OSHA can maintain only a bare bones regimen of inspections, with a triage-like focus on the most critical situations. There were also Texas agencies charged with oversight, but they communicated irregularly with each other and the federal agencies, and lacked the necessary expertise for the work–as evinced in the decision to allow a factory to process and store such dangerous and explosive chemicals in the close vicinity of schools, a nursing home, and residential housing.

The context of the Bangladesh catastrophe is especially grim. Just last November, Bangladesh was the site of another horrific mass death of 112 garment workers in a factory fire. Workers died in such large numbers because they had been locked inside the factory by management and so could not escape the fire, a scenario eerily reminiscent of the 1911 Triangle Shirtwaist fire in New York City that provided the impetus for the first meaningful occupational safety legislation in the United States. Dangerous workplace conditions are widespread in Bangladesh’s garment industry, with over 500 workers killed by factory fires since 2006.  Bangladeshi garment factories provide textbook examples of “sweatshops.”

In both Texas and Bangladesh, the international political economy has set the stage for needless human tragedy on a mass scale. The “race to the bottom” that has decimated industrial unions and undermined industrial working conditions in the United States now finds its nadir in Bangladesh, the world’s second largest exporter of apparel after China. Bangladeshi garment workers earn 18¢ an hour; with monthly earnings of about $36, it’s highly unlikely that any of these workers will ever be able to purchase a single garment she has produced. Unbelievably, these wages represent an increase over even lower pay, a concession wrested from the government in 2010 by massive worker unrest. Working conditions are also among the most unsafe and exploitative in the world, as Human Rights Watch thoroughly documented in its October 2012 report on one Bangladesh company. With some of the lowest labor costs in the world, the Bangladeshi garment industry is exporting $18 billion of garments annually, and is on the cusp of a major expansion.

To maintain the low wages that attract foreign investment, Bangladeshi law restricts the development of free and independent unions and creates “export processing zones” for the garment industry in which unions are illegal. Bangladesh’s authoritarian government treats the garment industry as a national security issue and actively suppresses worker efforts to organize. The New York Timesreported last year that “a high-level government committee monitors the garment sector and includes ranking officers from the military, the police and intelligence agencies. A new special police force patrols many industrial areas. Domestic intelligence agencies keep an eye on some labor organizers.” And according to Human Rights Watch, over a dozen Bangladeshi union leaders are now facing spurious criminal charges.

The brutality of the repression aimed at Bangladeshi garment workers takes on a human face in the story of Aminul Islam, a key garment worker organizer and labor leader. Islam “was vocal, and he was fearless,” Babul Akhter, the head of the nonprofit labor group where Islam worked, told the New York Times. “Whenever workers came to him, he took them as his own case, as if it was his own pain.”

And that made him enemies. A little more than a year ago, he was tortured and killed. To date, his murder remains unsolved.

Islam’s troubles started when his fellow garment workers elected him to represent their grievances. Fired from his job, he became a full-time worker advocate at the Bangladesh Center for Workers Solidarity. In 2010, state security revoked the Center’s registration and arrested Islam’s two bosses, charging them with inciting workers to riot. Police wire-tapped Islam’s phone and regularly harassed him. In June 2010, he was kidnapped and beaten by domestic intelligence agents, who threatened his life and the lives of his family members. In March 2012, a dozen members of the Industrial Police took Islam in for questioning, and on April 4 he disappeared. Two days later his corpse was found. His knees had been smashed, his feet had been broken, and a hole had been drilled into one knee. He had bled to death.

Torture and extrajudicial killings by police are common in Bangladesh and, as reported by both the International Crisis Group and Amnesty International, are often done at the behest of wealthy individuals and business executives. Despite the focus of the international human rights community, Islam’s murder remains unsolved after a year, leading many to suspect that the police forces had a hand in his demise.

Who profits from this state of affairs?

The factories in the collapsed Bangladesh building were producing garments for Wal-Mart, The Children’s Place, Dress Barn, Primark, Mango, Benetton, C&A, KIK, Cato Fashions, Matalan, and Bon Marché, among others. Wal-Mart garments were also being produced by the company that locked in its workers last November, leading to the death of 112 workers in a factory fire. The Gap, Calvin Klein, H&M, Target, and Tommy Hilfiger are other well-known brands and retailers that also manufacture their garments in Bangladeshi factories. In the classic “race to the bottom” production model in use in Bangladesh, China, and other low-wage settings enforced by anti-union authoritarian governments, these Western firms outsource their actual production to foreign-owned factories, such as those in the collapsed Bangladesh building, seeking to create plausible deniability about what is entailed in the manufacture of their products. 

In the days since the building collapse, corporate public relations machines have been in overdrive, seeking to distance their firms from events in Bangladesh. But only the most gullible can believe in the ignorance or innocence of those at the helm of these corporations. It is worth noting, for example, that in the wake of the November 2012 factory fire, Wal-Mart and Sears, who profited from the lax conditions, refused to provide compensation to the victims and their families who bore the cost. Wal-Mart was also one of a group of corporations that rejected a 2011 plan put together by labor advocacy groups and international and Bangladeshi unions to create an independent inspectorate “to oversee all factories in Bangladesh, with powers to shut down unsafe facilities as part of a legally binding contract signed by suppliers, customers and unions.” Wal-Mart’s representative to a meeting on the plan declared that it was “not financially feasible … to make (the) investments” required to implement the inspectorate. In other words, Wal-Mart and the other corporations know full well that the low production costs they enjoy are subsidized through the blood, sweat, and lives of Bangladeshi garment workers. It simply could not happen otherwise.

Sadly, there its little new to distinguish these stories from the several decades worth of news reports on the ways in which workers worldwide suffer from the greed and indifference of multinational corporations. But my point is this: We ignore these stories at our own peril.

The global “race to the bottom” in which multinational corporations outsource the manufacture of products to repressive, low-wage nations has exacted a very real price on American workers as well. The decimation of American industries and American industrial unions, with the resultant decline in the income, benefits,  and health and safety protections of the current U.S. workforce, has been made possible by the outsourcing of the work they once did to, in the words of William Blake,  the ”dark satanic mills” of Bangladesh and other, similar nations. The collapsed factory in Bangladesh and the obliterated factory in Texas are artifacts of an international political economy in which power and wealth is concentrated more and more in the hands of a small elite, at the expense of working people the world over. And, as U.S. unions and the American middle class they helped to create have weakened, states have become more prone to engage in their own “race to the bottom” behaviors, attempting to lure or retain corporate investment by offering to lower wages or business taxes or relaxing the environmental and workplace oversight that could have prevented the West, Texas, explosion from happening.

That is, the future of American workers and their unions is tied, inextricably, to the future of the workers of other nations. So long as the world’s most exploited workers can be denied the right to free, independent unions that would empower them to fight for their own rights and freedoms, our rights and our unions will be in danger. So long as the workers of Bangladesh and similar nations can be harshly exploited, our living standards and our working conditions are also open to exploitation.

International solidarity is no longer simply a virtue; it has become an imperative. Without it, our “single garment of destiny” could well be a shroud.  

- Leo Casey

*****

* See Worker Safety Under Siege: Labor, Capital, and the Politics of Workplace Safety in a Deregulated World. Vern Mogensen, editor. M. E. Sharpe, 2005.

I ended my previous post by arguing that (1) if teaching is at least as valuable as research, and (2) nontenure-track (NTT) faculty teach at least as well as tenure-track (TT) faculty, then the very large pay disparities between the two classes of faculty that characterize American universities today violate a basic principle of workplace fairness: equal pay for equal work. When conditions (1) and (2) are met, then, all an institution can do to defend current practice is plead poverty: we can’t afford to do what we ourselves must acknowledge to be “the right thing.”

But what about places like the University of Michigan-Ann Arbor, where I work? Is condition (1) met in what are sometimes called “R1” universities like mine? If not, maybe big pay disparities are warranted by the fact that, in such universities, research is a much higher institutional priority than undergraduate teaching. If teaching is a low enough priority, current pay inequalities could be justified by the fact that NTT faculty are not paid to do research and publishing – even though many of them do it – and, conversely, that most TT faculty pay is for their research and publishing, rather than their teaching.

We can estimate what we might call the “implicit” value of teaching at a place like UM-AA, by starting with the unrealistic assumption that TT and NTT faculty are paid the same to teach a course. At my university, the median full-time NTT faculty member, if they start (as most do) as a Lecturer I, will be paid an average of $38,289 to teach six courses. That is about $6,381 per course. The median Assistant Professor will be paid $80,361 to teach three courses. Three courses at $6,381 per course is about $19,144. This implies that the value of the median Assistant Professor’s non-teaching work (mainly research, though there is some service work here too) is $80,361-$19,144 = $61,217.

Survey work conducted for a recent study by my union – Teaching Quality: What the Principle of Equal Pay for Equal Work Means for Lecturer Pay at the University of Michigan (2012) – shows that TT faculty at UM-AA spend about half their time on teaching-related activities during the academic year. So an hour of TT faculty time spent on teaching is worth about 15 minutes spent on research (or service) according to our illustrative estimate of the implicit value of teaching at my institution.

Does that assessment of the relative value of undergraduate education and cutting edge research make sense for a public R1 university? I don’t think so, for two basic and inter-related reasons. First, it is radically out of whack with the way the university’s academic activities are funded — a disjuncture that is only likely to grow. Second, it is not aligned with the core mission of the public university, which says that research and teaching that contribute to the understanding and solution of pressing public problems must be prioritized over other kinds of research and teaching.

On the financial side, tuition fees – the bulk of them from undergraduates – now account for almost 68 percent of the UM-AA’s General Fund, the fund that pays for almost all academic activities (i.e., compensation for faculty and staff, internal research funding, scholarships and subsidies to students, library expenses, heat, light, building maintenance, etc). Only three significant university activities are funded without much General Fund money: the UM Hospital, our sports teams, and research that is directly supported with outside grant money. This last category is quite large – projected to be about $850 million in FY 2013 — but it is important to understand what that sum supports.

Part of it – overhead charges for the use of university infrastructure (aka “indirect cost recovery”) – goes to the General Fund, where it accounts for about 11 percent of the total revenue, as indicated in the 2008-09 estimates in the figure below (roughly $200 million out of $1.8 billion overall). The rest supports project-related expenses: extra salary for the principle investigator(s), the hiring of grad research assistants and other support staff, lab equipment and so on.

While this money enables us to support a much higher level of research activities than we could conduct without it, these grants mainly pay for research. They do not subsidize non-grant activities such as teaching, with the important exception of its contribution to financing the PhD training of graduate research assistants. On the contrary, teaching heavily subsidizes research by paying the lion’s share of TT faculty salaries, while demanding only half of their time during the academic year. This is so even in an R1s like mine, which get more outside research grant funding than all but a handful of universities.

The graph also shows that this was not always so. In the late 1960s, state transfers accounted almost two thirds of UM’s General Fund revenues. But those days are long gone, and the data make clear that the share of all revenues accounted for by tuition has been steadily rising since the late 1970s. It is not likely to shrink any time soon, even if worries about student indebtedness and accessibility to higher education restrict further growth. Given that reality, we ought to give more recognition to the contribution of undergraduate teaching to the fiscal viability of our university, and more respect to the NTT faculty, who specialize in undergraduate teaching, and account for about 30 percent of all undergraduate student credit hours on the Ann Arbor campus.

Turning to our mission as a public university, the fact that we are an R1 means that research plays a more important part in our mission than for non-R1 public universities. But we are still tasked with providing an “uncommon education for the common man,” as UM’s longest-serving President, James Burrell Angell (1871-1909) once put it, at a price that students whose parents are not rich can afford to pay without going deeply into debt. As things stand now, only about 15 percent of our undergraduate student body comes from families at or below Michigan’s median family income of about $50,000. Many from the bottom half of the state’s income distribution no longer even apply – they see the sticker price and conclude UM-AA is far beyond their reach. And many of our students – including those from families between $50,000 and $100,000 — like their counterparts around the nation, are going deep into debt to fund their time here. Student debt – most of it accumulated in the last decade, during which tuition fees have sky-rocketed – now surpasses the sum of all credit card debt in this country (see the stories by Andrew Martin in the New York Times, May 12 and 14, 2011).

Over the longer haul, unless we get much more substantial financial support from state governments, public R1s are going to have to scale back their research ambitions. This will likely happen in two ways, and signs of both are already visible: first, we will have to scale back the share of our faculty who, as members of the tenure-track research elite, are allowed to devote half of their time to doing research; and second, we will have to stop competing with richer private R1s to attract and retain the most highly prized members of this elite, across all domains of academic research.

On the first point, the latest AAUP data show that TT faculty in 2011 made up just 24 percent of the higher education faculty workforce nation-wide, down from 32 percent in 2008. UM-AA is a relatively rich university, so these trends are less pronounced: in 1995-6, Lecturers in Ann Arbor constituted just 17 percent of teaching faculty; by 2010-11, their share had increased to 24 percent, still much lower than the national average. Nonetheless, this was a 41 percent increase in the share of all UM-AA faculty who are nontenure-track.

On the second point, the gap in pay for tenured profs in public and private R1s has grown significantly in recent years: on average, private R1s can now afford to pay their faculty 25 percent or more than their public R1 competitors (Tamar, “Gap Widens for Faculty at Colleges, Report Finds,” New York Times, April 8, 2013). They can do that because they have much bigger endowments and/or are willing and able to charge their students much higher tuition fees, which is itself enabled by the fact that they do not have the same mandate to provide affordable education to working and middle class students. Yes, the top private R1s claim that anyone they admit will be provided with sufficient financial aid to attend their schools. But they do not admit nearly enough such students to fulfill the equality of higher educational opportunity mandates to which public universities must adhere.

The changes at the UM-AA, like other top-tier public R1s, are less dramatic than they have been for lower-tier publics, but we are nonetheless moving in the same direction. And we are already at the point where we can predict with some confidence that, for all but a limited number of research domains, the public R1s are going to have to cede research pre-eminence to the private R1s. The public R1s’ real choice is not whether they can halt this trend, but whether they respond to it by redefining their mission and the balance among their teaching and research priorities, or by clinging to their traditional ambition to match the private R1s in competition for research eminence across the board.

So far, we have pursued the latter path. The longer we do that, the greater the inequalities between our TT and NTT faculty we will have to create, and the greater the financial burden we will have to place on our students and their families, in order to fund this competition.

We in the public R1s need to re-think the relationship between our core mission and our priorities. We need to define more precisely the kinds of research and teaching that public universities ought to prioritize, giving primacy to the development and dissemination of knowledge that helps us to better understand and respond to our most pressing social needs. What are these?

First, we must identify and promote ways of reversing the profound inequalities of income and wealth that are undermining the quality of our democracy, delegitimizing our system of government and betraying the promise of equality of opportunity. Second, we must find ways of reducing inequalities that are compatible with making our economy ecologically sustainable. Finally, but most important of all, we must nurture our students’ commitment to democratic principles and the public good. It is they who will (or will not) turn the solutions that we discern through our research, and share through our teaching, into realities. If they emerge from our universities as careerists pursuing their self-advancement above all else, our research and teaching alike will not contribute nearly as much as they could to the public good.

We need to pay attention not just to research agendas and the content of course syllabi but also to our “hidden curriculum.” If we want our students to commit themselves to build a more just and ecologically sustainable world, we must model what that looks like in the community that we ourselves constitute and lead. Our failure to acknowledge and address the injustices associated with the creation of a major new subordinate class of nontenure-track faculty is a tacit but powerful lesson for our students. By asserting, as UM’s current Administration does, that Lecturers ought to be paid “whatever the market will bear,” without reference to the principle of equal pay for equal work, we are teaching our students that the highly unequal way in which we organize the UM’s labor market is perfectly appropriate, not only here but in the wider world. It is as if we built vast, energy-guzzling buildings while preaching conservation in courses and public statements.

To conclude, we must reconsider the priorities of the public R1 university in light of the profound historical shifts in our funding sources, and in our political, economic and ecological contexts. Among the most important changes needed is the recognizing the centrality of undergraduate teaching, and the faculty who do it, to the mission of public R1s. We must give our teaching faculty the resources they need to do a very challenging job, and the respect they deserve for doing it with excellence. From this vantage point, it makes good sense to treat an hour of faculty time devoted to teaching as equal to an hour of time devoted to research and writing. If we do that, the principle of equal pay for equal work implies that we must substantially reduce the pay inequalities between TT and NTT faculty.

- Ian Robinson

Spring 2013 marks the 30th anniversary of two landmark publications. One, an essay by E. D. Hirsch, Jr., in The American Scholar titled “Cultural Literacy,” sparked a small but steadily growing movement dedicated to educational excellence and equity. The other, A Nation at Risk, set off a firestorm by conveying fundamental truths about the inequities in our educational system with prose so melodramatic they have proven unforgettable.

In the 80s, only one leader seemed to fully grasp the importance of both of these publications: Albert Shanker. Shanker, then-president of the American Federation of Teachers, was prominent partly due to his position, and largely due to the force of his intellect. He saw that schools were in trouble. He agreed that, as stated in A Nation at Risk, “the educational foundations of our society are presently being eroded by a rising tide of mediocrity that threatens our very future as a Nation and a people. What was unimaginable a generation ago has begun to occur—others are matching and surpassing our educational attainments.”

Mediocrity is what filled the void as schools slowly retreated from teaching all children rigorous content. That retreat happened throughout the 20th century: Progressive educators’ misunderstandings of the essential role of specific, relevant knowledge in reading comprehension and critical thinking resulted in weak curricula being the norm and pockets of excellence typically being reserved for our most advantaged youth.

E. D. Hirsch was a professor who shared that misunderstanding until his own research awoke him to the (now well-established) fact that broad literacy depends on broad knowledge. Shanker was by far the most prominent educator to grasp the veracity and power of Hirsch’s work.

Rigor is the antidote to risk.

According to Richard Kahlenberg’s terrific biography of Albert Shanker, Tough Liberal,* Shanker “believed, with E. D. Hirsch, Jr., that if one really wished to be a political progressive concerned about disadvantaged kids, one needed to be an educational ‘conservative’ who stood for teaching students certain core knowledge that was essential to upward mobility in American society” (p. 10).

It was in the early 1980s, when Shanker read both A Nation at Risk and “Cultural Literacy,” that his particular form of progressivism took shape: Shanker saw that poor children needed a whole array of supports—including a traditional, rigorous curriculum that would give them all the knowledge that wealthier children get from their college-educated parents.

While virtually all education leaders panned A Nation at Risk, Shanker did not. According to Kahlenberg, Shanker’s reaction was “pivotal”:

When the … report was released … Shanker and a group of top union officials sat together and read the document. Sandra Feldman recalled: “We all had this visceral reaction to it. You know, ‘This is horrible. They’re attacking teachers.’ Everyone was watching Al to hear his response. When Al finished reading the report, he closed the book and looked up at all of us and said, ‘The report is right, and not only that, we should say that before our members.’ ” (p. 275)

Shanker did just that in a speech to members less than a week after the report came out. And then he spent the remainder of his life (he passed away in 1997) fighting for several major reforms. A few of the noteworthy ones were peer assistance and review, charter schools, and standards.

Thanks in part to Hirsch, Shanker had a very clear sense of what educational standards needed to accomplish. According to Kahlenberg:

Shanker disagreed with education-school professors who favored general thinking skills over gaining specific-content knowledge. He believed students needed both, and that John Dewey’s education theories had been misinterpreted by some “progressive” educators…. “Dewey himself was shocked when he went into some of these progressive schools and saw what was going on in his name.”

In the 1980s, Shanker became an early advocate of University of Virginia English Professor E. D. (Don) Hirsch Jr.’s argument that American students needed to be “culturally literate”—to master a body of facts that literate American’s know—in order to be successful in mainstream society. A full two years before Hirsch’s bestselling book Cultural Literacy became a phenomenon, Shanker embraced Hirsch’s view that knowing subject matter was important to reading comprehension…. “To read well you need background information that is culture-specific,” Shanker argued. Students needed to be taught Shakespeare and mythology so they could understand common cultural references.

Shanker was also taken by Hirsch’s argument that when students know particular content matter, their interest and curiosity are more likely to be aroused. A student who knows something about dinosaurs is more likely to pick up a book on dinosaurs when browsing through the library. “Subject matter,” Shanker argued, “is the life’s breath of learning.” While some “progressive” educators dismissed Hirsch’s approach as emphasizing “mere facts,” Shanker wrote thirteen separate columns mentioning Hirsch’s theory, invited Hirsch to speak at the AFT’s biennial QuEST Conference, and featured Hirsch on the cover of American Educator….

Shanker … believed that the core knowledge of the dominant culture was essential for all students to master if they wished to advance socioeconomically within the society…. Shanker argued:

Some people have been very critical of Hirsch’s proposals on the grounds that they try to impose the dominant culture on groups that would rather have their children learn their own culture. But the thrust of Hirsch’s proposal is egalitarian. He believes that by starting early and by giving all children the same core knowledge to learn, we can prevent the creation of an educational underclass…. (p. 323-324)

Despite their best efforts, neither Shanker nor Hirsch succeeded in bringing the need for knowledge-building curricula into mainstream reform efforts.

But now, the tide is finally turning.

The Common Core State Standards demand rigor—and a strong curriculum. The Common Core State Standards for English language arts and literacy, the need for a knowledge-building curriculum is plainly stated and explained:

While the Standards make references to some particular forms of content, including mythology, foundational U.S. documents, and Shakespeare, they do not—indeed, cannot—enumerate all or even most of the content that students should learn. The Standards must therefore be complemented by a well-developed, content-rich curriculum consistent with the expectations laid out in this document. (p. 6)

To build a foundation for college and career readiness, students  must read widely and deeply from among a broad range of high-quality, increasingly challenging literary and informational texts. Through extensive reading of stories, dramas, poems, and myths from diverse cultures and different time periods, students gain literary and cultural knowledge as well as familiarity with various text structures and elements. By reading texts in history/social studies, science, and other disciplines, students build a foundation of knowledge in these fields that will also give them the background to be better readers in all content areas. Students can only gain this foundation when the curriculum is intentionally and coherently structured to develop rich content knowledge within and across grades. (p. 10)

Shanker, no doubt, would applaud the effort. Hirsch certainly is. As more and more states take implementation seriously and support schools in creating the content-rich curricula they need, we all should be applauding.

- Lisa Hansel

*****

* In quoting Tough Liberal, I have not included the endnotes. 

A big part of successful policy making is unyielding attention to detail (an argument that regular readers of this blog hear often). Choices about design and implementation that may seem unimportant can play a substantial role in determining how policies play out in practice.

A new paper, co-authored by Elizabeth Davidson, Randall Reback, Jonah Rockoff and Heather Schwartz, and presented at last month’s annual conference of The Association for Education Finance and Policy, illustrates this principle vividly, and on a grand scale: With an analysis of outcomes in all 50 states during the early years of NCLB.

After a terrific summary of the law’s rules and implementation challenges, as well as some quick descriptive statistics, the paper’s main analysis is a straightforward examination of why the proportion of schools meeting AYP varied quite a bit between states. For instance, in 2003, the first year of results, 32 percent of U.S. schools failed to make AYP, but the proportion ranged from one percent in Iowa to over 80 percent in Florida.

Surprisingly, the results suggest that the primary reasons for this variation seem to have had little to do with differences in student performance. Rather, the big factors are subtle differences in rather arcane rules that each state chose during the implementation process. These decisions received little attention, yet they had a dramatic impact on the outcomes of NCLB during this time period.

As is so often the case when using the small samples of state-level datasets, the authors are limited in their modeling options (though it bears mentioning that these data were very difficult to assemble, and, by the way, are available to the public). Instead, they identify five factors that made the most substantial contributions to the variation in AYP outcomes:

1. Deviation from NCLB rules. During the early years of NCLB, a few states didn’t quite follow the law. (Note that this is the only one of the five factors that has been largely rectified.) In at least one case, such failure was due to simple human error – Iowa’s one percent AYP rate in 2003 seems partially to have been a result of a leave of absence taken by the staff member responsible for the data, who suffered an injury. In other cases, states bent the guidelines set forth in the legislation. Texas, for instance, petitioned the U.S. Department of Education for flexibility on a rule that permitted a maximum of one percent of a school’s special education students to use alternative assessments. Their petition was turned down, but they went ahead with the plan anyway and, as a result, 22 percent of Texas schools that would have failed to make AYP in the first year actually made it.
2. “Generosity” of confidence intervals. As is fairly well known, if just one of a school’s “accountable subgroups” (e.g., low-income, students with disabilities, etc.) fail to meet proficiency targets (or “safe harbor”), that entire school does not make AYP. In order to account for the inevitable fact that, in some schools, these subgroups would consist of very few tested students, NCLB allowed states to apply “confidence intervals.” Basically, these adjustments meant that smaller subgroups (i.e., those consisting of fewer tested students in a given school) would be required to meet lower targets. However, states were given flexibility in how much “leeway” they granted via these confidence intervals, and a few specified none at all. Florida, for example, did not use them, and thus a fairly large group of schools that would have made AYP had this rule been applied did not do so.
3. Different targets across grade levels. States had the option of either setting the same proficiency targets for all grades or letting their targets vary by grade (and subject). Using the former system – the same targets for all grades – basically meant that schools serving particular grade configurations would have an advantage in making AYP (if their starting rates were higher) whereas others would have a disadvantage (if their starting rates were lower). For example, Pennsylvania set uniform targets, but their high schools’ starting rates were much lower, on average, than those of elementary schools. The end result was that 27 percent of the state’s high schools failed to make AYP in 2004, compared with just 7 percent of elementary schools. 
4. Number of “accountable subgroups” and minimum sample size. As mentioned above, NCLB required schools to be held accountable for the performance of student subgroups. But states were given flexibility not only in how many subgroups they chose (and which ones), but also in setting minimum sample sizes for these subgroups to be “included” in AYP calculations. For example, schools with only a handful of students with disabilities in a given year could be exempted from having this subgroup count at all. As a rule, states that chose to include fewer subgroups in AYP, or set higher sample size requirements for their inclusion, tended to have lower failure rates, all else being equal. Once again, states varied in the choices they made, and this influenced their results. 
5. Definition of “continuous enrollment.” Finally, states had to specify the rules by which mobile students (e.g., transfers) were or were not counted toward schools’ AYP calculations. Some states set more stringent enrollment requirements than others, which meant that they excluded more students from being counted in their testing results. For instance, Wisconsin’s rules excluded students who were not enrolled in late September of 2003 (the tests were administered in November 2003). Thus, fairly large proportions of students who took the test were not counted. To the degree excluded students’ performance was different from their “continuously enrolled” peers, these choices affected failure rates.

Now, it’s important to note that all of these rules interacted with each other (as well as with other rules and factors, such as school and student characteristics) to produce outcomes.

For instance, 23 states opted to use the most generous confidence intervals (which, by itself, “inflates” AYP rates), but these states did not have appreciably higher AYP rates, on average. The reason is that they also tended to choose other options (e.g., minimum subgroup sample size requirements) and/or they exhibited characteristics (e.g., tested grades, student characteristics) that, on the whole, “cancelled out” the differences.

From this perspective, it would be fascinating to look at the results of this paper in terms of general strategies and principles reflected in the decisions states made. Such coherence may not be easy to decipher – some choices increased failure rates, whereas others were certain to lower them. States varied in their decision making structures and resources, and they only had a relatively short period of time to devise their plans (which often had to be rectified with pre-existing accountability policies).

Many state-level NCLB configurations ended up being complex, sometimes inconsistent webs of rules that reflected varying incentives and priorities. Making things worse, the ESEA waivers that most states have submitted will only result in more heterogeneity (see this paper by Morgan Polikoff and colleagues).

This is not quite how accountability systems are supposed to work, and, as the authors of this paper note, it illustrates the ever-present difficulty of finding the right balance between standardization and flexibility in policy design/implementation.

(It should be lost on no one that these issues are extremely relevant to the current efforts in over 30 states to install new principal and/or teacher evaluations.)

Finally, and perhaps most importantly, accountability systems can play a productive role in education, but this analysis demonstrates very clearly that, when it comes to the design and implementation of these systems, details matter. Seemingly trivial choices can have drastic effects on measured outcomes.

In short, phrases such as “we need to hold schools accountable” sound wonderful, but what really matters is how we do so.