The wiki/syllabus (a living, breathing document) is located HERE.

You can participate and help my students’ (and my own) learning in a couple of ways.

First, tagging…anything that you think might be related to what the course is about (see e.g. http://sites.google.com/site/edpolitics/syllabus) can be tagged with “adms707“.  I know, just about everything education-related is also about the politics of education, but that’s OK. Tag away. We will all be feeding Google Reader with a subscription to the tag and we can filter out what’s useful or not.

Second, please consider subscribing to and/or reading the blogs my students are using as reflection spaces.  You can get to individual blog posts HERE.  Or, you can see the blog URLs HERE.  Of course, comments are helpful as I want this blogging endeavor to be meaningful in a connectivist sort of way.

Thanks for considering being a part of our learning experiences this semester!

photo credit: helgasms!

Thanks to Clay Burell for granting me this opportunity.

Please join me over there for some good (and courageous) conversation.

SOURCE: Tyack, David (1974). The One Best System. Cambridge, MA: Harvard University Press. Page 286

For Wes Fryer.

Because I’m trying to steal the attention of his vast audience.

Click here to see the post if you’d like.

photo credit: AchimH

I have a doctorate in Politics and Education and when I’m asked what that means, I usually speak to a definition of politics I’ve “borrowed” (re-mixed?) from an adjunct professor with whom I took a course while in graduate school.  Dr. Dale Snauwaert, an adjunct professor at TC at the time, wrote about politics as the intersection of power and justice.  Combining my interests in the politics of education and educational technology, I’ve written much about justice and educational technology (see e.g. this article).  I have not, however, written much about power and educational technology…until now.

In my courses on the politics of education, I guide our exploration of power with two questions: (1) who has power? and (2) how is power organized/distributed?  Much has been written about who has power in the area of educational technology, though there’s more that needs to be written.  Today, though, I explore what I believe to be a major obstacle to school reform through the lens of educational technology: how power is distributed around educational technology.

Educational governance is ultimately about control and how that control is (or is not) partitioned among the various stakeholders matters immensely. I argue that in education, the system is multi-layered and overly partitioned.  I compare our educational system to onions and silos.

“The way authority is structured and exercised shapes the intellectual and moral character of the school, thereby profoundly influencing student development” (Snauwaert, 1993).

ONIONS

The U.S. education system is like an onion in that it has many levels and the more you try to peel away at those layers, the more you start to tear up.

Policy decisions are made by federal, state, and local education agencies.  Even locally, decisions are made at the district, school, department and classroom level.

In addition to aiding or hindering quality education, there are many consequences to the multilayered system, including the phenomenon of mutual adaptation (which I’ve written about here).  As McLaughlin wrote in an article in 1990 about school reform, “…it is exceedingly difficult for policy to change practice, especially across levels of government” (p. 12).

I was reminded of the onion last week at NECC, and my reflections from the conference reinforced my thinking.  The largest ed. tech. conference in the U.S. is nearly entirely classroom-focused and the conversations are nearly totally absent of policy context.  Yet, alongside NECC proper, SETDA (the umbrella organization of state education technology officers) was holding their Emerging Technologies Forum & Annual Convocation.  There was some overlap between the two events, but from my perspective, the state-level policy makers were meeting in parallel with the school and district-level folks at NECC.  Similarly, shortly after NECC, the Education Commission of the States held their annual National Forum on Educational Policy.

This sort of parellel play doesn’t advance anyone’s cause.

SILOS

Even within the same levels of decision making in education, we have a serious silo problem.  Like policymakers across levels of governance, educators within any given level exist and work within separate silos; i.e. they play in parellel.  Think of all the silos: subjects, grades, departments, etc.

One silo problem that is particularly problematic is the curriculum vs. technology distinction.  I’ve long wanted to do an examination of school district organizational charts to see how technology is related to curriculum.  I know that in some districts, they are separate departments, each with its own director.  In some districts, there is an IT department (hardware, networking, etc.) that is separate from the instructional technology folks who may or may not live/exist under the direction of the curriculum folks.

I used to do evaluation research for education technology vendors who would often tell me stories about the “curriculum witch.”  They would usually pitch their solution(s) to the technology department and come very close to making a sale only to have the “curriculum witch” show up at the 11th hour and declare the program/software/etc. inconsistent with the curriculum goals of the district.  I’m certain there has been wasteful spending across numerous districts because the “curriculum witch” never did intervene.

In Virginia, our ISTE affiliate is VSTE.  They recently moved their annual conference to an early December date.  In fact, their conference is November 30-December 2.  From December 2-December 4 is the annual conference of VAASCD, the Virginia affiliate of ASCD.  That organization is focused mostly on issues of curriculum and professional development.  That these conferences are back-to-back in different parts of the state makes it nearly impossible for anyone (myself included) to be able to attend both.  So, the technology people will meet with the technology people and the teaching/curriculum people will meet amongst themselves.  I know people who I respect greatly that lead each of these organizations and I’m not at all blaming anyone for this situation.  I’m only pointing this out as a situation that reinforces the silo problem about which I am writing.

I could go on, but I need to bring this around to Leadership Day 2009.  For me, true school reform will not happen until leaders at all levels and across the many silos get together to think about governance arrangements. Especially at a time when collaboration and communication are easier than ever, we need to work together across levels of government and annihilate the silos in our education agencies.  Tha t is a huge leadership challenge.

“The real work of learning happens in the classroom, in the interaction between teacher and student.  This interaction is affected by innumerable large and small decisions made by principals, school boards, superintendents, state legislatures, education department officials, and the federal government.  These decisions and their implementation can either aid or hinder quality education in the classroom.” (Committee for Economic Development, 1994, p. 2)

Photo Attributions:

• Leadership Day: Scott McLeod
• Onions: Darwin Bell
• Silo: Chris Knight

About NECC.

For the fine folks at Dell.

Check it out there.

Thus, to get this stuff out of my system, I’m taking more of a rapid-fire, shotgun approach to this post.  In no particular order…

Finally, Let me make clear that I’m not anti-Gladwell. I think he’s a fine storyteller…in short form. He’s also an engaging speaker. When he ventures into book-length work, I think he really struggles.

That’s it; I’m glad I finally got that done. On to more important things…

I’ve had some quick back-and-forth with some folks on Twitter about Outliers, and the main theme of those arguing against me has been that I shouldn’t take Outliers as any serious work of social science.  I should understand Gladwell’s work as a collection of loosely-tied, interesting stories.  To that, I say “HOGWASH!”

The New York Times refers to Outliers as “The Sociology of Success.”  Gladwell himself subtitles the book “THE [emphasis added] Story of Success.”  In other words, Gladwell suggests quite explicitly that he is proposing a theory to explain success.   I simply do not know how to understand the book as anything other than a serious effort to develop an explanatory model.  That, to me, is a social scientific effort.

My contention, then, is that Gladwell’s work is the result of (weak) inductive thinking/reasoning.  In other words, Gladwell is attempting to ascribe “properties or relations to types based on an observation instance (i.e., on a number of observations or experiences)…” (Wikipedia).  Or, as is quoted in Wikipedia, Gladwell attempts to take us “”beyond the confines of our current evidence or knowledge to conclusions about the unknown.”  That process is depicted in the following figure.

SOURCE: Trochim, 2006

I won’t write a treatise on inductive reasoning, but I encourage you to read the Wikipedia article.  It’s quite well-written, IMHO.  If you’re into philosophy, I’d also encouarage you to read David Hume’s text on the problem of induction, which is referenced in the Wikipedia article.

My problem here is that Gladwell relies on incredibly weak induction.  Over and over, he uses a single case as a premise for his ultimate conclusion.  For example, he uses the case of Christopher Langan to make the point that genius alone does not lead to success.  Langan is a guy with an IQ of 165 who works on a horse farm in Missouri.  He has a higher IQ than Einstein, yet he works on a horse farm.  Gladwell’s logic, then, is that since Langan did not become highly successful, it must be that IQ is not enough.  In other words, here’s the logic:

• Langan has a really high IQ
• Langan was never in an environment that provided an opportunity to capitalize on his IQ
• Langan did not become successful
• Therefore, nobody with a high IQ can succeed without the right environmental circumstances.

As Lev Grossman of Time magazine wrote, “Gladwell’s weapon of choice when assaulting myths is the anecdote.”  I would add that Gladwell’s weapon of choice when reaching all of his conclusions is the anecdote.  In other words, if he can find one case that fits his thinking, he readily draws a conclusion by generalizing from that one case.  Over and over again, he states a conclusion and backs it up by telling a story.

There is a place for inductive thinking in the social sciences.  In fact, as Trochim points out, social science research often involves a cyclical process involving both deductive and inductive thinking.  But, some inductions are stronger than others.  For example, if you observe something very frequently, you might reach a conclusion that is reasonably probabilistic. That is a strong induction.

A real example:  Imagine if Martians had come to Earth for the first time and landed at Edubloggercon ‘09.  They would have observed nearly 100 edubloggers, all of whom were light skinned.  From that premise, they could have concluded that all edubloggers are light skinned.  THAT would have been a strong induction.

[TOMORROW: Part 2, where the fun stuff happens]

I’m disconcerted that we’re here at Edubloggercon ‘09 which is billed as an unconference where conversations happen.  The goal is to stand in contrast to NECC proper where there are formal sessions with (usually) presenters speaking to the audience.  Yet, even in the smaller breakout “conversations” here at EBC, there are more heads in the machines having conversations in the backchannel(s) than there are having face-to-face conversations.  I would take a picture, but I don’t want to “out” anyone.

I’m all in favor of backchannels…in the right place(s) at the right time(s).  If I wanted to chat online with the folks here, I could have stayed at home.  Additionally, if I were facilitating a discussion among a really large group of people, I might encourage a backchannel there.  But, I came here to have meaningful, face-to-face, real conversations about real issues in education.  Fortunately, I’m finding those in small pockets off to the side; I’ve had a few really meaningful conversations and it’s only 11:30.

But, I hope others join me this afternoon in using this particular opportunity, Edubloggercon, to get our heads out of the machines and to look each other in the eyes and to engage in “real” conversation.

PART V: SUMMARY AND RECOMMENDATIONS

[NOTE #2:  I know, I know...I'm a couple of days late on this one. Sorry.]

Before I sum up and conclude, I should point out one other major flaw in this study.  Marzano and his team use percentile ranks incorrectly.  On page 18 of the report, they write: “Of particular interest is the column entitled ‘% Gain.’ Again, this column contains the percentile gain (or loss) in achievement associated with the treatment (i.e., use of Promethean technology).”  Two problems here.  First, percentiles are not the same as percentages (or % as it is written in the report). Second, they then go on to write: “This value [the percentile gain] was determined by consulting a normal curve table for the area for each reported effect size.”  This would be fine if the scores on the dependent variables are normally distributed, which they most definitely are not.  For Marzano to go around saying that incorporating Promethean IWBs into instruction will improve student achievement by 17 percentiles is wrong on lots of levels.

It should be clear by now that if I were reviewing the Marzano IWB study report as a manuscript submitted for publication in a peer-reviewed journal, I would reject it.  I would not even mark it as “revise and resubmit.”  The problems with the work are too critical and, in most cases, impossible to fix.

In summary, those problems are:

• Misuse and misapplication of meta analysis.
• Incorrect usage of key terms.
• Serious problems with measurement validity and reliability.
• Major threats to internal validity.

Those last two points are with respect to each of the 85 classroom-based studies that serve as the basis for the meta-analysis.  The ultimate problem, then, is that the hallmark of good meta-analysis is the use of strong criteria as decision points for including individual studies.

As a point of comparison, I’m linking to two reviews of research.  Each is described as having used “best-evidence synthesis” which very closely resembles meta-analysis.  The methods used in the studies reported in the articles below are also consistent with those used by the What Works Clearinghouse.

Effective Programs in Elementary Mathematics: A Best-Evidence Synthesis

Effective Reading Programs for Middle and High Schools: A Best-Evidence Synthesis

In the first article, you’ll notice on the seventh page of the document (p. 432 of the article) a list of criteria for inclusion.  The authors of those articles also provide a list of studies that were considered for inclusion but that were ultimately excluded along with the reasons for exclusion.  This combined approach is critical; it gives the consumer of the research confidence that the data used in the meta-analysis come from many solid studies.

The impact of sample size for any given study included in a meta-analysis is another important point raised in the articles above.  According to the authors of the second article, “[p]revious research (e.g., Rothstein et al., 2005; Slavin, 2008; Sterne, Gavaghan, & Egger, 2000; Taylor & Tweedie, 1998) has shown that studies with small sample sizes report larger effect sizes than studies with large samples.”  As a result, in their meta analysis, the authors weight the individual findings by sample size.  In each of the separate sites/studies used by Marzano and his team in their meta analysis, sample sizes were tiny.  Consider for example site #34, teacher #57 where there were 9 students in the control group and 5 in the treatment group.  There is no way that study gets included in any decent meta analysis.

There is a bit of irony in my choice of articles to post as exemplars.  The lead author in each of those studies is Dr. Robert Slavin, the developer/founder of Success for All.  Slavin has been frequently critiqued for being the lead researcher/analyst/author on many evaluation studies of Success for All, the program that he created.  In other words, he has been accused of producing biased research.  I don’t know enough to say if his research is biased or not; it’s certainly legitimate though to raise the question of bias where he is involved in the research.  What I do know, though, is that each of the articles appears in one of the most well-respected, highly selective peer-reviewed journals.  The math study appears in the Review of Educational Research which is dedicated to only publishing exquisite and top-notch reviews, syntheses, and meta-analyses in education.  Thus, there is good reason to believe that those two articles present exemplars of how meta-analysis type research should be done.

I wrote earlier that doing good, comprehensive program evaluation in education is difficult and resource-intensive.  That said, I believe it would actually be reasonably easy to evaluate the impact of IWBs on student achievement.  In this era of standards and accountability, in any given state, we have year-to-year state test scores (at least in math and reading/language arts) from grades 3 to 8. So, Marzano’s team could have focused on one or two grade levels in one or two subject areas in one state.

Let’s say they focused on 8th grade student achievement.  All they needed to do was to find about 20 middle schools that were willing to participate.  In those 20 schools, there would be one subject-area teacher teaching in a classroom with the IWB and one teacher teaching a comparable class (NOTE: comparable here refers to students who are demographically similar and who are no different with respect to student achievement at baseline) without the IWB.  Surely there are at least 20 middle schools in any state where there are two 8th grade teachers teaching comparable classes.

A common way to get schools and teachers to participate in such a study would be to offer an incentive.  For Promethean, the promise of a free IWB to the teacher/classroom in the control condition the year after the study would be a wonderful incentive.  Given this sampling framework, Marzano’s research team could work with the schools, districts or state departments to get student achievement data on the students in those 40 classrooms (20 treatment + 20 control).  This could easily be done without violating any privacy laws.  The students’ scores on the 7th grade state exams could serve as the pretest or the covariate.  Their scores at the end of the 8th grade year would be the dependent variables.  Over 40 classrooms, we’d be talking about a sample size of well over 800, with well over 400 students in each condition.  Such a study would have lots of power.  Analytic decisions would have to be made with respect to the unit of analysis.  Marzano and his team could use the classroom as the level of analysis and conduct matched-pairs statistical test.  Or, they could use the student as the unit of analysis and account for the nesting or lack of independence by using multilevel modeling techniques.  Either way, this design would be much more appropriate and powerful for estimating the effects of IWBs on student achievement.

In the last couple of days, I spoke about this series of posts to two professors who I respect greatly.  Interestingly, each one was very surprised to hear my opinion that Marzano was affiliated with sloppy work.  One said, “he’s always been so careful.”  That may very well be.  I don’t intend for this series to be an indictment of Marzano (or even of IWBs).  My hope is that I’ve provided a sensible critique of research that is being widely disseminated.

I often lament that decisions in education are too often made in the absence of empirical evidence.  I wish policymakers in education would consult research more often.  However, if educational decision makers decide to make an investment in interactive white boards, I would strongly urge them to do so for reasons other than the evidence offered by the Marzano Research Labs.