Education Sector’s five principles for use of data re-imagine the relationship between data and education. Today, too much good data lives in silos — in state agencies, in districts, in schools, even at vendors. The data in a particular silo serves only the silo’s owner. Obviously, this prevents data from working together to create opportunities for analysis and synthesis. Critically, it prevents much good data from reaching the classroom to help students learn and teachers teach.

Data should be linked to students, enabling innovative new curriculum and allowing for the development of systems that can truly support teachers and administrators, along with parents, students, and community members. That data has to be more user-friendly than a spreadsheet – a truly Learner-Centered, Cross-Institutional system which provides secure but Open access to the Right data to drive instruction. But how do we move from the current closed system, with too little data living in inaccessible silos, to a robust hub of data that be accessed not only by teachers, students and parents, but also those outside the system who are interested in developing applications that can leverage this data? Can we take advantage of the moment when many states are re-thinking their assessment strategy? Can the current collection of technology systems truly handle the multifold increase in data that our systems could soon be producing?

• First, we must consider how assessments can be used to capture data at the instructional level – not just for informing instruction in the moment, but providing data to analyze instruction in more detail

• Second, we must build a technology infrastructure that can support the collection of far more data than we collect currently

The first step is to focus on how implementation of the assessment system impacts the collection of data. Too often, assessments are developed without  the needs of data analysts in mind. Assessments aren’t vertically aligned, course codes aren’t standardized, etc.  There’s little point to collecting cross-institutional data if we can’t see it across institutions.   Optimally, the data and assessment systems would create a feedback loop,  in which assessment captures more data and the data provides for the implementation of more innovative assessment.

Behind this potential vision of a data-assessment partnership is a comprehensive technology strategy that allows the central collection of that data. Few states have sufficient infrastructure to collect the amount of data that will be generated over the next 10 years. Many states do not yet collect transcript information, in particular grades per class, but soon states may feel compelled to go beyond this and collect grades by standard or formative assessment. This order of magnitude increase in data may mean that the current infrastructure needs to be rebuilt, not merely renovated. That’s before we start talking about eTextbooks and click-stream data.  Right now, it can take months to scrub data when it arrives from districts; that process will need to be reduced to hours.

Why do we need all this data? Will it really help our education system? Too often in education we are unable to answer the question of what really drives change. Much of this is due to the limited amount of data we have about actual instruction. Simple experiments such as teacher logs have shown that when we do have more detailed data we can do more powerful analysis. Like other fields, education is making the transition from art to science. Data makes that transition possible.

On the particulars, Thorne seems to share many of my views about the need to improve preparation, reduce remediation, facilitate credit transfer, help people learn and graduate and get good jobs and so on. But underlying her critique is a strong thread of conviction that too many people are going to college. And I just don’t believe this to be true. For decades now we’ve been investing huge amounts of time and money in growing the supply of college graduates. And for decades the job market has responded by increasing the wage premium for college graduation. If there was an oversupply of college graduates, we might expect employers to have taken the recent recession as an opportunity to shed these more expensive employees. Instead, the opposite has occurred–people with college degrees have weathered the storm the best while people without postsecondary credentials have taken the biggest hit.

For NAS, the too-many-graduates thesis goes hand in hand with the idea that academic standards in higher education are too lax (which is what, in their view, allows too many people through.) And indeed many colleges could do more to set higher standards and provide a better learning environment in which students can meet them. That’s why I suggested that colleges all submit an annual, “public learning audit” of the kind that Earlham College President Douglas Bennett recently proposed. But–tellingly–Thorne rejects this idea by deliberately mis-reading what such an audit would entail. Says Thorne:

While it is desirable for the public to know what colleges are actually teaching, we’ve found that the bureaucratic focus on student learning outcomes has unintended consequences in the direction of actually lowering academic standards.  Outcomes assessment encourages teachers to set the lowest possible goals for their classes so that, come assessment, they can prove that they have met those goals.  It also forces teachers to concentrate on aspects of their subjects that are easily measurable, rather than intrinsically important.

A bureaucratic focus on low standards would definitely be a terrible idea. But that’s not at all what Bennett is proposing. Unless Thorne believes that there is no possible way for a university to understand and report how much its students are learning–an idea that I think is absurd on its face–then her rejection of the learning audit makes me suspect that NAS really has no interest at all in a set of policy solutions that could help more students earn a high-quality college degree. It just wants to hoard college credentials for the privileged few. At worst, these notions slide into Charles Murray-style genetic determinism. At best, they ignore the evidence that increasing academic expectations would result in more people earning college degrees, not fewer.

Education Sector’s five design principles powerfully reframe the conversation about how, when and where to use data to support student learning so that it will not die in vast data warehouses, but will live to guide the daily actions of teachers, families, afterschool and summer programs, tutors, and ultimately of students themselves as they assume responsibility for their own learning.  While the principles are not now widely implemented, they are not just pie in the sky either—real communities, districts and schools are working on them and there is much to be learned from their efforts.

We are documenting early lessons from them in a set of papers about cradle to career data pathways soon to be available at the HFRP site. Our documentation shows that data systems incorporating the five design principles and accompanied by strong tools and outreach for families, afterschool programs and others can increase the chances of effective family engagement as families understand that the pathway to college begins in early childhood and that they, along with teachers and others, can help students stay on track through K-12 with the aid of data in the accessible form of GPS performance navigation system-type tools. At the cradle end we are finding early childhood programs providing regular developmental feedback and related ways families can support learning thereby creating the demand for actionable performance data when children start school.

Some of the most innovative K-12 work is going on in New York City with the Department of Education’s ARIS Parent Link and New Visions’ 9^th Grade Parent Involvement in College Readiness initiatives. Both efforts are piloting and learning from their substantial outreach efforts to engage low income schools and families in getting and using data to support learning in and out of school.  Each provides attendance and selected performance data and ARIS conveys grade level and subject area expectations as well. Their experience underscores the need for creative and persistent parent coordinators and other staff dedicated to this work at the school and district levels.

New Visions has created two powerful tools, the School Data Snapshot for teachers and individual Student Tracker for families, students and others. Both track performance on key benchmarks including attendance courses, credit accumulation, and grades necessary to graduate and to be college ready. The Tracker is a planning and performance progress tool informing parent, teacher and student conversations and triggering efforts to stay on track, including credit recovery.  New Visions co-designs their outreach efforts with families, creates concrete ways for them to support student achievement, and is beginning to share data with afterschool programs as well.  A case study of their work and how it supports whole school college readiness efforts will be available on the HFRP web site.

In a pilot effort to stimulate low income family and student use of ARIS Parent Link in 24 schools in Lower Manhattan, the NYCDOE schools are trying different outreach strategies  Early lessons learned include the importance of a welcoming space for parents with access to computers and training in how to use them,  the value of utilizing parent and student volunteers (in one school students train their parents and bring back notes confirming they logged their parents in), the importance of training teachers to ensure they are comfortable using ARIS and in having data-driven discussions with parents, and of principal leadership and of persistence and creativity (at another school the librarian opened early for ARIS workshops and at another a parent coordinator emailed parents who had not logged in).

Who cares who gets RTT awards? Not Jay Mathews—but he’s willing to be proved wrong. (Class Struggle)

Is it longevity or student outcomes? Cuban asks: What defines great superintendents? (Larry Cuban)

Want to see the human side of “college- and career-ready?” (The Educated Reporter)

Cap enrollments and increase tuition?  Or cap tuition and increase enrollments? (Inside Higher Ed)

Education Sector’s Five Design Principles raises important questions about the technology tools we provide to schools. A lot of us think about what data can do for school improvement, but we don’t pay enough attention to the unprecedented role that new technologies play in analyzing and distributing that data. These tools have the potential to shape what we see, and thus, what we attend to as educators and policymakers.

In these design principles we continue to search for the right metaphors, and I like that. We stretch to comprehend how it is that our phones, navigation systems, and other technologies have become so interwoven into everyday life. Besides this, the edges around our technologies have blurred such that our phones aren’t even really phones anymore. So, when it comes to tomorrow’s educational data systems, we have high hopes for the malleability and richness of new technologies, knowing that tomorrow’s tools may be unrecognizable to us.

Even so, we can start with good products, but the real work will be dealing with people and what they understand about the data systems in their lives and jobs. No tool, no matter how well designed, will ever “fit” perfectly. On one hand, tools are made by people at a particular time and place, guessing at the needs of others. On the other, it is up to users to embrace or resist them; they will adapt, work around, or expand their uses in unexpected ways.

For example, my wife, in her vehement loyalty to MS Office 2003, goes through a variety of emotions (and even contortions) whenever trying to work with anything else. This isn’t about technical skill. It’s about her perceptions about where things “ought” to be, how work ought to be, and whether or not the things she does brings about the things she wants.  Indeed, we are all aware of the role of perceptions in how we select and hold onto cars. I held onto my old VW for as long as I could—which is to say that at some point little flaws started to grow. They gnawed at my sense of satisfaction and accomplishment.

So, part of helping educators use data to inform decisions will also involve helping them think differently about data systems. The bicycle we know today is the result of many changes in opinion. It didn’t emerge linearly, with inventors contributing improvements ever onward.[i] Instead, people at first thought that air tires were stupid. In addition, the phrase “taking a header” comes from how one beloved design that was prone to flip its riders’ heads straight into the ground.

In the end, technology is what you make of it. Most of us need little convincing that riding metal rims on cobblestone is a bad idea. My hunch is that effective data use in schools will depend as much upon technological advancement, as it will depend upon the sense we make of technologies fit into our lives.

We began a concentrated focus on using data about six years ago. We made the common mistake of purchasing an administrative system that really did nothing for teachers–and more importantly–nothing for students to understand where they are and where they are going. With a new superintendent we switched our focus and today we’ve seen the results with not just higher test scores, but more importantly, with overall growth for all of our students. That means our students who used to score proficient are now moving into advanced categories. Students who started behind are catching up, since we’ve targeted those students for more growth.

One of my teachers at the Fox Chapel Area High School, Jill Tabis, has shared her insights as part of a case study assembled by the Consortium for School Networking (CoSN). Jill uses a data portal (EdInsight) provided by OnHand Schools, a Pittsburgh-based company. Like many of its competitors, the system gives teaches and administrators quick and easy access to key information about student achievement, attendance, discipline, and assessment data over time. You need as Bill points out, “the right data.” Here’s how Jill sees the situation:

In my classes, if I see that there is a struggling student or if I notice a struggling student, I’ll invite them to my room during QRT (homeroom). There, I’ll pull up their grades from 3-4 years back and talk to them about their strengths and weaknesses over the past few years, and see how it could POSSIBLY relate to their current achievement/struggle. We’ll pull up attendance and talk about tardies and how important it is to attend school (IF attendance is an issue). I ask them the same questions that I ask my QRT students: what are they proud of, what can they improve on, and how I can help them. Sometimes it’s one student and we’ll work on study skills; test taking strategies; or sometimes just working on the student’s self confidence issues.

As Bill Tucker points out you can have a great system, but you need teachers to use it effectively. Jill is a great example of someone who has taken the process of data analysis to the next level – getting students involved. Ultimately as Bill suggests this needs to include parents, after-school providers, and anyone who will work with a given student to provide support and guidance.

In Fox Chapel, we know we moved to a “learner-centered” solution, but we’re still trying to address some of the other issues raised by Bill. In Pennsylvania, like many other states, we have a state-wide information system, but right now data flows in one direction and that direction is not back to the school, teacher, or student. We also do not have Internet access to data other than our own data. We need the state to move in this direction. The “Race to the Top” may provide the next step, but in the meantime we’re looking at other ways to make this happen, e.g. purchasing solutions that are shared by neighboring school districts with vendors who have designed simple ways to share data.

What happens when teachers turn Young Money’s Bedrock into a test-prep anthem? (h/t Joanne Jacobs)

Curiouser and curiouser—is “Alice” just a critique of the (then) new math? (New York Times)

How will the Obama Administration’s view on civil rights influence pressing education policy decisions? (Flypaper)

Diane Ravitch: The Ayn Rand of education? (Sherman Dorn)

Since the 16 finalists for Race to the Top funding were announced last Thursday, there has been a general gnashing of teeth and rending of garments.

“A sad day for reformville,” Petrelli writes. Today’s WaPo says the list isn’t “exclusive” enough.

I think that Secretary Duncan did just what you’d expect a college ballplayer to do: he kept his eye on the ball. And the ball, in this case is encouraging politicians at the state and local level to make changes.

Politicians, like entrepreneurs, aren’t risk takers. For them to get involved with a reform effort—especially one that takes on entrenched interests—there has to be a pretty clear sign that there could be a prize in the bottom of the Cracker Jack box.

The Department’s math—about one-third of the applicants made it into the Sweet Sixteen—seems about right to someone who spent nearly two decades as a policymaker on the state and local level.

To use another basketball metaphor (hey, it’s March), the Department is using the clock. All 16 won’t get funded, of course. But now, while state legislatures are still meeting, is the time to push legislators to act.

Already folks like ConnCan, one of the best state education advocacy organizations, are using the results to pressure for reforms in their state. Because winning still seems like a realistic option, my guess is they’ll make good progress in moving their legislators toward significant legislation.

And that, after all, is what RTT is really about. So I say the box score on this one should read: Duncan FTW.

So I think it’s only a matter of time before RTT gets franchised out to other policy areas. The first should be higher education, where the administration is on the verge of investing billions of new dollars but still lacks a sharp state-level policy agenda to ensure that those resources make a difference. For a complete rundown of what the Race to the Top for Higher Education (henceforth: RTT4HE) should look like, see my new column in the Chronicle of Higher Education.

In the past decade, school districts and states have spent more than a billion dollars to build and implement data systems. Data about student learning—and the systems that collect, organize, and report on this data—are what U.S. Secretary of Education Arne Duncan calls “the driving force [behind education] reform.”

In the next five years, our nation will likely spend a billion dollars more. Influential persons and institutions, from legislators to governors to the Gates Foundation, extol the virtues of better data. Data systems are a key criterion for Race to the Top fund applications. And the American Recovery and Reinvestment Act—the stimulus package—injects an additional $250 million federal investment into longitudinal databases and requires states to assure that they are building these data systems as a condition for receiving stimulus funds.

The rhetoric around educational data is compelling: With better data, policymakers can identify effective schools and educators, expose problems, make better decisions about the allocation of resources, and build political will for reform. At the classroom level, better data will inform instruction—enabling teachers to better understand what approaches work for specific students—and lead to better teaching and improved learning.

Our first billion-dollar investment has yielded important results—building the capacity and systems for better collection and management of data. According to the Data Quality Campaign, by 2011, all 50 states will have longitudinal data systems that track student performance from year to year. Many districts have followed by building their own extensive data warehouses.

But, despite states’ and districts’ tremendous progress in building data systems, policymakers are not yet routinely using these new data to improve accountability systems, support performance management processes, evaluate programs, or influence resource allocation decisions. More importantly, the data is not yet being used where it matters most—in the classroom.[i] A 2009 Department of Education report found that “even in districts with a reputation for leadership in using data, electronic data systems are barely influencing classroom-level decision-making.”[ii] Many systems aren’t even designed to provide teachers—or students and their families—with access. And much of the data—in particular, scores on annual state tests—are not useful or appropriate for informing day-to-day classroom instruction. Data may be everywhere, but the systems to access and use the data are disconnected and many times impossible to use.

We’re now entering the next phase in our nation’s thrust to use data to improve educational outcomes. The challenge is no longer whether or how to build institutional data systems, but to use better information about teaching and learning to improve outcomes for every student. A focus on the actual use of data must drive our next billion-dollar investment. And this focus has clear implications for how we think about, design, and implement data initiatives. Going forward, five principles should inform these initiatives:

1. From Institutional- to Learner-Centered: It sounds simple and obvious, but designing learner-centered systems would mean dramatic shifts in current practice—away from compliance-focused data needs, such as reporting for federal programs, and toward information that students, families, and teachers can use to support day-to-day decisions about learning and achievement. Like a GPS navigation system, learner-centered systems would track a student’s progress and provide individualized guidance on the paths and actions to take toward his or her learning goals. They would combine both hard data and qualitative information to give a full picture of learning and performance throughout a student’s entire educational experience. The information would be timely and accurate. And, it would be specific enough to inform the actions of teachers, students, their families, and the variety of persons who support learning, including instructional coaches, tutors, after-school and youth workers.
2. Information Flows Across Institutions: Students are increasingly mobile and not just across schools, districts, and states, but across a number of different learning opportunities within both the traditional school day and out-of-school programs. Yet, a teacher cannot use information for a transfer student when it’s trapped in another district’s data system; if you can’t mobilize information, you cannot use it. A learner-centered system would operate across institutions, integrating important information from a wide variety of schools, programs, and interventions into a more complete and accurate depiction of a student’s progress. It would allow adults who work with students to better communicate and to understand more about those students. And, it would enable schools, districts, states, and the federal government to improve the accuracy, timeliness, and efficacy of data-gathering. Internet-based platforms that enable this seamless exchange of information, rather than costly projects to integrate systems on a one-by-one basis, are critical to learner-centered approaches.
3. Usefulness and Usability Drive Adoption: Good teachers strive to know their students better. To be successful, data initiatives must inform the work flow, incentives, and actual day-to-day practices of educators, but avoid “one-size-fits-all” solutions. And, unless systems are designed to be valuable to these educators in their daily work with students and provide insights into students’ goals, use of data will be limited. The best tools won’t require extensive training or campaigns to convince educators of their value—they’ll be so intuitive and easy to use that adoption comes naturally. Importantly, educator use will ensure that data is more accurate, allowing for better information and decision-making at all levels.
4. Common, Yet Open Systems: Our nation’s massive and ongoing investments in similarly focused health care initiatives, such as electronic medical records, which strive to use information technology and better data to radically reform and improve health care, provide an important lesson: Open, yet standards-based systems are critical. Here, good governance is essential to ensure that common policies, technical standards, privacy protections, and usage protocols across institutions are enacted and attended to on an ongoing basis. This facilitates use and allows educators to easily exchange information. But, monolithic systems are neither effective nor easily adopted. Local districts, schools, and a wide variety of other programs, from after-school to internships, must be able to tailor and customize systems for their particular information needs. The iPhone (and other “smart” phones) provide good examples: They adhere to recognized (or what might be considered standards-based) protocols so that they can operate across common voice, data, and GPS networks. The software, however, also enables the easy development and integration of an unending variety of useful applications, or “apps,” that users can install to customize and extend the use of their phones. State longitudinal data systems, which currently exist separately from district systems, could be designed in a similar way—so that they provide the core of data for every district, but enable districts to easily tailor and extend beyond the core for their unique needs.
5. Get the Right Data: The right data allows us to not only assess performance—for students, educators, and administrative systems—but more importantly, to understand the processes and changes that could lead to improvement. While improvement is still not optional, the message is no longer just “You’re failing.” Instead, it’s “Here’s where you are. Here’s where you need to go. And here’s data, information, and suggested actions to help you get there.” Just as using data to inform student learning can empower students, providing high-quality data to educators, schools, and districts about their practice will also be enabling. The most successful performance management initiatives—those that actually change practice—will engage educators and put student success and responsibility at the center.

The potential for better data to improve education is real. Data and data systems are used to help inform and improve decision-making in almost every field of human endeavor—from health to sports to crime prevention to finance to even the production of motion pictures. Schools and districts, in cities such as Charlotte, New York, Dallas, and Houston, are beginning to see results. But it is astoundingly difficult to impact day-to-day classroom practices. And unless we design data systems with a primary goal of improving classroom teaching and learning, our investments will show little return.

[ii] U.S. Department of Education, “Implementing Data-Informed Decision Making in Schools—Teacher Access, Supports, and Use,” 2009, (http://www.gesci.org/assets/files/Knowledge%20Centre/Implementing%20Data%20Informed%20Decision%20Making%20in%20Schools-Teacher%20Access,%20Supports%20and%20Use.doc)

]]> http://www.quickanded.com/2010/03/five-design-principles-for-smarter-data-systems.html/feed 2 http://www.quickanded.com/2010/03/five-design-principles-for-smarter-data-systems.html