Of That

inBloom - For My Concerned Friends

2013-05-02T22:57:00.001-07:00

In my post about the Common Core State Standards I wrote about how concerned pundits have lumped together five related but independent efforts. Today I'm writing about inBloom which I'll contrast Statewide Longitudinal Data Systems – two more of those five.

InBloom is a service designed to help students achieve academic success through personalized learning. Those of us who helped develop the Shared Learning Collaborative (which was renamed inBloom in February) are convinced that personalizing the learning experience is the best way to improve student achievement. Whether personalization is being done by a teacher, an online learning system, or a synergistic combination of the two, it happens when information about what the student needs to learn intersects with information about available learning materials.

With that in mind, we set out to supply teachers and students with the data they need. That's what inBloom does. It taps into existing student data systems at schools, districts and states and makes that data available, in a secure way, to authorized teachers, students and parents. Simultaneously it indexes a library of teaching materials and makes them available to those same individuals.

A lot of work went into preserving student privacy. inBloom requires two things to happen before any student data can be retrieved. First, the application they are using must be authorized by the school district. Second, the individual using the application must be logged into inBloom and be authorized to access the requested data. This protection of student privacy is compliant with and goes beyond the requirements of FERPA and state data privacy laws.

So, who can access student data? Teachers can access data about students who are enrolled in their classes. Parents, if authorized by the school or district, can access their children's data. And students can access their own data. An application, such as a personalized learning system, can only access private student data if an authorized user is logged in to the app.

To match student achievement data against available learning resources, we need a common taxonomy of what it is that students need to learn. It's not sufficient to know that Johnny got an "A" on assignment number 5 but a "C" on assignment number 7. We need to know what learning objectives were represented by each of these assignments. That's why inBloom makes use of the Common Core State Standards. In the data, we can show that assignment 7 was on multi-digit multiplication. And, since it appears that Johnny needs some more practice, we can search the library for multiplication practice that's suitable to his age and preferences.

In a nutshell, inBoom supplies the student and content data needed for effective personalized learning.

Statewide Longitudinal Data Systems

For whatever reason, some people have confused inBoom with Statewide Longitudinal Data Systems (SLDS). The SLDS effort was launched more than a decade ago by the Bush Administration and funded by the Educational Technical Assistance Act of 2002. While a separate statute, it's related to the No Child Left Behind Act of 2001. The official SLDS website describes it this way:

Better decisions require better information. This principle lies at the heart of the Statewide Longitudinal Data Systems (SLDS) Grant Program. Through grants and a growing range of services and resources, the program has helped propel the successful design, development, implementation, and expansion of K12 and P-20W (early learning through the workforce) longitudinal data systems. These systems are intended to enhance the ability of States to efficiently and accurately manage, analyze, and use education data, including individual student records. The SLDSs should help states, districts, schools, educators, and other stakeholders to make data-informed decisions to improve student learning and outcomes; as well as to facilitate research to increase student achievement and close achievement gaps.

Under grants from the SLDS program, 47 states are developing longitudinal data systems that aspire to collect student data from preschool through college and even into workforce placement. Analysis of the data should help researchers understand the impact of different factors and programs on student achievement.

Before being analyzed to find trends, the data is either anonymized or aggregated in order to preserve the privacy of the students. However, the databases themselves necessarily contain personally identifiable information (PII). That's because the data comes from multiple sources: K-12 schools, colleges and workforce databases. In order to connect all of the data about an individual together, you need to be able to match up records and that requires the personal identity information about each individual.

This concentration of individual data spanning decades of educational experiences spooks a lot of people. Two factors help moderate those fears. First, according to federal regulation, data is not combined between states nor is it reported to the federal government. Only aggregate data (sums, averages and so forth) is reported to the federal government. Second, the Family Educational Rights and Privacy Act (FERPA) prohibits the release of any student information without permission from parent. Of course, that doesn't reassure everyone. The mere fact that such databases exist concerns many.

I have a different concern. I've previously written about Theories of Change for educational improvement. In this case, the theory is that over time the collected data will help government officials, education officials, teachers and curriculum developers make better decisions based on what really works. But if we're trying to figure out how a particular curriculum choice in elementary school affects a student's college prospects, it may take 10 years or more to have the data to measure that effect. My concern is that this effort will take a long time to make a difference.

~ ~ ~ ~ ~

inBloom and SLDS both collect student data. Both leverage CEDS definitions for the data fields they collect. But the purposes of the data sets and the people who have access to the data are entirely different. Of the two, I'm more optimistic that inBloom will achieve the impact on student learning that our country needs.

The Common Core State Standards - For My Concerned Friends

2013-04-23T23:58:00.002-07:00

Even before their adoption in the Summer of 2010, the Common Core State Standards (CCSS) had their advocates and their critics. Recently, however, that criticism has made its way into the popular press. Knowing that I've worked on related projects at the Bill & Melinda Gates Foundation, friends and family have asked my opinion.

Several of the pundits have conflated five different projects as if they were all the Common Core. These are to some degree related but each has it's own sponsors and they are being managed and adopted separately. They are:

The Common Core State Standards themselves.
State or district curriculum.
State Longitudinal Data Systems (SLDS).
inBloom
Race to the Top Assessment Consortia

In this post I'll address the Common Core and what distinguishes it from a curriculum. In a future post I'll write about inBloom and other data systems. And one more post will cover the assessment consortia.

The concept of state core standards gained prominence during the Bush Administration as part of the No Child Left Behind act. In a recent blog post I wrote about how they are part of the Standards and Accountability theory of education reform and how later and more promising theories also rely on quality standards.

The result of NCLB and related efforts is that each of the 50 states developed its own core standards. This has the vague advantage of more local influence but it has two significant disadvantages. First, there are differences between what students learn in different states. So colleges and universities don't have a consistent standard of preparation to expect from students. Second, developers of tests and curriculum spread their resources 50 different ways. The result is lower quality teaching materials and examinations.

Starting in 2008 a consortium of state representatives developed the Common Core State Standards for ELA/Literacy and Mathematics. They don't include Science, Social Studies, History or any other subject. However they do specify literacy standards for Science and Social Studies. In other words, they specify that reading should be a significant part of those subjects without specifying the actual titles or subjects to be read.

The standards are written in the form of "competencies" – that is, descriptions of things that students should be able to do. For example, standard CCSS.ELA-Literacy.RL.8.5 reads, "Compare and contrast the structure of two or more texts and analyze how the differing structure of each text contributes to its meaning and style." Standards like the common core describe what is to be taught while curriculum describes how it should be taught.

Here are some ways that distinction applies: The Common Core describes the difficulty of text to be read at each grade; curriculum gives a list of actual books and stories. The common core describes the kinds of problems a student should be able to solve; curriculum specifies the order concepts will be taught and includes exercises to be performed. The rivalry between Phonics and Whole Language is not resolved by the Common Core; that decision remains in the hands of district curriculum committees.

Critics of the core have missed an opportunity here. Since curriculum involves textbooks, lesson plans and teaching materials, it consists of thousands of pages, tens of hours of video and other media. It's also copyrighted. All of this makes reviewing a curriculum a daunting task – albeit an important one. Meanwhile, the standards are relatively short and accessible. They are released under an open license and you can read them online at http://corestandards.org. They total somewhere around 200 pages long including appendices so you can review them in an afternoon.

They are different from previous standards. The ELA/Literacy standards start with a 50/50 balance between literary and informational texts (fiction and non-fiction) in the lower grades and increase that to a 30/70 split when social studies and science reading are included in upper grades. Reading in the English classes remains a 50/50 split through all grades.

The focus in all texts, whether fiction or non-fiction, is on critical thinking and extracting arguments and meaning from the text itself. As a result, "response papers" where a student expresses their opinion or thoughts about a document are discouraged in favor of more analytical writing that identifies arguments, contrasts perspectives and uses evidence from the documents themselves.

Many English teachers have objected to the shift away from an emphasis on fictional reading and writing. Picking up on that, one pundit suggested that Huckleberry Finn would be eliminated in favor of the phone book. Of course, the phone book isn't what the Common Core means by "informational texts". A sample list can be found in Appendix B of the common core. Remember that actual reading lists are the domain of curriculum. That's why this is in an appendix; it's not normative to the standard. Examples of informational texts in that list include the founding documents of our country, Lincoln's "Gettysburg Address" and Ronald Reagan's “Address to Students at Moscow State University”. This isn't the phone book.

To get an idea of how these texts might be taught, I recommend this video from David Coleman. He was a coordinator and key author of the ELA standards. In this video he demonstrates how to teach the standards using Martin Luther King's "Letter from a Birmingham Jail" and Lincoln's "Gettysburg Address." In both cases he shows the brilliance of the authors and how it's not necessary to teach a lot of background because the authors include the needed information in the texts themselves.

Regarding the math standards, there are two important shifts from existing teaching practice. First is that they have reduced the total amount of information to be taught. The overall theme is narrower and deeper. For example they require fewer methods for solving quadratic equations (narrower), but they also introduce complex numbers and the possibility of an imaginary result to a quadratic (deeper).

The second change is that they teach mathematics at three levels: conceptual understanding, computational and procedural fluency. The overall goal is to help children become "numerate." That is, students should naturally apply mathematics to interpret things in their daily lives.

So, what are the objections? A common one is that this is a federal program to control what our students learn. They only control teaching if the standards are considered to be limits to what is taught. But they are really a floor, not a ceiling and most of the details remain left to the curriculum.

Other objections come from academics arguing for or against certain pedagogical theories that the rest of us aren't familiar with. But the common core aren't as opaque as all of that. As I wrote a couple of months ago, the English standards focus on a few basic skills applied to increasingly complex texts. The math standards cover the familiar topics of arithmetic, algebra, geometry and so forth.

The biggest issue is that change is difficult and frequently unpopular. The changes demanded by the common core aren't easy ones. They require changes to curriculum; they require new lesson plans; and they require teachers to approach subjects in new ways. Many people are excited by the possibilities but it's not surprising that some would prefer to preserve the status quo. Unfortunately, status quo isn't good enough.

The Common Core State Standards offer two important advantages over previous state core standards. First is simply that they are common. We hope that by concentrating their efforts on one standard instead of 45, developers of curriculum and examinations can do a better job than before. The second advantage is that the Common Core is a second-generation standard built on a foundation of the best state standards and informed by the experience of those who built the first generation.

Are they perfect? Not likely. But these new standards are better than previous ones and they will become a valuable tool in our personalized learning arsenal.

A Taxonomy of Education Standards

2013-03-29T16:54:00.000-07:00

I've previously posted and updated A Four-Layer Framework for Data Standards. When working with education standards I've also used the following taxonomy that categorizes standards according to their purpose. For convenience, this taxonomy is also available in PDF form under a CC0 dedication.

Types of StandardsThere are three types of standards that are involved educational efforts: Academic Standards, Data Standards and Technology Standards.

Academic Standards include achievement standards like the Common Core State Standards (CCSS) plus curriculum and testing standards. Contemporary practice in the U.S. is to describe academic standards in the form of learning objectives – descriptions of skills that students can acquire or demonstrate. Historically it was more common to describe standards in syllabus form – as a list of subjects to be studied.

Encouraged by the No Child Left Behind Act, the 50 states have each defined core curriculum standards. More recently, the CCSS standards for Mathematics and ELA-Literacy have been adopted by 45 states. Using a similar process, the Next Generation Science Standards have been proposed for multi-state adoption. In higher education there is no such consistency. Some institutions have developed their own sets of standards but most leave the objectives up to the professor. A few industry organizations publish standard sets. These include the AAAS Benchmarks for Science Literacy [3] and the National Center for History in the Schools standards for History.

Data Standards define the data elements and structures used to store and exchange educational information. In the Four-Layer Framework data standards may include layers 1-3 (Data Dictionary, Data Model and Serialization).

For education, the three major domains of data standards are Student Data, Educator Data and Content Data. Important metrics like graduation rate, student financial aid repayment or college-going rate are derived from data sets but aren’t data in and of themselves.

Student Data includes traditional demographic information as well as a student record which includes academic achievements, assessment results, learning activities, attendance and so forth. Educator Data includes information about teachers and staff. It includes qualifying information like academic credentials, a portfolio of creative works and publications and data about teaching performance. Content Data, often called metadata, is information about learning materials including textbooks, assessments, multimedia and digital resources. Content data often indicates the alignment between learning resources and academic standards like the CCSS.

Technical Standards define how systems interoperate. Accordingly, they usually include the protocol layer of the Four-Layer Framework. A wide variety of standards may fit into this category but the majority of education-related technical standards involve Content Packaging Formats, Interoperability Protocols and Data Exchange Protocols.

Content Packaging Formats support the transport of learning content (e.g. text, video, graphics, etc.) and assessments between systems. Examples include IMS Common Cartridge and SCORM.

Interoperability Protocols support interoperability among learning systems. The most common use case is integration of learning tools (like simulations, games or assessments) into learning environments (like a learning management system). Key functions are to identify the user to the learning tool, ensure that they are authorized to access the content, transfer control to the tool, and collect data back. Common examples include OpenID, SAML, OAuth and IMS QTI. Data Exchange Protocols represent layer 4 in the Four Layer Framework for Data Standards. Thus, data exchange protocols are usually paired with a corresponding data standard. Frameworks for setting up data exchange protocols include ESB, SOAP and REST.

Progress Report: The Personalized Learning Model

2013-03-20T18:37:00.001-07:00

A bit more than two years ago my colleagues and I at the Gates Foundation came up with the Personalized Learning Model. Eighteen months ago I introduced it on this blog. Two weeks ago, at SXSWedu, we celebrated the launch of inBloom which is a set of services that support the Personalized Learning Model.

The concept of personalized learning was not new or unique to us. Indeed, we chose it because the benefits have been well-proven. Our model was a way to describe how technological supports could be designed to facilitate personalized learning. As we've been working on this for a couple of years now, it's time for a progress report.

Learning Objectives
In 2010 a consortium of states, coordinated by the Council of Chief State School Officers (CCSSO) and the National Governor's Association (NGA), introduced the Common Core State Standards for English/Literacy and Mathematics. They were rapidly adopted by 45 U.S. states. Having common standards across states is, of course, convenient but these standards seek to be an improvement on the previous generation.

The Common Core State Standards were written by building on the best and highest state standards in existence in the U.S., examining the expectations of other high performing countries around the world, and careful study of the research and literature available on what students need to know and be able to do to be successful in college and careers. No state in the country was asked to lower their expectations for their students in adopting the Common Core. The standards are evidence-based, aligned with college and work expectations, include rigorous content and skills, and are informed by other top performing countries. They were developed in consultation with teachers and parents from across the country so they are also realistic and practical for the classroom. (From the CCSS FAQ.)

In August of 2012, the CCSSO and NGA released official identifiers and an XML representation of the Common Core thereby facilitating alignment of digital learning resource to the core standards. Driven by the need to measure and prove coverage of the standards, finer-grained identifiers are being assigned to individual learning objectives within the common core standards.

The Next Generation Science Standards are also under development with an expected release before the end of March. Following their release, state education boards will consider adoption.

Postsecondary education is taking a different approach. There's little formal agreement between colleges and universities on the learning objectives that compose common courses. However, college and university departments are defining the objectives for core curriculum and there is growth in the sharing of these objectives within university systems. Colleges are also considering use of the Common Core for developmental education courses.

Student Data
Common Education Data Standards (CEDS) is a project to create a common data dictionary and logical data model for education data. Applications that align to CEDS use the same definitions for data fields making data exchange easier and increasing fidelity.

The inBloom Data Store uses CEDS for its data model and ingests data in SIF and Ed-Fi data formats. It offers an API through which personalized learning applications can store and retrieve common student data. Security features preserve the privacy of data and ensure that only authorized people can access it.

Newer data stores align student activity and assessment data to standard learning objectives. The goal is derive a model of what the student knows, what the student is learning and what the student has yet to learn. This enables rich reporting on student competency levels on an objective-by-objective basis and the stimulation of targeted interventions.

Content
I prefer to talk about educational content as learning activities. There are the traditional passive media such as reading, lectures, video and so forth. More engaging are interactive activities like virtual labs, simulations virtual worlds and games. For both active and passive content, education doesn't need special formats. The web content formats managed by the W3C are adequate and well-supported. What is needed is a way to represent the alignment between the content or activities and the standard learning objectives.

The Learning Resource Metadata Initiative (LRMI) is a standard way to describe educational materials including their alignment to standards. It's based on the Schema.org metadata standard adopted by Google, Yahoo!, Bing and Yandex.

LRMI metadata can be shared between systems using the Learning Registry. The inBloom index consumes LRMI data from the learning registry and offers a search service that can find educational content suited to specific student needs.

IMS Global defines standards for packaging learning content for import into learning management systems. However, I prefer the approach IMS uses for Learning Tools Interoperability. Instead of packaging content, this protocol allows content from other sites on the web to be seemlessly integrated into the learning experience. Integration in this way avoids limitations imposed by the packaging format and lets the developers of learning activities collect data about the use and effectiveness of their products.

Assessments
In my opinion, assessments are presently the weakest part of the Personalized Learning Model but that's changing rapidly. Two multistate assessment consortia, Smarter Balanced and PARCC are developing new assessments aligned to the Common Core State Standards. Both are committed to supplying formative and interim assessments in addition to year-end summative exams. CoreSpring is pooling assessments from a more than six different sources to supply a bank of good quality assessments that can be used in class, for quizzes and in interactive learning environments. MOOC developers such as Coursera, edX and Udacity are having to invent new ways to offer interactive assessments at extremely large scale.

In the long run, I expect the line between learning activities and assessment activities to blur. After all, much of learning occurs when the student demonstrates understanding. With adequately instrumented activities, the accumulated data about student competencies should reduce the need for big summative exams at the end of the year.

~ ~ ~

We've come a long way in the last couple of years. Pioneers in this space like DreamBox, Knewton, Read180 and GrockIt had to build a whole infrastructure. But now there's a solid set of building blocks on which developers can build personalized learning applications. I anticipate a lot more innovation at the place where student data and content come together.

Theories of Education Reform

2013-03-06T20:16:00.000-08:00

My oldest son was a junior in high school when the standardized tests associated with No Child Left Behind were rolled out. One day, shortly before the exams, he asked me, "Why do we have to take these tests anyway?"

I answered truthfully, "They're not evaluating you, they're evaluating your school."

I found out later that with that information, he and his friends challenged each other to get the lowest scores possible. I sometimes use this story to illustrate broken feedback loops. It was nine months later before the scores had impact. When he returned to school the next fall he found he hand been enrolled in remedial math despite aceing Pre-Calculus the previous year. He had to meet with the counselor to get into the right class.

Today, however, I want to explore the theories of education reform that drove the deployment of these exams. There are three prominent theories of reform with a few variations. Most contemporary efforts to improve education are based on at least one of these.

Theory: Standards and School Accountability
This is the primary theory represented by No Child Left Behind (NCLB). It's based on the broader theory that measuring something and reporting on those measurements will bring about improvement – especially if improvement is incentivized. It also represents the truism that if you don't measure something, you can't tell whether you've changed it for the better.

In order to bring about accountability, NCLB requires states to define learning objectives for each year or grade. These objectives are commonly referred to as the state core standards and each U.S. state has its own set. Furthermore, any public school receiving federal funding must administer a state-wide standardized test to every student in grades 3-9 and at least once in grades 10-12. Student scores are compared with previous years' results to determine whether they have achieved Adequate Yearly Progress (AYP). Certain consequences are tied to individual schools' success or failure to achieve progress for all students.

The core of the theory is this: If we set higher standards, measure against those standards and report performance then learning will improve. Unfortunately, 11 years into this experiment the quality of U.S. student learning is nearly flat.

There are numerous criticisms of standards and testing; but my personal concern is that by themselves they are a blunt instrument. In the absence of a proven formula for improvement the result is a form of natural selection – schools that underperform are taken out (actually they "receive interventions") while better performers survive. Natural selection is proven to work but it takes many generations and a lot of the population are brutalized before measurable improvement occurs.

Despite the lack of success, it's not time to abandon standards or accountability. Prior to 2002 most states didn't have well-defined core standards nor was student performance consistently measured. Now all states have standards, we are measuring regularly and 45 of the states have recently agreed to the Common Core State Standards. While standards and testing are inadequate remedies by themselves, they are important assets on which to build.

Theory: Highly Qualified Teacher
Where the Standards and Accountability theory focuses on school improvement. This theory focuses on teacher improvement. It's certainly intuitive; most of us have had one or more great teachers and we know they make a big difference. It's also justified by the data. Studies confirm that teacher quality is an important factor in student achievement and that the variation in achievement between classes within the same school is greater than variation between schools.

NCLB includes a mandate for states to supply highly qualified teachers to every student but it leaves it up to states to determine what it means to be highly qualified. And that turns out to be a problem. Studies show that certain teachers are consistently more effective than others; value added measures can identify which ones they are (albeit with a moderate error rate); but individual teachers often don't know what they need to do to improve.

In raw form this becomes another application of natural selection. If we reward teachers who perform well and eliminate those who don't then eventually performance will improve – assuming we don't run out of teachers beforehand. But many generations will be required and a lot of brutal actions will be taken in the meantime. No wonder there's so much controversy around teacher evaluations being tied to wages and promotions.

I'm actually in favor of merit pay for teachers so long as good quality performance measures are used. But those evaluations need to be deployed concurrently with professional development that informs teachers on how they are doing and what they can do to improve. Conveniently, resources are emerging to support that. For example, the Measures of Effective Teaching project used the Danielson Framework for Teaching to identify teacher behaviors that are well-correlated with student performance. These and similar frameworks can be used to inform teachers on how they can do better.

Even so, effective teachers alone are not enough. In our current educational system, teachers account for approximately 8.5% of variation in student achievement. School-, teacher-, and class-level factors combined account for about 21%. Meanwhile, background characteristics such as race, parental achievement and family income combine to account for 60% of variation in achievement levels.

So, if every teacher in the country was equivalent to our very best, it still wouldn't be enough to overcome the cycle of intergenerational poverty. To achieve that dream, we have to increase the influence school has over student achievement. That can be done by adapting the learning experience to the needs of individual students.

Theory: Personalized Learning
There's a pattern to these theories: The Standards and School Accountability theory introduces the concept of measurement and uses it to assess whole schools. The Effective Teachers theory takes those same measures and applies them at the teacher level. For this third theory, feedback is applied at the student level.

Personalized learning leverages the same standards as the other theories. It can also incorporate the same measures. However, annual testing alone is insufficient for personalization. Instead, understanding is measured weekly, daily or, in the best adaptive learning systems, continuously. Measurement must happen soon enough and feedback given quickly enough to affect learning activities. A truly personalized system selects activities according to student needs and also adapts to student behavior within an activity.

Bloom's Two Sigma experiments and the follow up work they inspired make me optimistic about Personalized Learning. These and other studies have shown that personalized learning experiences enabled by immediate feedback consistently deliver one to two standard deviations improvement in learning. We believe that is sufficient to overcome background factors thereby enabling a majority of students become high achievers.

Personalized learning is the natural result of 1:1 tutoring which is why tutoring is so effective. To do personalized learning at classroom scale generally requires 1:1 computers and a role change for the teacher as she shifts from "deliverer of knowledge" to "facilitator of learning." As with the other theories, there's a lot of skepticism and resistance to change. But pilot deployments are showing great promise.

Variation: School Choice
School Choice attempts to bring competitive pressure for schools to perform better. In this way, it's a variation on the Standards and School Accountability theory. Like NCLB, School Choice needs standards to be set and school performance must be measured against those standards. Performance is reported to parents who are expected to make an informed choice of which school their students should attend.

Since allocation of school funds is tied to enrollment, the theory is that schools seeking students will compete, not only on standards and their measures, but also on the basis of any other factor that's important to parents and students.

School Choice efforts include charter schools, magnet schools and voucher programs. The idea is to give public and private schools more freedom to experiment thereby accelerating the identification of viable formulas for improved leaning. Studies have shown this to be the case as the average of charter school outcomes is similar to that of public schools while variation among charter schools is much greater. Therefore, some charter schools are substantially better and should be emulated while others are substantially worse and should be shut down or reorganized. It's exactly this kind of variety and freedom that school choice advocates seek.

School choice can incorporate Highly Qualified Teachers and Personalized Learning. Indeed, since both of these theories have been shown to be effective, the expectation is that schools that incorporate these principles will be the best rated and will attract more students.

Variation: Small Classes
The small classes movement is based on studies showing that students learn better in smaller classes – all other factors being equal. But other factors are not equal. Lowering the student:teacher ratio costs a lot of money and other factors such as teacher skill have a greater impact than class size. For example, when California mandated smaller classes they had to hire many more teachers. For at-risk populations, the impact of less-experienced teachers overcame the benefits of smaller classes resulting in lower performance instead of the expected improvement.

Variation: No Excuses
The No Excuses model centers on maintaining high expectations for student performance without making excuses for external issues such as background, troubles at home and so forth. It's associated with charter management organizations such as KIPP and BES. Proponents emphasize pillars such as college expectations, culture of respect, voluntary participation and high discipline. They also have extended hours and extended school years. A key value is the whole school's commitment to each student's success. If a student is struggling or falling behind, they discover that early and engage counseling, tutoring and other supports to ensure the student succeeds.

No Excuses engages all three theories, overall school performance is measured, they hire and train highly effective teachers and they adapt the learning environment to the needs of individual students, albeit most No Excuses schools do adaptation with limited use of technology. Over the last decade, No Excuses schools have demonstrated that background factors can, indeed, be overcome by a supportive school structure. On the other hand, their high reliance on supportive interventions sometimes leaves students underprepared for the independent learning discipline required in college. Recognizing this, No Excuses organizations are updating their practices to better train students to become independent learners.

~ ~ ~ ~ ~

As with the variations listed here, most reform projects mix two or more of these theories. Even NCLB includes a mandate for Highly Qualified Teachers. Personalized Learning efforts are more common at charter schools than conventional public schools.

Education Reform will remain an important part of our civic dialog for a long time. Unsurprisingly, it means different things to different people. For some it's a moral crusade. To those being asked or forced to reform it's more threatening. All too often arguments about reform neglect the research (which is abundant) and fail to fully express the theories on which they are based. That shouldn't be the case as there are decades worth of data and studies behind each of these theories – sufficient for advocates and policy makers to make informed decisions.

The data tells those of us seeking to eliminate poverty that incremental improvement to existing schools is insufficient. Personalized learning with an eye toward training independent learners seems to be the most promising approach. Deploying this at scale requires whole-school changes to the way programs are funded, to the choices of curriculum and technology, and to the roles of educators.

Winds of Change: Higher Productivity in Higher Education

2013-02-21T14:40:00.002-08:00

Note: This first appeared last week as a guest post on the Next Generation Learning Challenges Blog. I highly recommend both the blog and the NGLC website.

My first lecture hall experience was American Heritage at Brigham Young University. The course was required for all freshmen and more than 500 of us at a time attended two lectures a week. In a third “lab” period we met with a TA. The professor was charismatic and the instructional design team supplied him with carousels full of colorful slides. Still, a large fraction of the class was asleep at any given time.

Large lecture hall courses are one common method of increasing productivity in higher education. Another is weed-out courses – those designed to convince students that they should choose another, less expensive major. For me the weed-out subject was Discrete Structures. This Computer Science subject is rich with metaphors like trees, maps, chains and links. It can be taught through story, modeling, manipulatives and real-world application. But our version was deliberately dry with an emphasis on precise vocabulary and obscure notational forms. The pass rate hovered near 50% and hundreds of students were convinced that they weren't capable of understanding computer science.

Higher education in the United States is sandwiched between twin pressures, increasing societal needs and expectations on one side with flat or declining funding on the other. To meet this challenge, institutions will have to dramatically increase productivity. But traditional productivity boosts like large lecture halls, weed-out courses or greater admissions selectivity won’t be enough this time around. What’s required is fundamental change to the way we support learning. We need a more personalized approach.

Societal Needs and Expectations

Employment projection is from the Bureau of Labor
Statistics Job Outlook. Supply is based on National
Center for Education Statistics data on annual
Computer Science BS degrees awarded. Attrition
is based on a 40-year career span.

While the U.S. unemployment rate hovers around 8%, there is a shortage of engineers and technicians. In 2012, the unemployment rate for software developers was only 2.8%. An Association for Computing Machinery study indicates that the United States will need more than 150,000 new computer scientists each year through 2020 yet our collective colleges and universities only produce 40,000 degree holders to fill those jobs. Healthcare workers are also in short supply. In 2012 the unemployment rate for physicians was 0.8%. For Physical Therapists it was 2.0% and for Registered Nurses, 2.6%.

At a recent Technology Alliance conference it was noted that colleges and universities in Washington State produce less than half as many engineers, technicians and software developers as the state’s employers consume. The rest have to be imported from other states or countries. A speaker from the University of Washington pointed out that they have increased introductory Computer Science enrollment from roughly 1200 to over 2000 per year. But the Microsoft representative responded that they have 3,600 engineering and computer science openings and they’re competing with Amazon, Boeing and many others to fill those spots.

Of 4.3 million freshmen who started college in 2004, only 2.2 million (or 51%) graduated within six years. This isn’t a perfectly accurate figure. Because of the way records are kept, it’s hard to count students who transfer and complete at a different institution. But inadequate record keeping is another symptom that institutions haven’t focused enough on ensuring their students are successful. Higher completion rates will save a lot of wasted student time.

As we move into the 21st century the fraction of unskilled jobs continues to diminish while those requiring advanced skills increase. It’s no longer appropriate to sort students by “aptitude.” We must give students the support and guidance they need to master advanced subjects.

The Funding Landscape

Education is the largest item in most state budgets. In California it accounts to between 52% and 55% of the state general fund. With the recession hitting state revenues and the expiration of stimulus supplements, state fiscal support for higher education dropped by 4.7% between 2011 and 2012, remaining flat in 2013. Overall, annual support has dropped by 10.8% since 2008. On a per-student basis, state and local financing dropped 24% in the 10 years preceding 2011.

At the same time, tuition is rising much faster than inflation. Tuition and fees at U.S. public universities rose 4.8% for the 2012 school year to an average of $8,655. At nonprofit private colleges tuition and fees rose 4.2% to $29,956. In addition to drops in public funding, the cost to provide education is increasing and the recession has diminished private endowments.

Total student debt in the U.S. now exceeds $1 trillion making it higher than the nation’s credit card dept. Student loans aren't a big problem if they are correlated with significantly higher earning potential. But loan approval is not connected with choice of academic major or the graduation rate of the institution.

Personalized Learning

The demands on higher education are greater than ever. We need more graduates – especially in certain fields. We need better completion rates. We need to support students in tackling challenging subjects. Moreover, we have to do this with flat or declining budgets.

The Bill & Melinda Gates Foundation has assembled representatives from a dozen colleges and universities that are trying new approaches with promising results. The Personalized Learning Network, as it's called, includes innovators like Western Governors University and American Public University; pioneering programs at Arizona State University and UC Berkeley; and NGLC grantees like the Kentucky Community & Technical College System, Rio Salado College and Southern New Hampshire University.

Recently I had the privilege of meeting with this group. There’s a lot of variation in their personalized learning programs but they share these common features:

Mastery Learning and Independent Pacing: Students have to master the current topic before moving to the next step. Self-pacing grants this freedom and ensures that there aren't gaps in understanding due to bad days or illness. And students don’t waste time on topics that they already understand.

High Expectations: The institutions make a commitment to support all students sufficiently so that they can master the material.

Feedback: Students and instructors are constantly informed about conceptual understanding and progress through the material.

Adaptive Learning: The learning system adapts according to individual student actions and performance.

Individual Attention: The programs facilitate abundant 1:1 time between students and faculty.

Motivation: Systems and attitudes that foster student motivation include interesting activities, student autonomy, recognizing good performance and avoiding frustration either due to anxiety or boredom.

All of this is enabled through strategic use of technology. Most use some form of blended online and in-person learning. The key point is not to simply add technology but to apply technology in the service of personalized learning.

Personalized learning programs should be able to address higher education pressures for better success and completion rates. But can they also help educate more students at lower cost? I believe so. Technology can automate many tasks that cost a lot of educator time. Video lectures are a personalization technology because they allow students to view on demand and replay as needed. Not only do they save the time in class but they also save the instructor time preparing the lecture. Objective assignments can be graded automatically and feedback given instantly to the student. Feedback to instructors can help them optimize their interactions with students. Subjective grading, while still consuming human time, can also be made more efficient. All of these factors help institutions increase capacity and reduce per-student costs.

Equally important are the savings offered to students. Immediate feedback helps students learn concepts more efficiently and avoids time wasted on misconceptions. Students can advance immediately upon understanding a concept and get credit for things they learned previously. And authentic learning activities support a better and more complete understanding of each topic. In one study by Carnegie Mellon’s Open learning Initiative they were able to teach students the same material in half the time with better retention.

Changing higher education is like turning a glacier. Features like accreditation, tenure, financial aid, credit transfer, and faculty autonomy interlock to form a seemingly insurmountable barrier protecting the status quo. But the twin pressures of increased expectations and diminishing funding result in an unprecedented incentive for change. Like the Maginot Line, traditional barriers won’t be overcome but simply bypassed.

The Common Core State Standards for Literacy are Two Dimensional

2013-02-13T09:24:00.000-08:00

The ideal school librarian would know every student in the school – what their interests are, what their current reading level is and what their teachers will be teaching next. With this knowledge, she would use her comprehensive knowledge of the school's book collection to suggest books or activities that would be both enjoyable and yet challenging to the student's abilities. That is, books that are in the student's Zone of Proximal Development.

It's not really possible for a librarian to have such a comprehensive view of both students and the book collection. But under Race to the Top grants, several states are developing Instructional Improvement Systems that, among other things, will support recommendations like these. Such systems operate at the intersection of student data and content data. And to support them, inBloom (formerly the Shared Learning Collaborative) is deploying student and content data services.

The Common Core State Standards (CCSS) and the Learning Resource Metadata Initiative (LRMI) work together to support the content data side when teaching reading and writing. The CCSS for ELA-Literacy have two dimensions to their basic structure. The grid below shows one way to view the Common Core Standards for Reading. Making up the horizontal dimension are Anchor Standards 1-9. These describe specific skills that the student should be able to apply when reading. The vertical dimension is Anchor Standard 10, the requirement that the other nine anchor skills should be demonstrated against texts of increasing difficulty as the student advances from Kindergarten to 12th grade. Notably, grades 9 and 10 share a level as do grades 11 and 12.

Common Core State Standards for Reading Literature

Here's an example of how this works: Anchor Standard for Reading number 6 states:

CCSS.ELA-Literacy.CCRA.R.6 Assess how point of view or purpose shapes the content and style of a text.

On the diagram this is marked with a vertical gridline. One of the horizontal gridlines is Reading Literature Grade 4 Standard 10. It's statement is:

CCSS.ELA-Literacy.RL.4.10 By the end of the year, read and comprehend literature, including stories, dramas, and poetry, in the grades 4–5 text complexity band proficiently, with scaffolding as needed at the high end of the range.

I've marked the intersection of these two with the identifier, "RL.4.6". The statement for Reading Literature Grade 4 Standard 6 is:

CCSS.ELA-Literacy.RL.4.6 Compare and contrast the point of view from which different stories are narrated, including the difference between first- and third-person narrations.

Notice how this last statement is a refinement of anchor standard 6 targeted at a Grade 4 skill level. So, a source text or learning activity that satisfies RL.4.6 would have a text complexity level in the grade 4-5 text complexity band and it would at least use a first-person or third-person narration. Ideally the activity would include both narration forms and give the student a chance to contrast the two.

So, what are these text complexity bands and how do we tell whether a text is within a particular band? In other words, how do we place a text or learning activity on the vertical dimension?

Appendix A of the Common Core State Standards for English Language Arts describes a three-factor model for measuring text complexity. The qualitative factor refers to levels of meaning, structure and demands for prior knowledge on the part of the reader. "Reader and Task" considerations involve matching texts to the reader's needs or interests and the learning tasks that will be associated with the text. The quantitative factor is a numerical measure that is calculated (usually by computer) from word length and frequency, sentence length, vocabulary and text cohesion. A supplement to Appendix A lists six approved scales for indicating quantitative text complexity for the Common Core. The table below indicates which levels are appropriate for certain grade ranges.

Common Core Band	ATOS	Degress of Reading Power®	Flesch- Kincaid	The Lexile Framework®	Reading Maturity	SourceRater
2nd-3rd	2.75-5.14	42-54	1.98-5.34	420-820	3.53-6.13	0.05-2.48
4th-5th	4.97-7.03	52-60	4.51-7.73	740-1010	5.42-7.92	0.84-5.75
6th-8th	7.00-9.98	57-67	6.51-10.34	925-1185	7.04-9.57	4.11-10.66
9th-10th	9.67-12.01	62-72	8.32-12.12	1050-1335	8.41-10.81	9.02-13.93
11th-CCR	11.20-14.10	67-74	10.34-14.20	1185-1385	9.57-12.00	12.30-14.50

The grid diagram also includes an example of how a source text might be fully aligned to the common core literacy standards. In this case, To Kill a Mockingbird is shown as an appropriate text for teaching standards 1-7 at grades 9 or 10. So, the LRMI metadata for To Kill a Mockingbird would include alignment to standards RL.9-10.1, RL.9-10.2, RL.9-10.3, RL.9-10.4, RL.9-10.5, RL.9-10.6, and RL.9-10.7.

On the vertical dimension, To Kill a Mockingbird is positioned toward the middle of the grades 9-10 range. So, it would be considered moderately advanced for grade 9 and moderately easy for grade 10. To Kill a Mockingbird is rated an 870 on the Lexile scale. A quick glance at the table shows that 870 is in the 4th-5th grade range. The book is positioned higher on the grid than the raw Lexile number would indicate due to qualitative factors such as the complex moral dilemmas posed by the text.

The LRMI metadata schema is designed to be flexible enough to represent all of these dimensions. The AlignmentObject type represents the relationship between a text or learning activity and a node in a framework or taxonomy. The most obvious and common way this is use is with a alignmentType of "teaches" or "assesses" and the target node being a statement in the Common Core State Standards. In the To Kill a Mockingbird example, the "teaches" alignmentType would be used with targets of the six standards (RL.9-10.1 to RL.9-10.7). Any one of these six standards also implicitly brackets the vertical, text complexity dimension. In order to more finely position a resource, LRMI also defines a "textComplexity" alignmentType. Publishers of at least two of the quantitative frameworks listed above are in the process writing guidelines for their use with LRMI. It's also possible to use LRMI to indicate non-quantitative factors. To do so, we would need to define taxonomies for qualitative and "reader and task" factors with appropriate identifiers.

In these examples I've used Common Core Standards for Reading but the writing standards have a similar two-dimensional structure. Overall, it's a rich framework with great promise for improving student literacy.

We have achievement standards (CCSS) and data standards (LRMI). There are emerging services like inBloom that build on these standards. I expect very soon a combination of CCSS, LRMI, open libraries of content and custom recommendation engines will offer students custom reading lists and writing activities tailored to their individual learning needs.

Personal Rapid Transit and Driverless Cars

2013-02-05T12:13:00.001-08:00

An ULTra PRT vehicle on a test track. (Wikimedia Commons)

As a teenager in the 1970s I remember reading about Personal Rapid Transit in a number of places including this Popular Science article. Unlike conventional transit like light rail or bus systems, a PRT system uses small, individually switched cars on a specially designed guideway. Upon entering a station, you select your destination on a console. Within a few seconds a 3-6 passenger car arrives and whisks you directly to your destination. At least that was the dream.

Of the dozens of proposals and prototypes, the Morgantown PRT that links WVU campuses is the only one of that era ever to be deployed at scale. The rest were either cancelled entirely or were defeatured into automated people mover systems like you find at many airports.

In 2011 two new systems opened, ULTra PRT at London's Heathrow Airport and the 2getthere system in Masdar UAE. Both are relatively small systems each with fewer than five passenger stations and fewer than 25 vehicles. But the new systems also represent an important departure from previous PRT designs. Both use battery powered vehicles with autonomous control. They run on rubber tires and steer themselves so there's no switching gear on the guideway. They are powered by batteries that automatically recharge when the cars wait at stations. This contrasts with previous PRT designs that used powered guiderails and a central control and switching system.

The primary barrier to PRT systems has been the cost of the tracks or "guideways." It's estimated that it would cost beween $30 million and $40 million a mile to expand the Morgantown system. That's because the guideway has to incorporate precision guide curbs, power transmission, track switches and even a heating system to melt snow and ice to keep it safe in bad weather.

In contrast, the ULTra guideway is estimated to cost between $7 and $15 million per mile. That's because it's a simple concrete pathway with no active systems.

Which brings me to Driverless Cars.

In essence, the ULTra and 2getthere systems are self-driving electric cars in which the environment has been constrained enough to simplify the self-guidance problem. High curbs make it easier for the cars to center themselves in lanes, dedicated roadways minimize pedestrian and obstacle avoidance. Strategically placed charging stations let them be electrically powered using batteries of modest capacity.

Meanwhile, Google's self-driving cars have driven themselves more than 300,000 miles accident-free, on conventional roads, without special infrastructure. Like many, I've wondered why Google is building such cars. They're in the information business, not transportation. A talk by Big Data guru, Ed Lazowska clued me in. Before the Google people let a car drive a route by itself, they first have a human drive the car over the same route. During the trip, its sensors scan the environment, picking out landmarks and obstacles, measuring road conditions and fine-tuning its GPS map of the roadway. Google is interested in supplying data to enable driverless cars and they're doing research to determine what data is needed.

A recent Freakonomics post on the subject suggested that driverless cars will arrive incrementally starting with the already common cruise control, adding adaptive cruise control, collision avoidance and self-parking before fully driverless operation arrives.

But I'm afraid that calling these "driverless cars" is the 21st Century equivalent of calling automobiles "horseless carriages". In each case the focus is on what's missing (the driver or the horse) instead of what new capacity has been introduced. "Horseless carriage" doesn't exactly describe a vehicle capable of sustaining 65 miles per hour with a range of over 300 miles. Nor does it conjure images of the megacities it enables or the endless parking lots it requires.

Consider this possibility: driverless technology enables the PRT dream on existing infrastructure. Instead of dedicated guideways costing tens to hundreds of millions, a PRT system built on driverless technology would rely on GPS and 3G data networks, both of which are already in place. Initial deployments can be restricted to certain neighborhoods that meet high standards of traffic signals, lane markings and crosswalk protection. Even a system restricted to certain lanes and certain streets would offer PRT of greater scale and capacity than anything yet deployed. Yet the investment to get started is regulatory permission, a few vehicles and some signage.

Fancy stations aren't required – only some curb space. Cars would be summoned using smartphones. And it wouldn't just be a peoplemover. Cargo, also, could be sent unattended. Grocery stores could use the same infrastructure for home (or corner) delivery. In the long run, even mail delivery and garbage collection could be automated.

We can learn something from this:

There are some fundamental principles at work here that can be applied to other large-scale problems:

Infrastructure is usually the most expensive component. Whenever possible, use infrastructure that's already in place and share infrastructure with other projects.
Push control (or decision making) as close as possible to the application or beneficiary.
Inform the distributed control with global data.
Build systems that can be scaled incrementally; where adding capacity is a matter of buying more of the same rather than periodic large investments to get to the next capacity threshold.

Consider the above principles applied to education. (You know I can't resist.) Existing infrastructure includes the internet, inexpensive computers and tablets, content development tools, video standards and so forth. Personalized learning relies on giving more control to the student and teacher to adapt learning to individual needs while being informed by common standards. And web-scale technologies are required if systems are to grow to support millions of students.

The "Wouldn't it be Cool" Department

Walt Disney World has the most heavily used monorail system in the world. They also have a well-maintained private road system connecting their resorts and theme parks. Wouldn't it be cool if Disney deployed a PRT system (based on driverless car technology) to connect their resorts and parks together? Such an attraction would enhance the Disney experience while proving the viability of the concept to the world.

Bloom's Two Sigma Problem Revisited

2013-01-23T12:32:00.001-08:00

Benjamin Bloom's Two Sigma Problem has been both a guiding framework and a challenge to educators for more than a quarter century. A bit more than a year ago I wrote about the problem and some of the ways people are approaching it.

Here's the concise version: Bloom and some of his grad students compared classroom teaching with 1:1 tutoring. In both cases they used a mastery-based curriculum. The tutored students performed two standard deviations (two sigmas) better than their conventionally taught peers. While it would be nice to have a 1:1 student:teacher ratio, Bloom acknowledged that it's not practical and he proceeded to research ways to achieve similar results using more scalable means. He published the study in 1984. Since then, the Two Sigma Problem has served as a benchmark of how well students can learn if given the right supports.

A recent meta-study by Kurt VanLehn of Arizona State University compares no tutoring (conventional classroom), computer-based Intelligent Tutoring Systems (ITS), and human tutoring. VanLehn notes that a number of well-known ITS efforts have shown one-sigma improvements over conventional instruction. So, the conventional hypothesis is that computer tutors achieve one-sigma gains while human tutors achieve two-sigma gains as compared to conventional instruction.

VanLehn set out to test that hypothesis. He selected numerous studies that collectively yielded more than 100 comparisons between conventional instruction, three forms of ITS, and human tutoring. The result is surprising: answer-based ITS achieved an improvement of 0.31 sigma over conventional instruction. Step-based ITS achieved 0.75 sigma and human tutors achieved 0.79 sigma.

This is mixed news. On the one hand, the best computer tutors are almost as good as human tutors. That suggests that we can scale up much more effective learning than is achieved in conventional classrooms. On the other hand, VanLehn found no replication of Bloom's 2 sigma results. Is Bloom's goal out of reach or is there another factor involved?

To find out, VanLehn retrieved the dissertations from Bloom's grad students that contributed to the more famous paper. One key experiment yielded an effect size of 1.95 sigma – the probable source of Bloom's Two Sigma challenge. In that experiment both the conventional classroom and the tutors used a mastery learning technique. Whether in class or being tutored, students took a quiz after studying each unit. If their score achieved the mastery threshold, they advanced to the next unit. If not, they studied the unit more and were assessed again. This process was repeated until the mastery threshold was achieved.

The missing piece is that classroom students were required to achieve mastery threshold of 80% before advancing. Meanwhile, tutored students were required to achieve a threshold of 90%. Could it be that adjusting the mastery threshold could account for a full standard deviation improvement in achievement? If so, numerous online learning systems should be tuned accordingly.

Oleg Bespalov and Karen Baldeschwieler, with their colleagues at New Charter University, have evidence to confirm this hypothesis. In their ITS system, students receive periodic formative assessments in the form of multiple-choice quizzes and self-graded short answer questions. From these assessments they calculate a "readiness score" to help students know when they're ready to advance. Students aren't constrained by the score – merely informed.

This creates a natural experiment in which they can compare student performance on the final exam against individual readiness scores. They discovered that students with a readiness above 90 achieved a 98% pass rate. But for those with a readiness score in the 81-90 range the pass rate dropped to 69%.

Both of these projects indicate that there's a critical threshold somewhere between 80% and 90%. Clearly this is an area deserving of more experimentation and research. But we can already tell that that tuning the mastery threshold is a critical factor for improving student achievement.

Measures of Effective Teaching

2013-01-18T09:31:00.002-08:00

The Measures of Effective Teaching Project (MET) released its final reports last week. It got considerable press coverage as the study strives to inform teacher evaluation programs, a subject of considerable controversy.

Most of the stories, like this one from Reuters, focus on the the study's finding that teacher performance can indeed be predicted by performance measures. The best evaluations involve a weighted average of student test scores, teacher observations and student evaluations. Any one of these by itself is a much less accurate predictor.

There are nuances to this that can be gleaned from the project's Policy and Practitioner Brief:

The different measures (student testing, teacher observation and student evaluation) have some overlap but mostly they measure different aspects of the teacher's skills.
Different weightings are better predictors of different outcomes. Unsurprisingly, placing greater weight on test results is a better predictor of future student test results. However, equal weighting models or those that emphasize teacher observations are more reliable year over year.
Effective teacher observations are more than a periodic visit from the principal. Evaluations require a consistent framework and procedure. The MET project used the Danielson Framework for Teaching as a rubric. The reliability of teacher observations is greatly improved by having at least two evaluators.
When done properly, student evaluations are very reliable and an important component of teacher evaluation. As with observations, the key is to ask the right questions. The MET project used the Tripod Student Survey.
The "value added" theory is supported. When student scores are compared with the previous year's performance (a value added score) the result is a more consistent predictor of future teacher performance than just the most recent year's scores.

One problem with exclusively using standardized tests to evaluate teachers or schools is that it's a blunt instrument. These tests offer a measure of performance but they offer limited guidance to a teacher or school on how they can improve. Sure, we can fire ineffective teachers and close ineffective schools. But using natural selection to improve schools is slow and costly not to mention cruel. Basically you're just hoping for those teachers and schools that randomly find the right formula for success.

Among the advantages of teacher observations and student evaluations are that they supply rich feedback to teachers to help them improve their practice. Another MET project report, Feedback for Better Teaching, offers guidelines for using feedback. They placed cameras in classrooms and observers codified the techniques used by the teachers. The same video recordings were used by the teachers themselves to observe their own performance – usually with an instructional coach. Processes like these can continuously improve teacher skills and effectiveness.

I've written before about how immediate feedback can help the student learn more effectively. In that context, it's no surprise that feedback to teachers helps them to be more effective. Moreover, it supports the passion that got them into teaching in the first place.

Enterprise-Scale is not Web-Scale

2013-01-09T18:46:00.002-08:00

In the 90s, the IT world was talking about Enterprise-Scale. It's not that enterprise-scale was anything new. But up until then, the enterprise was the domain of mainframe and minicomputers. Upstart microcomputers – those with the whole processor on a chip – had previously not been capable of enterprise-scale operations.

It took a lot to achieve enterprise-scale with microcomputers. The leaders included Sun, Oracle and Cisco with Microsoft, Intel, Compaq and others playing fast-follower. They invented RAID arrays, symmetric multiprocessing, storage area networks, network load balancing and much more in the pursuit of five nines of reliability.

As microprocessor-based computers achieved enterprise-scale, pioneers like Google, Amazon, Yahoo!, SalesForce and others pushed right past enterprise into web-scale. User counts were measured in hundreds of millions, storage capacity was measured in petabytes and server arrays numbered in the thousands. Unlike enterprise-scale where key technologies had already been invented by the mainframe world, there wasn't any precedent for web-scale and the pioneers had to invent their own methods. I happened to work at Ancestry.com in the late 1990s/early 2000s and got to participate in some of that invention. But it wasn't until later in the decade before the pioneers started to share what they had learned and build products like Amazon Web Services, Google App Engine and Windows Azure to support the web-scale developer.

This is a important issue for education technology. The education industry is a bit behind the curve in moving to web-scale. For example, most learning management systems are built for enterprise-scale. They are intended to be installed on dedicated servers at a college or university's data center and they're architected to handle tens of thousands of students and teachers.

What happens when you move to the K-12 market or to community colleges? These organizations don't have the data centers and skilled staff needed to deploy, maintain and backup enterprise servers like these. In the past, their data systems only had to handle a few hundred or maybe a few thousand administrators. Teachers and students didn't directly access the district's data systems.

But districts are rapidly bringing all of their students and teachers online. And that means two orders of magnitude more users. Many districts have student counts numbering in the hundreds of thousands. Some get into the millions. And since their technology staffs are already overburdened, they seek hosted solutions, not enterprise-scale servers they have to manage themselves. Hosting a single district might not reach web-scale but a cost effective provider would serve hundreds of districts. And web-scale technologies can reduce the cost to something that districts can afford.

Here are some of the principles of web-scale architecture. For purchasers of products and services, these are the things you need to look for. For developers of those services, these are the principles you need to incorporate into your design.

Always Available
Web scale services use redundant servers to ensure that the service is always running – even during software upgrades and system maintenance. The term "24/7" was invented to describe services that have no weekly scheduled downtime. (And please don't write 24/7/365.)

Billions of Database Records
Today a district might keep a few dozen data points per student per year. In a district with 50,000 students that amounts to around 150,000 database rows per year. Eventually the database might grow to a few million rows total.

But personalized learning applications can collect thousands of data points per student per year. And an online service might serve a few million students. Thus, a web-scale learning service should be designed to store billions of data elements with provisions for orders of magnitude growth beyond that as clickstream data become more important.

Single Sign-On and Identity Management
Today's schools typically run a Student Information System, a Learning Management System and a few custom learning systems for specific subjects. Most of these applications have their own user database mandating separate logins and making requiring a lot of data entry to provision the sytems.

The low-hanging fruit is a single sign-on system that lets students and teachers use the same login account across all systems. But single sign-on is of limited value without automatic provisioning. So it's more important to have an identity management system that automatically shares demographic and enrollment information between the Student Information System and the various learning systems. The long-term need is to integrate the data among all of the systems so that all student performance data is accumulated in a common database.

Services-Integrated Security
Consider security in a Student Information System. The student, her teacher and her parents should have access to her school records, but no-one else (except maybe the principal or a counselor). Enterprise-scale security manages things like this through access control lists (ACLs). Record or element has an ACL granting certain levels of access to specific individuals. For example, the teacher has permission to view and change grades while the student and parent only have permission to view them.

The ACL approach becomes fragile at web scale. With millions of students and parents and thousands of changes to class enrollment it becomes difficult to maintain correct ACLs even when the process is automated. Roles and groups offer some relief but inevitably permission errors creep in and they become a technical support nightmare. Even worse, with regulations like FERPA in place, permission errors can result in significant liability.

Instead, web-scale applications use policy-based permissions. When a student is enrolled in a class, the policy says the teacher should be able to access that student's records. There's no ACL to be updated and permission naturally disappears if the student changes enrollment. The databases of these systems describe the relationships between elements (students, classes, teachers, parents, etc.) and the policies describe how permissions should be granted according to those relationships.

Services-integrated security also means that permissions are enforced at all levels of the system. The UI will control permissions that are offered to the user and the API enforces policy when read or write attempts are made. Thus a rogue or buggy application is still prevented from violating security policy.

Developer Note: Policy-based security can be processor and database intensive. The query to determine whether to permit a particular operation can easily be more expensive than the operation itself. This isn't a reason to reject the approach. Rather, use multiple levels of permissions caching to reduce the database burden.

Linear or Sub-Linear Cost Curves

If you graph number of users on the horizontal axis vs total cost on the vertical axis enterprise-scale systems have costs that grow exponentially. This is because they achieve scale by using progressively bigger and more complex servers and one 32-processor server costs many times more lot more than 32 single-processor servers.

In contrast, web-scale architectures have a linear or sub-linear cost curve. They achieve this feat by using software, database and hardware architectures that spread the load across many commodity-priced servers. As demand increases you simply add servers so variable costs are linear and fixed costs get diluted over a large number of users.

Developing scale-out software like this is complicated and expensive. Because of this, enterprise-scale architectures can cost less in enterprise-scale deployments. But when user counts get into the hundreds of thousands or millions, web-scale becomes more cost effective.

Mashups
Web-scale applications rarely stand alone. In most cases, they are combined with other applications to create a complete solution. This is certainly true of our vision for personalized education. A complete solution includes at least the following:

Student Information System
Student/Parent Portal
Teacher Portal
Adaptive Learning System (probably multiple subject-specific ones)
Content Library
Assessment Bank
Analytics (Teacher, Department, Schools)
Interactive Professional Development

Plus, other innovative applications are likely to emerge. In a realistic system, these components will originate from a variety of sources. For time-constrained teachers and students to use them effectively, they will have to be seamlessly mashedup together.

Tools and Techniques for Web-Scale

At Ancestry.com we had to invent many of the web-scale tools we used. But the toolkit has matured in the last few years. The easiest approach is to build on one of the cloud platforms like Windows Azure, Google AppEngine or Amazon Web Services. The downside is that doing so locks you into that vendor's hosting service.

Here are some of the other tools and techniques that help in web-scale development:

Ultimately, however, you can use all of these tools and still not have a web-scale application. You have to architect for web-scale from the very start.

Game Design and the Zone of Proximal Development

2012-12-18T07:00:00.000-08:00

It's an experience many parents have had: From time to time my kids invite me to play video games with them. We pick a multiplayer game like MarioCart or Halo and they proceed to beat me silly. I keep trying to show moderately credible performance with little success. And I wonder, "How long would I need to play this game to achieve some degree of proficiency?"

Part of the problem is that parents like me jump right into the multiplayer mode competing against our kids while simultaneously trying to learn the game mechanics. Most of these games have a single-player "campaign" mode. The campaign is designed both to be consistently entertaining and to step-by-step make you better at playing the game. The two primary rewards of the campaign are progressive discovery of the story (narrative) and progressive mastery of new skills.

The Flow Channel in Game Design

In his book, The Art of Game Design, Jesse Schell describes "flow", a concept he adopted from psychologist Mihalyi Csikszentmihalyi. An individual is in a "flow state" when he or she is entirely involved in the task. "The rest of the world seems to fall away and we have no intrusive thoughts." It's a state of sustained focus and enjoyment.

Figure 1: Flow Channel

In figure 1 we see four player states on a graph of player skill vs game challenges. This can apply to a variety of games from physical sports to puzzles to first-person shooters. In this example a player is at state P₁ where the player's skill is balanced with the challenge of playing. With practice, the player increases in skill and advances to state P₂ in which the game becomes easy enough that the player is bored. P₃ represents condition when the game is too difficult; perhaps the opponent is a lot more skilled or the game is too hard. In this case the player becomes anxious about their performance. Both states P₂ and P₃ can be rebalanced. The game can become more challenging (P₂ to P₄) or the player can gain skill (P₃ to P₄). But if the player remains in Anxiety or Boredom for very long they'll abandon the game because both anxiety and boredom lead to frustration.

Figure 2: Growth Path in the Flow Channel

One goal of game design is to keep the player in the "flow channel." Here the player experiences the flow state continuously while both skill and difficulty gradually increase. But it's usually not a straight line centered in the flow channel. It's more a zig-zag with alternating rewards of easier wins and greater challenges. These match up to Dan Pink's "mastery" and "purpose" motivators.

The Zone of Proximal Development

Russian psychologist Lev Vygotsky defined the Zone of Proximal Development (ZPD) as the area between the tasks that a learner can do unaided and the tasks a learner cannot do at all. So, tasks in the ZPD are those that the learner can do with assistance – with scaffolding. Vygotsky claims that all learning occurs within the ZPD.

Figure 3: Zone of Proximal Development (ZPD)

There are many ways to put this into practice. For example, the Lexile Framework and other text complexity measures allow the matching of reading materials to the student's reading ability. Texts that are close to students' abilities increase confidence while more challenging texts increase skill levels. Texts that are too easy (boring) or too hard (anxiety producing) can be avoided. Mathematics is a structured subject where concepts build upon each other. Therefore, concepts in a student's ZPD are those that build on concepts the student already understands.

Integration

The Flow Channel offers a model to maximize player engagement and enjoyment. The Zone of Proximal Development is a model for optimizing learning productivity. Similarity between the two isn't surprising. Csikszentmihalyi studied and built on Vygotsky's work.

One of the most important things that educators can learn from Flow is that boredom contributes just as much to frustration as anxiety does. Conventional schooling does a lot of redundant work to ensure that most students "get" each concept. The boredom that results from such redundancy means that students rarely experience Flow in their schoolwork. It's also inefficient because students spend a lot of time below their ZPD in which case they aren't learning. Staying within the Flow Channel/ZPD can ensure that effective learning occurs and simultaneously keep the student motivated and rewarded.

It brings whole new meaning to, "Go with the Flow!"

As We May Teach

2012-12-05T07:00:00.000-08:00

"My education was very similar to that of my parents. Theirs didn't differ a lot from my grandparents'. My children's schooling has been enhanced by media, word processing and the internet but the experience isn't fundamentally different from my own. They still go to a classroom, sit at relatively small desks and try to pay attention to a teacher in front of a board. The transformation of primary and secondary education in the United States is beginning now and will be well underway within a decade."

Four years ago I wrote the essay, "As We May Teach" to the Meridian School board of trustees where I was serving. I recently re-read the essay and it's just as relevant today as it was then; so I've posted it here.

The examples I used remain valid but we now have many more. Since then, "Blended Learning" has emerged as the term to describe the Cirrus High School experience I use in the essay. Rather than attempt to list the numerous new examples, I recommend you check my colleague, Scott Benson's "Running List of Blended Learning Resources."

In the essay I reference Clayton Christen's prediction that 5% of high school teaching would be online by 2012 and 50% by 2018. The 2012 edition of iNACOL's "Keeping Pace with K-12 Online and Blended Learning" report estimates that 5% of US K-12 students are taking part in at least one online course. So, four years after the prediction we seem to be on track.

I've elaborated on several themes from the essay on this blog:

Unique to this essay is the application of Business Process Automation to the education space. It's a useful lens that's compatible with other approaches to educational improvement.

Learning from Data - An Automotive Example

2012-11-30T08:29:00.003-08:00

Monday saw me driving 800 miles home from a family Thanksgiving celebration. Due to my wife's change in plans, my only companion for the drive was our small dog (who had a narrow escape the day before). I needed something to keep my attention. So I decided to perform an experiment in data collection. I learned a lot even from a small data sample.

The vehicle I was driving was a 2010 Subaru Forester. Some friends have the same vehicle and have been pleased with getting around 27 MPG on the highway. We typically get only 23-24 MPG on the highway and I had been wondering why.

Among the features of this car is an average gas mileage display that's tied to the trip odometer. So, sampling the gas mileage is as simple as setting the cruise control, resetting the trip odometer, driving a set distance and reading out the result. As I was crossing the relatively flat plains of Idaho (speed limit 75) this seemed to be a good opportunity to gather some data.

Over a period of several hours, I took a bunch of samples following the above method and using my GPS to track altitude changes. I abandoned samples where the altitude change was more than a few hundred feet. The result is 27 good samples. I've posted the raw data here in case you want to play with them. Most of the samples are for 20 mile segments but some are as long as 40 and some are as short as 5 miles.

As you can imagine, the lower-speed samples got a bit tedious. But I was curious enough that I even took a side trip on a remote road (off the freeway) to get samples below 45 MPH. There's considerable variability in those results as you can see in the plot below. Halfway through the trip I refilled with fuel. I switched from regular (87 octane) to premium (92 octane) to see how that might affect mileage.

2010 Subaru Forester Fuel Economy vs. Speed

The results certainly aren't what I expected. EPA city and highway ratings have always led me to expect relatively flat miles per gallon with city being much poorer due to stop-and-go driving. Instead, I got a nearly linear downward slope. Using Excel's curve-fitting feature and found that a polynomial curve worked better than a line. The formula is embedded in the graph above.

More data points would be required to really validate this curve but it certainly fits within the margin of error of my samples. Therefore we can make some cost estimates using this formula. Notable is that peak economy is between 40 and 45 MPH – much slower than I had expected. From this I was able calculate the cost of each hour saved on this long drive.

Here's a table of results for a 800 mile drive in a 2010 Subaru Forester with average fuel cost of $3.759 per gallon. Time saved and additional cost are from a baseline speed of 55 MPH. Note that the distance is cancelled out in the Cost Per Hour Saved so those numbers are accurate regardless of the length of the trip.

Speed	MPG	FuelCost	Time	Hrs Saved	Addl Cost	Cost Per Hr Saved
55	33.1	$90.83	14.55
60	32.1	$93.71	13:33	1.21	$2.89	$2.38
65	30.7	$97.88	12.31	2.24	$7.05	$3.15
70	29.0	$103.65	11.43	3.12	$12.82	$4.11
75	27.0	$111.55	10.67	3.88	$20.72	$5.34
80	24.6	$122.45	10.00	4.55	$31.62	$6.96
84	22.4	$134.31	9.52	5.02	$43.48	$8.66

Here are a few things I've learned from this:

My friend with the other Forester drives slower on the highway than I do.
I had not known how sensitive vehicle gas mileage is to speed.
Everything I have read led me to expect no benefit from higher octane fuel once the vehicle's requirements have been met. In the case of the Forester, higher octane actually reduced fuel economy. This observation is confirmed by the official EPA ratings.
I would love to see tables like the one above before purchasing my next car.

I learned a lot from this tiny data sample and my future driving habits will be changed accordingly. Now imagine what we could learn if there were a large public database of fuel economy data. Car manufacturers could optimize for specific driving patterns. Consumers would be better informed about fuel economies to expect. There are fleet tracking devices like this one that are already reporting that data but it's locked up in private databases. If anonymous fuel economy data (speed, distance, altitude and MPG) were released there's a lot we could learn about fuel economy under a variety of conditions.

I can't wrap this post up without relating it to education. A relatively small data sample taught me a lot and will impact my future driving behavior. In the same way, it doesn't take a lot of data fed back to students and teachers before they see opportunities to improve. And when we grow from little data to big data, revolutionary changes are on the horizon.

A Post-LMS Framework for Personalized Learning

2012-11-19T12:41:00.001-08:00

In the last few weeks I presented at iNACOL VSS and attended Educause. I’ve also met with the Shared Learning Collaborative team and the CEDS Stakeholders group. Educause included meetings with the Next Generation Learning Challenges organizers and grantees. All in all it’s been a concentrated opportunity to meet with vendors, standards developers and visionaries in the personalized, blended and online learning spaces.

There’s a new pattern emerging on how the technical components of a personalized learning system fit together and it’s a departure from the past. This model seems to apply both in K-12 and postsecondary education.

This new framework is being driven by three trends:

Innovative creators of courseware and learning systems need greater control over the learning environment than can be achieved in a Learning Management System (LMS).
Student Information Systems (SIS) and Portals are taking over responsibility for student/teacher communications, gradebooks and consolidating analytics into student and teacher dashboards.
Students and Teachers are seeking a coherent and seamless experience without separate credentials and logins for each of the systems they use.

A New Framework

The figure below shows the interaction of three systems. Each system may be hosted by a different provider but they’re integrated in such a way that the student should browse between them seamlessly.

The Student Information System (SIS) is generally integrated into the school’s portal. This is the site a student browses for consolidated access to all school information. It’s provided and managed by the school. The portal links to courses in which the student is enrolled.

In this new model, courses are an integrated experience delivered by learning systems custom-adapted to the subject matter. At a basic level, a course is a sequence of learning and assessment activities such as exposition (video, audio, text), virtual labs, exercises, quizzes exams and so forth. Key to personalization is that the selection and order of activities is adapted according to individual student needs.

Traditionally, the same learning system that hosts the course also hosts the activities. This is reasonably simple with conventional media types such as text and video. It gets more complicated with interactive media and assessments. The most innovative learning activities may be separately hosted because they are supported by custom services. These could include interactive labs, intelligent tutoring systems, virtual worlds and games.

Conspicuously absent in this new model is the Learning Management System (LMS). For the last decade or so, the framework has been that schools select and deploy an LMS – ideally with single sign-on and data integration with their SIS – but all too often as an independent system. The idea was that courseware publishers and instructional designers would install the course materials into the LMS using content packaging formats like SCORM and Common Cartridge. But this hasn't happened very much – especially with the most innovative courses. Cutting edge learning systems like DreamBox, Aleks or Read 180 can’t be packaged up and installed into an LMS. The environment is too constraining.

While LMSs are capable of much more, most actual LMS use is in support of teacher-student and student-student communications, not for delivery of instruction. And that communication function is being taken over by the SIS and portal. Contemporary SIS systems have expanded beyond enrollment and course-level data to include full gradebook functionality. Meanwhile, portals are including teacher and student dashboards, online forums, chatrooms and other communication features.

So the new model is composed of Portal/SIS, Learning Systems and Activities often supplied by different organizations. And it’s not just three systems that need to be integrated. A single school will likely have many learning systems. A single student is likely to use different learning systems for different subjects. And a single course may integrate activities from a variety of sources.

Student ID

In order for the student and teacher experiences to be coherent there needs to be a clean handoff between these systems. In the diagram I've shown this as Student ID flowing to the right and Student Data flowing both ways. Student ID may include authentication, authorization and/or provisioning.

Authentication, often provided by Single Sign-On (SSO) is the real-time indication of who the student is.
Authorization is a real-time assertion that the student should be granted access to a system or resource.
Provisioning is the transfer of teacher and student enrollment data so that a learning system or activity can grant access and coordinate a cohort of teachers and students. This may on-demand (coordinated with authentication or authorization) or it may be a periodic batch update.

Depending on features of each component, these work together in different ways. For example, an SIS may transfer provisioning data to a learning system. Then, at runtime the SIS uses an SSO protocol to authenticate the student to the learning system. At this point the learning system knows the identity of the student and the provisioning of the classes, therefore it can internally decide whether to authorize access.

On the other hand, the learning system may use an authorization protocol to grant student access to a learning activity without authentication or provisioning. In this case, the activity provider doesn't know the identity of the student, it only knows that a trusted agent (the school) has indicated that the student should be granted access.

Student ID protocols can transfer three levels of information depending on the needs of the systems:

Personally Identifiable Information (PII): This might include the students name, grade, enrollment information and so forth. It's sensitive information governed by FERPA regulations.
Persistent Identifier: This is just enough information that a learning system or activity can identify repeat visitors. The system doesn't have any personal information about the student but knows this is the same student as in a previous visit.
Authorization Ticket: This is just a trusted indication that a student should be able to access content. The learning system or activity is not assisted in coordinating repeat visits.

Student Data

Most of the student data flow is upstream as student activities and performance are reported to the Learning System and the SIS/Portal. Traditionally that data has been simple scores and grades. But systems are beginning to collect richer information like frequency of access, time on task and clickstream data. these are used in analytics such as student and teacher dashboards. This same data can also be reported downstream for the use of adaptive learning systems and custom activities.

Protocols

The difficulty is that there isn't much consistency in the protocols used for Student ID or Student Data. To their credit, the builders of SISs, Learning Systems and Tools all have APIs for integration with other systems. But in most cases APIs are custom to the application. And upstream systems aren't necessarily prepared to collect the rich data that downstream systems are prepared to share.

Here's a survey of what is available or under development:

SAML and OAuth are two commonly-used protocols for authentication and authorization. The SSO subset of SAML has become common due to its use by Google Apps. OAuth is an authorization protocol that can optionally carry personal information or a persistent ID according to needs. Shibboleth is an open source reference implementation of SAML.

IMS Learning Tools Interoperability (LTI) supports the interaction of Learning Systems and Activities. It incorporates OAuth for the authorization step. LTI v1.0 (also known as Basic LTI or BLTI) coordinates the authorization of the activity (called a Learning Tool) seamlessly embedding it in the Learning System. Later versions of LTI support reporting of simple student performance data. Most mainstream LMSs support LTI 1.0 or better.

LearnSprout and Clever are two companies supporting data integration with SISs. This allows builders of Learning Systems to write to one API (either LearnSprout's or Clever's) and gain integration into a number of prominent SISs. However, they are limited to the data types supported by the SIS.

The Shared Learning Collaborative (SLC) is building a web-scale common student data layer that can be used by the SIS, Portal, Analytics, Dashboard, Learning System and Activities. A rich set of data types is pre-supported and applications can store custom data for persistence and sharing. It also supplies a common student identity framework including authentication services. So, in the SLC instead of handing off student data between systems, they all rely on the same underlying service.

The SLC Approach to the New Model

The new model divides the functions once concentrated in the LMS. Today, custom systems integration must be done to achieve a seamless experience. But protocols and services are under development that should simplify that in the future.

Video: Feedback Loops for More Effective and Personalized Learning

2012-11-17T22:44:00.001-08:00

Last month I presented at the iNACOL VSS conference. I posted my slides and resources here.
I experimented with using a Bluetooth microphone and PowerPoint's recording feature to generate a voiceover video of the presentation.
I apologize in advance. The audio quality is fairly poor. It's especially bad at the beginning but improves later. I think I was near the range limit of the microphone. And PowerPoint's recording/video feature is still buggy. In a couple of slides, the sound drops out entirely.
Flaws aside, I'm pleased with how the subject came together. Quality feedback loops are a key component in personalized learning solutions. In researching this topic I found a lot of relevant research that can guide the development and selection of products.

Election Technology Update

2012-11-06T19:47:00.000-08:00

It's election day in the U.S. and most of us are fatigued by the campaigns and will be relieved to have them over. Barring an electoral college anomaly there will be more voters who are pleased with the result than dismayed by it (it's a tautology).

I wrote about my misgivings with touchscreen direct entry voting systems in 2009. Things haven't improved since then. The big risk is indetectable vote manipulation. Of course all voting systems, whether electronic or paper, are subject to some form of manipulation. The key is to set up protocols so that manipulations leave evidence. For example, paper balloting systems often count the number of ballots cast and compare that with the number of ballots counted. The number of ballots cast is transmitted to the counting location through a different means from the transmission of the ballots themselves.

In 2010 I wrote about King County, Washington's vote-by-mail system. In addition to mailing ballots, voters can deliver them to dropboxes conveniently located around the county. Not only do they save postage, dropboxes appear to be a more secure delivery method as observers from both major parties watch the sealing and collection of ballot boxes. Other observers watch the opening and counting processes.

As a paper and optical scan method, King County's is among the more secure – once the ballot is delivered to a dropbox. The glaring weakness is privacy. Vote-by-mail opens the door to voter coercion because there's no inspector and booth to ensure privacy when the vote is actually cast. It's entirely possible for others to pre-fill the ballot and simply ask the voter to sign – with intimidation if necessary.

Of course, manipulation of this sort doesn't scale well. Sure it can happen in pockets but widespread, coordinated vote manipulation would be hard to achieve as the more voters are intimidated, the greater the likelihood that someone complains. Therefore it's reasonable to assume that deliberate manipulation will be a small fraction of total votes cast.

This leads to an interesting conclusion: Though we aspire to make every vote count, there's some degree of error regardless of the way votes are cast and counted. Sometimes it's deliberate fraud, manipulation or intimidation. Sometimes it's poorly designed ballots, miscalibrated voting machines or natural disasters. There are two ways to deal with this. Our current system presumes that if the difference in votes is within the margin of error, democracy is preserved regardless of which candidate takes office. That presumption was tested in the 2000 U.S. election.

The alternative is to require another election if the vote is within the margin of error. That approach carries a tremendous cost in terms of time, money and extended uncertainty. Despite misgivings, I have to agree with those who wrote the constitution. I may not like the outcome when the vote is close. I may even believe that the count is inaccurate. But I do believe that Democracy is preserved.

Feedback Loops for More Effective and Personalized Learning

2012-10-23T12:18:00.000-07:00

Today I'm presenting at the iNACOL Virtual Schools Symposium on "Feedback Loops for More Effective and Personalized Learning." Here's the slide deck and resources related to that presentation:

Learning Maps, Common IDs and the Common Core

2012-10-22T14:55:00.001-07:00

Today we presented at the iNACOL VSS conference on "Learning Maps, Common IDs and the Common Core. Here are primary resources associated with that presentation:

Update: 7:55pm: In addition to the above primary sources, I've written the following on the same subject:

Also I corrected the link to LearningRegistry.org.

Many thanks to the panelists: Maureen Wentworth, Michael Jay and Sharren Bates.

Things Every Education Tech Entrepreneur Should Know

2012-10-18T19:20:00.000-07:00

This weekend I'm volunteering as a coach for Startup Weekend Edu in Seattle. Preparing for this got me to thinking about things people building education technology should know. The following list isn't comprehensive but it's a good starting point. Follow the links to learn more about these topics.

Theories of Change
You need to have a good theory of how your technology will improve education. There's a lot of money to be made in record keeping and ERP-type applications. But the things that interest me and I hope interest you are those that directly improve student learning. And you need to be specific about the expected improvement. Do you want students to learn more in the same amount of time or take less time to learn a skill? Are you seeking better comprehension and retention? What about "deeper learning" – getting beyond recall and demonstrating the ability to apply concepts or solve problems.

Most ed tech theories of change start with Bloom's Two Sigma Problem. In a 1984 paper, Benjamin Bloom discussed how they had achieved two standard deviations improvement in student learning through a combination of Mastery Learning and one-on-one tutoring. Noting that 1:1 student-teacher ratios are impractical, Bloom's challenge is to find scalable ways to achieve the same results.

The following resources should stimulate your theoretical juices:

The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring. Benjamin Bloom's 1984 Two Sigma Paper. Here's an earlier introduction I wrote earlier on this blog.

A 2011 Metastudy by Kurt VanLehn gives a progress report of Intelligent Tutoring Systems and an update on progress toward Bloom's Two Sigma Problem. In particular, see page 210 (the 15th of the paper) in which VanLehn explains that about half of Bloom's two sigma gains were due to changes in Mastery Learning parameters.

Personalized Learning is "instruction that is paced to learning needs, tailored to learning preferences, and tailored to the specific interests of different learners. In an environment that is fully personalized, the learning objectives and content as well as the method and pace may all vary." This definition is from the National Education Technology Plan which is an excellent read so long as you skip the executive summary.

Higher Ed Disruption, Not So New. Alexandra Logue gives a history of Personalized Learning theories dating back to the early 20th century.

Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. The ultimate meta analysis, John Hattie reports the effect size of 138 factors on student learning along with nuance explanations of the what is known about each factor. I summarized some of the high points in a recent blog post. You should seriously check this book first before running with some theory of educational improvement.

Let's Use Video to Reinvent Education. Salman Khan speaking at TED. Notably, at 14:30 he talks about measuring individual teacher-student time as opposed to teacher to student ratios. He also introduces the flipped classroom concept.

As We May Teach. My own essay from a couple of years back in which I apply the IT principles of Business Process Automation to teaching.

Curing Baumol’s Disease: In Search of Productivity Gains in K-12 Schooling. Paul Hill and Marguerite Roza discuss the long delay from the introduction of automation to industry and measurable productivity gains. Then they apply the concepts to public education.

Changing the Rules to the Game of School. My presentation at VSS 2011. Games are composed of a goal, rules, feedback and voluntary participation. So is school. What can we learn from this?

Cognitive scientists talk about the Zone of Proximal Development. Game designers talk about Gameplay Progression. They are similar concepts and they both involve motivation and increasing skill levels. In fact, the motivational reward from this form of gameplay is achievement of greater skill.

Feedback loops are an essential component of Personalized Learning. (From an earlier post in this blog.)

Blended Learning, using a combination of online and in-person learning, is a prominent thread thread of research. That's based on studies (such as this one by the Open Learning Initiative) that show online and conventional teaching to be roughly equal but blended learning to be superior to both. An open question is which model to use when blending? My colleague, Scott Benson posts a periodic list of blended learning resources.

The Puzzle of Motivation: Dan Pink explains the growing science of motivation without which, even the best instruction may fail.

Building Blocks
A number of organizations including the federal government, technology standards groups, associations and foundations have assembled building blocks to support innovative education technology. Some of these can improve time-to-delivery, some help interoperability between applications and some ensure that your application is based on tested learning theories:

The Personalized Learning Model is a framework that some of us at the Gates Foundation have used to talk about how key components in a learning system work together. It's very similar to frameworks used by others in the community.

The Common Core State Standards (CCSS) are a common set of learning objectives that have been adopted by 45 states and DC. For primary and secondary education this gives a framework to which learning resources and assessments can be aligned. Recently, the CCSS have been published in machine-readable form with canonical identifiers.

The Partnership for Assessment of Readiness for College and Careers (PARCC) and the Smarter Balanced Assessment Consortium (SBAC) are two multi-state consortia developing standardized assessments for the Common Core. As with the CCSS, these are expected to be a generational improvement on existing assessment practice.

Common Education Data Standards (CEDS) is a coordinated effort to define a Data Dictionary and Logical Data Model for the education industry. The standards span primary, secondary and postsecondary education. Version 3, presently in public comment, adds early childhood, workforce and adult learning to the mix. The Four Layer Framework for Data Standards helps understand where CEDS fits into the mix of other standards.

The Learning Resource Metadata Initiative is a metadata schema for identifying learning resources (text, video, virtual labs, assessments, etc.) and aligning them to education standards like the Common Core.

The Learning Registry is a system for sharing metadata about learning resources. It's synergistic with LRMI and other metadata formats.

MyData Button is a federal government initiative to allow students or their parents to download their student data so that it can be used by other systems.

The Schools Information Framework (SIF) is a standard protocol for exchanging educational data among applications and institutions. SIF recently announced plans to incorporate all of the CEDS standard into SIF v3.

The Postsecondary Electronic Standards Council (PESC) defines data models and protocols for exchanging data among postsecondary institutions. PESC standards cover admissions applications, test score reporting, student aid applications and reporting, digital transcripts and more.

IMS Global defines educational content standards (where SIF and PESC concentrate on student and institutional data). IMS standards like QTI and Common Cartridge define how to package assessment items and courseware for exchange between systems. My favorite IMS standard is Learning Tools Interoperability which is a protocol that allows rich, custom learning tools to be integrated into other learning environments.
Ed-Fi is a data model and set of tools to support teacher and student dashboards indicating student progress.

The Shared Learning Collaborative (SLC) "is an alliance of states, foundations, educators, content providers, developers and vendors who are passionate about using technology to improve education." It's an ambitious multistate project that leverages many of the technologies listed above into a coherent whole. Vendor outreach programs are at dev.slcedu.org

Jim Goodell has posted his recent presentation on how several of these initiatives – SLC, LRMI, Learning Registry and MyData Button – all fit together coherently.

Product and Service Categories

There are a handful of existing education technology product and service categories with new ones emerging. Here are key categories with some examples. Note that the examples I've listed just happen to be well-known systems. It's far from a comprehensive list and I don't necessarily endorse these products. In each category there are emerging products that may be more innovative than the ones I name.

Student Information Systems (SIS) maintain student records including admissions, current enrollment and historical achievement. For K-12 attendance information is also maintained. Newer systems include a gradebook – at least for current enrollments. Examples include PowerSchool (K-12), Ellucian (Postsecondary), PeopleSoft Campus Solutions (Postsecondary) and Kuali (Open Source).

Learning Management Systems (LMS) manage class interactions such as syllabus, assignments, learning materials, quizzes, forums, gradebook and so forth. While LMSs are capable of delivering a rich online learning experience, most deployments are supplementary to conventional classroom learning and only a fraction of their capabilities are used. Well-known examples include BlackBoard, Desire2Learn, Moodle, eCollege, Sakai, BrainHoney and Canvas but there are numerous others.

Instructional Improvement Systems are an emerging concept. Like an LMS, an IIS manages student learning. However, an IIS uses accumulated student data as well as effectiveness data about learning resources to customize the learning experience to individual student needs. To support continuous improvement, the IIS should place equal emphasis on data collection and data use. Most action in the IIS space is being driven by state-level RFPs often with Race to the Top funding.

Learning Object Repositories (LOR) are collections of learning resources. They come in many forms. Some incorporate a Content Management System (CMS) to support authoring and curation of learning resources. Some, like CK12 or Khan Academy, are composed of internally authored components. Others, like Curriki or TheGateway collect user-generated content. Commercial vendors like Learning.com and NetTrekker build custom-curated repositories for schools, districts and states. Khan Academy augments their LOR with assessments and learning management. An Open Educational Resource (OER) Repository is a LOR that exclusively includes open-licensed resources. Prominent OER Repositories include Connexions, Hippocampus, Curriki, Merlot, OERCommons, CK12, TheGateway and Khan Academy. Commercial efforts include Learning.com, NetTrekker, TeachersPayTeachers and xpLOR. CoreSpring is an emerging OER repository for assessment items.

Public Education Datasets are available from the National Center for Education Statistics and other federal and state education agencies. The Digest of Education Statistics is a compilation of many government and privately-sourced datasets. Other public datasets include EdFacts and IPEDS. There some interesting opportunities to consuming existing public data and analyzing it in new ways.

There's much more that could be added but I think I've reached the point of diminishing returns. Please use the comments to point at other important theories, building blocks or initiatives.

Tips For Using the Common Core XML

2012-10-12T14:00:00.004-07:00

The Common Core State Standards (CCSS) have been out for a couple of years now and adoption efforts are progressing. To facilitate use of the CCSS in learning applications and with metadata frameworks like LRMI, they have recently posted canonical identifiers and machine-readable XML for the standards. I wrote about that in a recent post.

Today I'm going into the nuts and bolts of how a developer can make use of the XML. There are some useful features that aren't obvious at first glance. For background, I recommend that you read the announcement memo that accompanied the release of the XML on the corestandards.org website.

Canonical Identifiers

The canonical identifiers for the common core state standards are available in .csv form here: http://corestandards.org/assets/E0607_ccss_identifiers.csv. The first column lists the URLs that were formerly on the Corestandards.org website. They are included to support conversions for legacy applications.

Notable is there is an exact 1:1 mapping between the three ID types and there are 1844 IDs in the table. So if two applications are using different forms of IDs (e.g. one uses GUIDs and another uses URIs) the translation is a deterministic table lookup. A closer examination will show that there's a simple algorithmic conversion between the "dot notation" identifiers and the corresponding URIs. I've written functions in c# to do the translation and posted them here. It should be easy to port them to Java or any other language.

Hierarchy

The standards for Mathematics and ELA/Literacy follow different hierarchies that are suited to the way the standards are written and are intended to be used. The Dot Notation and URL forms of the identifiers can be parsed into the corresponding hierarchies as shown in the following examples.

Math Example

Dot Notation: CCSS.Math.Content.HSA-SSE.A.1b
URL: http://corestandards.org/Math/Content/HSA/SSE/A/1/b

Initiative	CCSS	(Common Core State Standards)
Framework	Math
Set	Content	(Options are 'Content' and 'Practice')
Grade	HSA	(High School Algebra)
Domain	SSE	(Seeing Structure in Expressions)
Cluster	A
Standard	1
Component	b

You can reference the math standards at the Component, Standard and Cluster levels. Thus, the following are all valid CCSS URI Identifiers:

If you add an ".xml" suffix to the URL then you get the computer-readable XML version of each:

The XML at the cluster and standard levels includes all child items. So the cluster includes all standards in that cluster and all components within that standard.

While the canonical IDs don't include Domain or Grade levels, you can retrieve all standards within a domain or grade by hacking the URL as follows:

And, you can retrieve all of the math standards in one XML document with this URL:

http://corestandards.org/Math.xml

ELA/Literacy Example

Dot Notation: CCSS.ELA-Literacy.W.9-10.3d
URL: http://www.corestandards.org/ELA-Literacy/W/9-10/3/d

Initiative	CCSS	(Common Core State Standards)
Framework	ELA-Literacy
Set		(Optional, not used in this example)
Strand+Domain	W	(Writing)
Grade	9-10
Standard	3
Component	d

You can reference the literacy standards at the Component, Standard and Grade levels. Thus, the following are all valid CCSS URI Identifiers:

If you add an ".xml" suffix to the URL then you get the computer-readable XML version of each:

And, you can retrieve all of the ELA/Literacy standards in one XML document with this URL:

http://www.corestandards.org/ELA-Literacy.xml

CEDS and the Four-Layer Framework for Data Standards - Updated

2012-10-04T10:21:00.001-07:00

About a year ago I posted a Four-Layer Framework for Data Standards. It was developed as Common Education Data Standards (CEDS) working groups were discussing the space in which CEDS operates and what makes its contribution unique. Today I'm updating the framework document – adding some clarity but mostly reconciling terminology with that used by CEDS.

In the June CEDS stakeholders' meeting the group emphasized that CEDS works strictly at layers 1 and 2 (Data Dictionary and Logical Data Model) leaving serialization and protocol to other standards organizations. This leads a unique approach (at least unique to the education standards space) in which the focus is on alignment instead of compliance.

To support this strategy, CEDS has posted the Align and Connect tools. The Align tool allows State Education Agencies, software vendors and other organizations to post their data models and show how their elements align to CEDS. Organizations can choose to make their data models public; in which case Align can be used to report the degree of alignment between two data models. The new Connect tool addresses the sharing of metric definitions like graduation rate, student financial aid repayment or college-going rate. Metrics like these are not in the data model, they are derived from that data. And different organizations may combine the data in different ways. Connect supports the sharing and eventual standardization of these metric definitions.

Another question I've gotten is how the four-layer framework overlaps with the OSI 7-layer model. Layers 1-3 (Data Dictionary, Logical Data Model and Serialization) in the four layer model map to the Application layer (layer 7) which is at the top of the OSI model. All other layers in OSI are combined into the Protocol layer in the four-layer model.

The latest four-layer document is here. It's released into the public domain under a CC0 disclaimer.

Schema.org, LRMI and the Learning Registry

2012-09-29T12:57:00.000-07:00

Try this:

1. Browse to google.com
2. Search for "potato salad"
3. Experiment with the recipe search tool that appears on the left.

Most people are aware of Google Shopping – Google set up a way for merchants to list the details of things they have for sale. More recently, Google published a way to mark up recipes so that the search engine can tell what's the title, what are the ingredients, what's a photo of the recipe and so forth. We call this, "metadata" or data about the data.

Get ready for a lot more of this kind of thing. The developers of, Bing, Google, Yahoo! and Yandex have cooperated on a common metadata vocabulary at Schema.org. It's a perfect example of coopetition – the search engines are cooperating how metadata should be embedded in web pages. That way webmasters don't have to code four different kinds of metadata. Meanwhile, the search providers will compete on what they do with that metadata.

The results of this are already emerging, try searching for a movie title, for example, or for a type of restaurant. Any of the major search engines will give you a nicely structured result.

We wanted to do the same for learning resources – videos, exercises, simulations, learning games and so forth. Wouldn't it be great if a teacher or student could search for "fractions" and get a search tool that allows the results to be filtered by age, grade level, subject or learning objective? Conveniently, the Schema.org folks had expressed their interest in new submissions, so long as they represent an industry consortium. So, a bunch of organizations got together and launched LRMI. You can learn much more about the co-sponsors, advisory groups and the specification itself on LRMI website.

Schema.org embeds the metadata right in the webpage (using HTML microdata). That makes sense for search engines, but it means that only the publisher of the webpage can post metadata about it. Yes, there's such a thing as third-party microdata but the search engines don't pay attention to it. Plus there are other kinds of data that need to be shared between learning solutions. Conveniently, there's a complementary alternative.

The Learning Registry is a peer network of LR servers that exchange metadata similar to the way email providers exchange mail messages. In the LR architecture, metadata consists of assertions. Here's are some example assertions rendered into plain English:

"CCSS.Math.Content.K.CC.A.1" is a "Learning Objective" defined as "Count to 100 by ones and by tens."
"http://www.youtube.com/watch?v=rG_uUws1oC0" is a "Video" that "Teaches" learning objective "CCSS.Math.Content.K.CC.A.1" and is targeted at "ages 2-4".

The first of these is one of the common-core standards. That data is now available in XML form on the web. The second of these is an LRMI statement. Of course, both are plain-English renditions of what could be machine-formatted.

LR assertions also include their provenance, that is the name of the organization making the assertion, when the assertion was made and a digital signature. This lets users of the LR have confidence in the origin of statements and filter for reliable sources.

You might have noticed that in describing the Learning Registry I used an LRMI example! That's because these are compatible technologies and the groups are coordinating with each other. This diagram helps show the relationship between the efforts:

You'll see that "Schema.org" appears twice in the diagram. That's because Schema.org defines both a vocabulary (a set of metadata tags) and a way of sharing that metadata. LRMI is an addition to the Schema.org vocabulary that enhances descriptions of educational content.

Schema.org and the Learning Registry offer two complementary ways of distributing metadata. They can even be bridged – there are experimental web crawlers that will extract HTML microdata from a page and inject it as assertions into the Learning Registry.

I could write a whole lot more about both of these efforts but far better to link to existing resources:

Common Identifiers for the Common Core

2012-08-29T18:06:00.000-07:00

In order to do personalized learning at scale, with a mix activities and assessments from a variety of sources, we need to agree upon a common set of learning objectives. The Gates Foundation and the Shared Learning Collaborative, have endorsed the Common Core State Standards.

Side note: As we are in the election season, there is a lot of rhetoric around national curriculum and federal mandates. The Common Core is only a set of commonly agreed upon learning objectives. It's not a curriculum (national or otherwise) and it was developed by a voluntary cooperation among states with the federal government staying clear.

As we were developing the LRMI project, we anticipated the need to be able to reference the Common Core as well as other learning objectives. Unfortunately, the Common Core didn't specify a standard set of references. At the time (approximately 12 months ago) there were at least five different and incompatible ways to reference the Common Core. So, we turned to the coordinators – the National Governor's Association (NGA) and Council of Chief State School Officers (CCSSO), and with the help of Student Achievement Partners they developed a consistent set of identifiers for the Common Core.

In the next few weeks, they will be updating the corestandards.org website so that the URL identifiers will link directly the the specified standards. Also, the standards will be available in machine-readable XML format to facilitate a variety of learning applications.

It's a great step and will make a big difference. But in the process we identified another issue. Frequently a particular standard in the common core will include more than one learning objective. Here's an example:

CCSS.Math.Content.6.NS.B.3
Fluently add, subtract, multiply, and divide multi-digit decimals using the standard algorithm for each operation.

There are at least four learning objectives in this standard. The developers of assessments (such as PARCC and SBAC) have a problem with that. A typical assessment item will only test one one of these skills. Without finer-grained identifiers, they can't show complete coverage of the standards. Similar problems exist for learning activities and student records.

A few weeks ago, the NGA, CCSSO, SBAC, PARCC and SETDA announced a collaborative project to address this issue. By the end of the year, they expect to publish an open set of learning objectives based on a fine-grained parsing of the common core. They will also define a standard data format for publishing standards like these. That will be based on the data models proposed for Common Education Data Standards 3.0.

Of course, this still isn't the whole picture. Not all states are adopting the Common Core. The Common Core only addresses Mathematics and English/Literacy, the 50 states still have standards for other subjects. Other countries have their own standard learning objectives.

There needs to be a way for developers of educational standards to publish and share those standards.

The Achievement Standards Network, operated by JES & Co., maintains an open database of all 50 states' existing standards, plus the Common Core, plus those published by the American Association for the Advancement of Science and many others. Under a new grant, they will be enhancing the database to accept the new standardized identifiers and they'll incorporate the learning objectives defined by the granularity project.

Concurrently, the Learning Registry is emerging as a distributed system for sharing Achievement Standards Data, Learning Objectives, Cross-References between standards and an index of learning activities that are aligned to standard objectives.

It may seem like chaos but this is more like a dance. And very shortly we should have a coherent foundation for developers of learning tools and instructional systems.

There are a lot of links up there. Here are repeat links to the three most important announcements:

ACT Scores: Most HS Graduates Aren't Prepared for College

2012-08-23T14:23:00.000-07:00

The ACT just released it's annual report on "The Condition of College & Career Readiness." Curious to me is how different news outlets spin the results:

Associated Press: ACT Scores Show Slight Progress in U.S. High School Seniors' College Readiness
Inside Higher Ed: ACT Scores Are Flat
Education Week: ACT Finds Most Students Still Not Ready for College
Forbes: High School Grads Even Less Prepared but At Least Doing Better in Math and Science

The scores are indeed flat and have been for five years. The AP achieves a positive spin by noting that the number of students taking the test has increased by 17% in those five years. If you assume that those who take the test are the top students, then the 17% addition represents the lowest performers on the exam and things have improved. However, the 17% addition could also indicate that more students are taking both the ACT and SAT rather than selecting one or the other.

Of primary concern to us at the Gates Foundation is the low rate of college readiness. Of those who take the exam (a subset of all high school students) only 25% are prepared for college in all four subject areas (English, Reading, Mathematics and Science). The goals of our U.S. College Ready team are to elevate the standard of high school graduation to mean college ready and to increase the graduation rate beyond 80%. The ACT report reminds us just how far away we are from that goal.

Is that a worthy goal? In an earlier post I noted that society is turning to education as the solution to poverty. I offer two additional facts to support this argument:

In the State of Washington, we import 43% of our bachelors' degree employees and 55% of graduate degree employees. Many of these people are immigrants because our school systems aren't preparing students for the high-tech jobs that are available.
In December 2010, U.S. unemployment of those with a Bachelor's degree was approximately 5.1% while unemployment of those with just a high school diploma was nearly double at 9.8%. Those without a degree of any sort endured an unemployment rate of over 15%.

(Edited 2012-08-23 to include the Forbes headline)