Carnegie Learning Company Blog

How're we doing?

By Dr. Steve Ritter
Co-founder & Chief Product Architect, Carnegie Learning

The latest news about US math performance is out, and it's not good. In the most recent international comparison, US students scored 25th out of 30 countries.

But didn't we just read that US students "generally equal or outperform students in other parts of the world"? A related study reported that individual states performed better than most countries.

Last year, weren't we just about average internationally?

And aren't math scores improving?

And aren't reports of poor performance exaggerated?

Well, the story is more complex than the headlines or the quick summaries. The truth is that there isn't any single answer. How we're doing depends on:

Who "we" are
Who we're being compared to
What we're being compared on
What we consider to be acceptable (or good) performance

Most of the time, when you hear about US math performance, they're either talking about PISA, an international comparison, TIMSS, a different international comparison or NAEP, a US-only study that compares performance over time, nicknamed "The Nation's Report Card." Each of these tests uses different US students, uses different comparison groups and measures different aspects of math performance. In addition, the headlines and commentary put a more or less optimistic spin on things.

PISA

PISA (the source of the most recent headlines) compared 15-year-olds from 30 different countries (in math; they also looked at science and reading). We ranked 25th. I guess the good news is that we ranked above Italy, Greece, Turkey and Mexico (also Portugal, but not significantly). The PISA countries tend to be economically developed nations, so the competition's tough. Still "bottom of the developed nations" isn't exactly a slogan for America's economic competitiveness. We didn't do any better on this PISA exam than on previous ones.

PISA purports to test "mathematical literacy." Some say that isn't really math. Whatever. Check out the questions for yourself. That might not be pure math, but I sure wish our 15-year-olds could estimate the area of Antarctica.

TIMSS

TIMSS stands for Third International Math and Science Study. They liked the acronym so much that they decided not to have a Fourth IMSS. Instead, there's a TIMSS 1995, 1999, 2003, 2007, etc. So, I guess TIMSS doesn't really stand for anything. It's just TIMSS.

TIMSS generally tests students in grade 4, grade 8 and at the end of schooling (grade 12 in the US), though the end-of-schooling test always gets lower participation, and they didn't test it in 2003 or 2007. The countries that participate generally vary from year to year and from grade level to grade level, but there tends to be a wider economic range in the TIMSS countries. Macedonia and Botswana participated in TIMSS, not PISA. On the other hand, Singapore was the best-scoring TIMSS country, but it wasn't in the PISA study.

In TIMSS 2003, the US ranked 15th of 45 countries at the 8th grade level and 12th of 25 at the 4th grade level. This is better than US performance in the 1995 and 1999 variants of the test, and the US beat the international average on the test. So, if you look at TIMSS, we're above average but not in the top tier of countries.

The TIMSS exam questions tend to be good questions that ask students to demonstrate understanding of basic math skills. They don't require the kind of application and integration that makes the PISA questions "math literacy" rather than math. Decide for yourself if that's good or bad.

NAEP

The National Assessment of Educational Progress is just about the US. It is a test given by the Federal government to a sample of US students every few years. Because the test has been given since the 1970s, it gives us the ability to see how well today's students are doing compared to students in the past.

NAEP questions tend to be basic math questions. The general trend is that students are doing better now than they were in the 1970s, especially for younger students, but the increases have not been very dramatic. Absolute performance levels are nothing to be proud of. According to NAEP, the average 17-year is "developing an understanding of number systems." That's not exactly preparation for rocket science. And 17-year-olds have not been improving. Educational policy wonks may argue about whether NAEP scores are improving more under NCLB than they were previously, but these arguments are really irrelevant. The argument is over whether NCLB has increased improvement perceptibly. No one argues that there is evidence of dramatic improvement.

Frankenstudies

Lots of studies try to dig in to NAEP, PISA or TIMSS, often searching for some bits of good news. One example is the study concluding that US students "equal or outperform" those from other countries. This study tried to compare performance for countries on TIMSS to individual states. Since TIMSS wasn't designed to measure performance of US states (just the country as a whole), they used NAEP scores to predict what the states' TIMSS scores would have been, had they taken the test.

There's a lot of clever statistics in there, but the studies end up saying nothing new. In TIMSS, the US did better than most countries, but we're still not in the big leagues. Massachusetts (our top-scoring state) doesn't do as well as Singapore. Despite the optimistic headlines, the studies admit that, on average, US students are not proficient in math.

The Bottom Line

Measuring mathematics performance for a country is complex, and comparing it across countries is even more complex. It is easy to be confused by all the different studies, re-analyses and interpretations.

However, the overall story is pretty consistent. US students are not equal to our expectations in math and science. They aren't meeting NAEP's proficiency standards and they aren't leading the world. At lower grade levels, they're probably above average. At higher grades, they're average, at best.

Younger students in the US have improved, both relative to the past (NAEP) and relative to other countries (TIMSS). Older students have not improved (NAEP and PISA), at least not enough to make any substantive difference.

The Asian countries have shown that dramatic improvements at a national level are possible. We can do better. If we want to lead the world, we need to do better.

PS

No study is perfect, but you shouldn't dismiss any of these results because of fundamental flaws. I've heard people complain that, in the US, everyone goes to school but, in those other countries, only the elites are being tested. Or that we might not do well on average but our best students outscore their best students. Or that we don't score well on tests but we're more creative. The people doing these studies are very, very smart. They've heard all of these objections, and they've tried to address them. Specialists can argue about how well particular issues are addressed, but it is very unlikely that there's some big, gaping hole that invalidates the whole thing.

On the other hand, the PISA guys did misprint the US reading test and had to throw the results out...

High School Student Responds to Post-Gazette Article

By Mary Murrin
VP Marketing, Carnegie Learning

The letter to the editor reprinted below is a response to an article recently published about Dr. Steve Ritter (our Chief Scientist) in the Pittsburgh Post- Gazette. It’s encouraging. Kids often care more about learning than we might think, and are looking for more and better experiences that engage and inspire.

------------------------------------------------------------------------------

Math comfort

The Sept. 3 article "The Thinkers: He's Taking Unknown Out of Teaching Algebra" struck a personal note for me. As a student who has occasionally had trouble in math classes because I was afraid to ask a question, I think the Cognitive Tutor software seems like the answer to many struggling students' prayers. It not only lets students learn at their own pace, and hone in on troubling areas, it also allows students to feel more comfortable to ask questions.

Students, like myself, often feel as though their questions are foolish, or the teacher does not want to spend time answering them. With this program, however, all students should feel comfortable asking questions, working through their problems and taking the time they need.

Though I am in my senior year of high school and will never get the opportunity to use the program, I hope that it will be implemented into math curriculums nationwide so students will be able to face math with confidence and succeed.

COURTNEY STRAUSS
McCandless

Courtesy of the Pittsburgh Post-Gazette: http://www.post-gazette.com/pg/07254/816368-35.stm

Follow-up to our post on Lobbying

By Mary Murrin
VP Marketing, Carnegie Learning

We were recently pointed to a blog entry with a favorable response to our message Lobbying for Rigorous Definitions of "research-based":

“Your editor has followed Carnegie Learning since its birth in 1998... Nevertheless, their blog was unknown until today.

Any educator considering the purchase of Carnegie's services should be reassured by their postings. By “reassured,” your editor means something more than what program managers in the Pentagon call “the warm and fuzzies.” Rather, the firm expresses a set of values that are most likely more than propaganda. In stating these values, Carnegie is asking prospective clients to hold the firm to a much higher standard than the market really demands today, and one that most providers can't meet.

Full posting: http://www.edbizbuzz.com/blog/_archives/2007/5/23/2968721.html

Technorati Tags: Math, high school, Pittsburgh, school reform, NCLB, US Department of Education, Institute of Education Services, lobby, research-based, curriculum, What Works Clearninghouse

Compelling Article in Response to DOE's Study on Educational Software

By Dr. Steve Ritter
Co-founder & Chief Product Architect, Carnegie Learning

Henry Kelly, president of the Federation of American Scientists, in Washington, makes one of the most lucid and compelling statements in response to the U.S. Department of Ed’s recent report to Congress that I’ve read.

"The U.S. Department of Education’s recent report to Congress on the effectiveness of reading and mathematics software products sounded a wake-up call to anyone pondering technology in education. ("Major Study on Software Stirs Debate," April 11, 2007.) Its authors conclude that math and reading software produce no better test results than conventional teaching methods.

How can a technology that is transforming the way we acquire information throughout the economy—revolutionizing businesses from games to banking—fail to benefit education? How can technology revolutionizing training in the U.S. Department of Defense fizzle in elementary schools?

The Education Department report is not evidence that technology cannot be a powerful learning tool. It proves only that results on standardized tests are not significantly improved by systems found in a sample set of schools. Moreover, the study focused on whether the technology was better than traditional teaching methods, failing to consider new technology as a productivity tool. Yet its authors admit that the study produced results no worse than for traditional teaching, and that 86 percent to 92 percent of the teachers in the program found the software sufficiently useful to keep using it. Clearly, these teachers perceive a value not registering on the tests."

Read the Entire Article Here: http://www.edweek.org/ew/articles/2007/04/11/32software.h26.html

Technorati Tags: math, math research, high school, Pittsburgh, school reform, Department of Education, education technology, Rand, differentiated learning

Response to the Latest Dept. of Education Study About Technology in Education

By Dennis Ciccone
CEO, Carnegie Learning

Last week, the U.S. Department of Education released the results of a study conducted throughout the 2004-2005 school year evaluating the effectiveness of the use of technology in education. The study compares the performance of students using technology to that of students using textbooks only. The study looked at technology implementations in several subjects and from multiple publishers, including Carnegie Learning. The results were given for the effect of technology in general, and did not assess individual curricula. In brief, the results show that technology has no significant impact on student performance based on scores from pre- and post- tests.

I want to share our thoughts on the findings. First, Carnegie Learning believes in the value of participating in studies like this one. Evaluation can present valuable opportunities to gather important data for our own analysis, to see our solutions through the eyes of an objective party and, most importantly, to apply new knowledge to product development and Professional Development services that improves our solutions and creates greater value for students, teachers, and administrators. Over the past eight years, we have participated in numerous studies, most of which have shown significant improvements in student performance. One, in particular, resulted in recognition by the Department of Education’s What Works Clearinghouse. In fact, we announced this week that we are embarking on a new five-year study by RAND Corporation which is funded by $6 million from the U.S. Department of Education and will specifically study the effects of Carnegie Learning’s Algebra I curriculum.

While we will evaluate the data from the Department of Education’s Effectiveness of Education Technology Intervention study more closely over the next few weeks, the initial results corroborate what we have learned in districts across the country:

1) A curriculum’s success is directly related to the level of buy-in from the administration and staff. Carnegie Learning showed stronger results in schools where the staff was enthusiastic, seeking a change, and had a part in selecting the solution.

2) Technology-based curricula have a greater chance of success if a strong technology infrastructure is in place. Hardware shortages and network performance issues frustrate students and teachers. In these environments students spend less time using the software, they lose continuity, and the value of technology-based learning is diluted.

3) Teachers want and need tools to help them help their students. Ongoing Professional Development is critical to a successful implementation of technology-based learning. As a result of studying our own implementations, in the past two years since this study began Carnegie Learning has expanded our services to teachers beyond initial training to offerings that share best practices year-round.

4) Learning is dynamic; it takes time, and has many variables. Sometimes you can see good results in a year, often you cannot. Two of the three districts that used our curricula in this study have made additional purchases from Carnegie Learning within the past year. Although, the advantages of the Algebra I curriculum did not show up in this study, these districts had enough confidence in the product to work with it in the future.

Finally, our own research shows that differentiated learning is critical to success in math, a subject that requires abstract thinking and individualized problem solving processes. Given the poor ratios of teachers to students in the classroom, we believe that technologies like ours – ones that learn students’ strengths and weaknesses -- offer exceptional teaching and learning tools that, more often than not, increase a student’s chance for success.

I hope that you will take a minute to read this news story from The Everett (WA) Herald in which a Carnegie Learning customer responds to the study in the context of their district’s Carnegie Learning implementation.

I look forward to answering any questions. Please feel free to contact me at 412.690.2442 or info@carnegielearning.com.

Technorati Tags: math, math research, high school, Pittsburgh, school reform, Department of Education, education technology, Rand, differentiated learning

Demand for high-stakes test preparation software

By David Hart,
COO, Carnegie Learning, Inc.

Carnegie Learning just announced Math Prep, a new product family designed to help students pass high-stakes high school exit exams.

The number of high-stakes tests being administered in our schools is skyrocketing across the country. A little over a year ago, one of our field consultants in Texas asked for a TAKS prep course in math. Our other salespeople concurred there was demand for effective test preparation courses but the range of existing solutions very broad and no single player was dominating the field. We learned most schools used "homegrown" methods to teach remedial math, and most struggled to combine assessment and remediation into a single integrated program.

One of the toughest things about the educational publishing business is the proliferation of local educational standards, all subtly different the one from the other. In order to simplify my own life I would welcome national standards in testing and grade progress. Establishing and maintaining correlations between your products and your customers' standards is a time consuming grind. It also divides the market into small pieces and creates confusion in product naming and management.

But, we need to focus on local standards to meet our customers needs. So, we have launched our Math Prep products as state editions, with CA, FL, TX, WA, and a national edition. The courses are designed to prepare students for high-stakes exit exams, providing up to 130 hours of instruction in subject areas covered on the exam. As with all Cognitive Tutor courses, Math Prep adopts its instruction to each student's needs, closely watching the student's learning and providing more instruction as needed.

More states will be coming from Carnegie Learning this year. Additionally, Math Prep is our first product to integrate a pre-test with the instruction, which better focuses remediation on areas where the student shows weakness. The pre-test data lets us better define the cognitive model of the student's skills at a very granular level. This is an exciting new development and we will be looking at ways to integrate this feature across other products.

Technorati Tags: Math, Test Preparation, CAHSEE, TAKS, FCAT, Exit Exams, Software, high school

Lobbying for rigorous definitions of "research-based"

By Mary Murrin
VP Marketing, Carnegie Learning

Last week, three executives of Carnegie Learning were on Capitol Hill to meet with staff for eight Congressional Offices integral to the House Education and Labor Committee; Senate Caucus on Science, Technology, Engineering, and Math (STEM) Education; and the Senate Health Education, Labor and Pensions (HELP). Among our agenda items was to recommend that the U. S. Department of Education continue to set the bar high when evaluating educational solutions and that criteria used to determine the definition of research-based curricula remain rigorous.

We are concerned by recent discussions among some education publishers that the Department of Ed loosen the definition of scientifically based research. The existence of a definitive source for qualifying and validating research-based curricula is critical to practitioners in the field, and we believe that diluting sound scientifically based research practice encourages school districts to spend money carelessly on unproven curricula. The truth is that while many curricula vendors claim to offer "research-based" programs, careless claims and false credentials abound. Under pressure, schools have neither the time nor the hard data to understand which curricula are truly proven in practical applications. Despite declining test scores, vendors would rather sell – and in many cases, teachers would rather teach – tired old solutions, instead of investing in better approaches based on new research and proven, quantifiable results.

The job of defining the parameters of scientifically based research is best left to the experts in the form of an independent organization called the What Works Clearinghouse (WWC). The U.S. Department of Education's Institute of Education Sciences (IES) established the WWC in 2002 to provide educators, policymakers, researchers, and the public with a trusted, unbiased source of scientific evidence of what works in education.

Having created NCLB, the federal government is obligated to provide a "best practices" clearinghouse to educators, and the WWC-approved solutions are the best way for schools to meet NCLB standards. The alternative is for schools to blindly pump money at the problem of educating the next generation of working citizens. Our children deserve the best possible education, not only because we wish for them rich, satisfying futures but because, as a nation, we hope to compete in global economy in the decades ahead. There is much at stake in determining what really works in education.

Technorati Tags: Math, high school, Pittsburgh, school reform, NCLB, US Department of Education, Institute of Education Services, lobby, research-based, curriculum, What Works Clearninghouse

High School Reform in Pittsburgh, PA

By Dennis Ciccone
CEO, Carnegie Learning

I was very pleased to see the announcement that Pittsburgh Public Schools Superintendent Mark Roosevelt is making high school improvement a top priority in 2007. The Pittsburgh school district is joining the ranks of urban districts nationwide that understand and are responding to the urgency and liability of falling graduation rates in America.

Like many societal issues, the mission of preparing every child to graduate is now receiving the attention it has long deserved because it has moved beyond a debate of social responsibility to one of economic preservation. At the Federal level, the President’s American Competitive Initiative includes proposals to improve math and science education in our schools in order to advance our nation's economic competitiveness. The National Governors Association is driving the Innovation America program at the State level with a similar mission, and Bill and Melinda Gates and their foundation have named high school transformation a key area of focus in their effort to reduce inequities in the United States.

Roosevelt’s commitment is, congruously, set in Pittsburgh, a city of companies and organizations that are emerging front and center in high school transformation initiatives nationwide. I am proud to be leading Carnegie Learning, Inc., a fast-growing mathematics software and textbook publishing company evolved from 20 years of cognitive science research at CMU and backed by success in diverse classroom settings across the country. Through the process put in place by the Bill & Melinda Gates Foundation, Carnegie Learning was chosen as one of two math curricula providers in the Chicago Public Schools where we are focused specifically on helping at-risk middle and high school students succeed in math and advance to graduation. Recently, Dr. Steve Ritter, a company founder and our chief product architect was invited to discuss our approach with members of the Board and staff of the Gates Foundation. We are also implemented in similar high school reform programs in Los Angeles Unified School District and Miami-Dade.

We are not alone in our expertise. The RAND Corporation, a Santa Monica, CA-based research and public policy institution, has a Pittsburgh office addressing K-12 assessment and accountability and school reform, among other issues. The Science of Learning Center at Carnegie Mellon University and the University of Pittsburgh’s Learning, Research & Development Center are leaders in theories on teaching and learning. The Heinz Endowments is committed to increasing and improving our region’s pool of highly skilled teachers who will create successful learning environments.

High school reform is a point of national concern and Pittsburgh is playing a major role in driving change, assessing results, and improving the futures of our young people. We hope that Mark Roosevelt, his staff, and the students of the Pittsburgh Public Schools will benefit from the expertise we have cultivated right here in town.

Technorati Tags: Math, high school, Pittsburgh, school reform, Bill and Melinda Gates Foundation, American Competitive Initiative