Thompson: How The NYT Got The IMPACT Evaluation Wrong

The New York Times' David Leonhardt may lack background information regarding education, but he is capable of understanding the recent  National Bureau of Economic Research paper, James Wyckoff’s and Thomas Dee's Incentives, Selection, and Teacher Performance.

However, given the inaccuracies in his A New Look at Teacher Evaluations and Learning, I wonder if Leonhardt read the study or if he just skimmed it.  

Leonhardt's sources for his misrepresentations are the commentator Nick Kristof and himself,with both basing their assertions on their misreadings of The Longterm Impacts of Teachers, by Raj Chetty, John Friedman, and Jonah Rockoff. Perhaps the Times should hold editorialists more accountable for carefully studying research even in the backwater field of education.   

Wyckoff's and Dee's NBER paper reports the effect of the Washington D.C. teacher evaluation system, IMPACT, on some aspects of teacher performance. It ignores the question of whether student performance was increased. It shows that IMPACT had an impact on some adult behaviors. It offers no evidence, however, that it was positive. 

The only valid conclusions that can be produced by the study’s methodology were reported by the Washington Post’s Emma Brown and Politico's Stephanie Simon.  Brown's Study: D.C.'s Teacher Evaluation System Affects Workforce explains, “Rewards and punishments embedded in the District’s controversial teacher evaluation program have shaped the school system’s workforce, affecting both retention and performance,” But the report is “silent about whether the incentives have translated into improved student achievement.”

Simon's Radical Washington D.C. Teacher-Evaluation Plan Worked, Study Says also recounted D.C.'s disappointing results in terms of student performance.  She quoted Dee as saying “This is a proof of concept.” (The concept is that carrots and sticks can have an effect.)  

Such a conclusion may mean something to theorists and commentators, but it says nothing about IMPACT's real-world impact. To paraphrase Mark Twain, a cat who sits on a hot stove won't do it again, but that does not mean that he has learned to cook.-JT(@drjohnthompson) Image via.

Comments

[Carol]

Thank you Alex for this accurate summary of the report. The grand conclusion was there was a significant increase in the percentage of minimally effective teachers (by rating) in the second year of the program who left.
We do not know if they were tenured or novices. Yet, on CNN, Michelle Rhee was claiming that IMPACT was a stunning success. Too many reporters these days only read the press releases.