About this blog Subscribe to this blog

Thompson: False Positives & Value-Added Evaluation

AlgorithmDaniel Goldhaber and Susanna Loeb's What Do We Know About the Tradeoffs Associated with Teacher Missclassification in High Stakes Personnel Decisions?, posted in the Carnegie Knowledge Network, argues that value-added evaluations could result in roughly 25% of teachers labeled as ineffective being wrongly placed in that category.  These mistakes are called false positives.

Then, they estimate that 25% of those who not are classified as being ineffective should have be in that category. These mistakes are false negatives.

Are false positives equally destructive?  Are there ways to work around the mistakes these systems are going to make?  Or should we be focused on much simpler, more concrete measures of teacher performance such as attendance, timeliness, and active participation in the classroom?

At times, Goldhaber and Loeb imply that false positives and false negatives are equally destructive.  Their otherwise excellent paper repeats the worn-out soundbite that such flawed measures are better than the nonexistent system we have now. Moreover, they cite a federal judge's conclusion that the social damage of false negatives positives could be worse than false positives.  

If we want to improve poor schools, accepting more unfair terminations to avoid false negatives would be disastrous.  It is unlikely that 25% of teachers in all types of schools would be wrongly evaluated.  Goldhaber and Loeb seem to realize that value-added has serious problems in terms of controlling for peer effects. Value-added can't determine whether instructional ineffectiveness is due to a teacher or due to the ineffectiveness of the school as a whole. 

So, the percentage of false positives would likely be greater in ineffective schools.  In inner city high schools, it is safe to assume, the false positive problem would be far greater. And, even though value-added advocates want to fire high school teachers with a statistical model that is virtually untested in secondary schools, common sense says that the false positive rate would go through the roof in the toughest classes in the toughest high schools.  

On the other hand, common sense points to a partial solution.

Let's first address the practical effect of solving the false negative problem.  The value of correctly identifying all teachers who are less effective than they seem would be determined by the answer to another question.  Could schools find qualified applicants to replace ineffective teachers?  In my experience, it is easy to identify bad teachers.  And, they are found disproportionately in the classes that are so challenging that years go by without a qualified applicant for them.  

It is hard enough to recruit and retain teachers in the toughest schools.  It is silly to believe that teaching in the inner city can become more attractive by unfairly punishing teachers in those schools. But, reducing false negatives by producing even more false positives would be even worse.

There is a partial solution hiding in plain sight, however.  When Chicago principals were first empowered to take action regarding teachers with no due process rights, they removed more teachers with poor attendance and poor value-added.  Rather than try to make value-added evaluations less unreliable, why not tackle the much easier challenge of identifying teachers who have too many unexcused absences?  Why not target teachers who do not show up to work on time, or sit at their desks and use their cell phones when they should be teaching? 

In other words, why not focus on what teachers do?  Rather than invest so much effort designing algorithms to be proxies for being effective or ineffective, why not evaluate teachers on whether they do their jobs or not?-JT(@drjohnthompson) Image via.   


Feed You can follow this conversation by subscribing to the comment feed for this post.

The comments to this entry are closed.

Disclaimer: The opinions expressed in This Week In Education are strictly those of the author and do not reflect the opinions or endorsement of Scholastic, Inc.