Bruno: Actually, Statisticians Are Cautiously Optimistic About VAM
It's always nice when experts come together to help to articulate and clarify whatever scientific consensus exists around an issue, so I was glad to see the American Statistical Association put out a report last week on the promise and peril of value-added modeling of educational effectiveness.
Interestingly, however, if you were to hear about this report only from the staunchest, most ideological opponents of VAM, you would think it says something else entirely. Valerie Strauss, for instance, claims the report "slammed" the use of VAM to evaluate teachers and Diane Ravitch seems to think it is a "damning indictment" of such policies.
The report itself is not nearly so hyperbolic.
For a useful summary check out Stephen Sawchuk, but the report itself is a mere seven accessible pages so I encourage you read it yourself.
The bottom line for the ASA is that they are optimistic about the use of "statistical methodology" to improve and evaluate educational interventions, but current value-added models have many limitations that make them difficult to interpret and apply, especially when evaluating individual teachers.
The report does, of course, emphasize that caution is needed when interpreting and using VAM data, but even here it is important to remember that all methods of evaluation have limitations. That reformers often do not understand those limitations is no excuse for pretending the alternatives are not limited in their own ways.
None of which is to say this report is a major vindication for reformers. Make no mistake: this report throws a great deal of cold water on the sorts of VAM proposals that prominent reformers are often associated with, which are often rooted in deep misunderstandings of statistics and labor markets and promoted with over-the-top, unrealistic rhetoric.
Nevertheless, it requires a considerable degree of cherry-picking and motivated reasoning to suggest that this report - which explicitly refuses to condemn any particular policy - is a blanket rejection of value-added modeling.
On the contrary: the advice in this report is entirely consistent with the frequent use of VAM in education, even in some cases as a component of individual teacher evaluation. Such accountability regimes may or may not be optimal, but they are by no means ruled out by the authors' conclusions.
In their over-eagerness to give reformers a black eye, reform critics are obscuring considerable nuance. That might be politically savvy, but it's probably not ideal for improving education policy. - PB (@MrPABruno) (image source)