About this blog Subscribe to this blog

Bruno: Standardized Tests & Classroom Teacher Bias

I really liked Kathleen Porter-Magee's and Jennifer Borgioli'spost on "the four biggest myths of the anti-testing backlash", especially the part about "teachers' instincts" so I'll permit myself to briefly take issue with fellow This Week In Education contributor John Thompson's characterization of their argument.

800px-Blind_spot_(watercraft-2).svg

Admittedly, Kathleen and Jennifer may have sacrificed a bit of clarity in the pursuit of brevity, but unlike John I do not read them as claiming that standardized assessment results are "more valuable" than teacher-generated assessments. The key point - which I have not seen refuted - is that teacher assessments of students are often biased in ways that needlessly disadvantage students on the basis of their race (or gender, etc.) and standardized assessment data may be useful in mitigating those biases.

It's fair to say, as John does, that the way standardized assessment data are collected and used may reduce or eliminate their usefulness in practice. That, however, wasn't the "myth" to which Kathleen and Jennifer were referring. Instead, they were referring to the myth that teachers don't suffer from big,  important blind spots when it comes to assessing students. I doubt John labors under that misconception, but there are many reform critics who often speak as though they do.

Incidentally, this is one of the reasons I don't really mind my students taking their 8th grade state science test and actually look forward to receiving the results: I'm acutely aware of the limitations of my own judgment when it comes to my students and standardized tests are one important - albeit imperfect - way for me to fill in my blind spots. - PB (@MrPABruno) (image source)

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

I would be extremely shocked if my state test results didn't map roughly to my own assessments. Aren't they supposed to? Sure, there will be outliers--a kid who overperforms, a kid who had a bad day--but if I was giving As to a kid who routinely scored below basic and Cs to a kid who scored proficient, then there's something wrong with what I'm doing in the classroom.

Paul,

I also look forward to studying the results of my History students' benchmarks. That does not compensate for the great harm that is done to my students by attaching stakes to the tests

Please address the second half of their title and argument. They are criticizing the "anti-testing backlash." I believe that the backlash against testing - which is reaching a "crescendo" - is the best thing that is happening in education today.

Do you disagree?

And, Education Realist, why would you be shocked? When I first started teaching twenty-something years ago, I taught with a bunch of others who had advanced degrees. Periodically, we would laugh at the incompetent test questions. In many cases, to answer the question you had to first ask how stupid the test-makers were. You'd have to speculate on why their committee decided on such silly answers and/or amateurish phrasing of the question before you could guess the answer.

Pedictably, the need for gallows humor has gotten worse. Any reading of Econ 101 and the laws of Supply and Demand would anticipate that the increased demand for tests would result in an increased supply of lower quality tests. That's certainly my experience. Every week, my state-created, consultant-created benchmark has at least one question where the answer is flatout wrong, and others that are flawed.

So, Ed Realist and Paul, how do you handle it when the Answer Key is wrong?

@Ed - Those sorts of comparisons are certainly one of the reasons I want to look at the test results. The biggest thing I look for isn't usually individual student results, though, it's the relative size of the gaps between my students and other students and in different sub-areas of the test. So, for example, I spend more time on the periodic table now than I used to because my kids weren't doing as well on that portion of the test as on the others.

@John - I'm ambivalent about the testing backlash you describe in part because a lot of the reasonable anti-testing arguments get muddled together with the nonsense arguments like those Kathleen and Jennifer point out. So I think it's good that people are unhappy with curricular narrowing, but is the movement really "the best thing in education"? I don't know how to evaluate that claim.

I also don't know what you mean by "the Answer Key". Standardized test results are (or should be) just data points; I don't think of them as "answer keys", if that's what you mean.

Paul,

When we get benchmarks, we also get released End of Instruction test questions. They provide the answer key and the explanation of what the test-makers thought the answer was. A good percentage of those questions are incompetently written and a lower but steady percentage of the answers are flat wrong.

But, what about my other questions? Can you articulate a scenario where increased EOI testing demand didn't drive down EOI test quality?

Ok, you don't think the anti-testing backlash is the best thing? Wouldn't you agree that testing has produced the worst and most destructive unforced errors of the last generation? If you don't think that high-stakes testing is the worst thing, would you agree its the 3rd or 2nd worst? Wouldn't you agree that it is wrong to stick with policies that have done so such harm for so little gain?

@John - Lots of EOI test are of middling quality or worse. Whether they've *declined* in quality is a separate question I don't know how to answer.

As for ranking atrocities, even limiting ourselves to education I'm not sure where high-stakes testing fits on a list. It's certainly had numerous negative effects, but it's also had some positive effects (e.g., making achievement gaps more salient) and there's not a lot of evidence it's wrecking overall achievement by much (if at all).

And I might think there's more competition for unforced errors than you do. I'd rank a lot of progressive educational theory & pedagogy up there, along with some treatments/classifications for students under SPED and EL programs.

In other words, I probably think the harms of high stakes testing are less - and the gains more substantial - than you do, so trying to figure out where it fits among the "most destructive unforced errors of the last generation" strikes me as a bit hyperbolic.

I think it could be hyperbolic for some, while it is dead one for others. How much harm are you willing to impose on my students for the meager gains, at the cost of tens of billionaies of dollars to others.

Progressive methods might have failed sometimes - or often- but they don't threaten the fundamental principles of public education.

And, what if Diane Ravitch is right and the result is privatization?

Again, I'm just don't assess the harms and gains as being on the same orders of magnitude as you do.

And I'm not sure what the "fundamental principles of public education" are, but if, for example, progressive educational principles often systematically favor already-privileged students over the less-fortunate (and I believe they often do), then that would seem to me to fit the bill. But, again, reasonable judgments about "fundamental principles" may differ; that's not a phrasing that has much meaning for me.

There are always "what ifs". I'm guessing privatization trends, to whatever extent they exist, have as much to do with unreformy factors - like cost growth - as they do with high stakes testing. (See also: health care reform.) But we could speculate forever.

We're also going to disagree on potential harm due to progressivism. I started teaching about 21 years ago. I've never seen a trueblue progressive teaching consistently in that manner. Have you?

Wouldn't it have been nice if we'd put tens of billions of dollars into creating environments where poor kids could be treated with the same respect we show to already-priveleged?

If we completely give up on that ideal, isn't that a major cost?

Consistently? No, because it works so poorly. But it's in a lot of math curricula, especially, and it infiltrates lots of science teaching in subtle but real ways. (And not-so-subtle ways...non-teacher adults typically want to see more progressive/constructivist/inquiry teaching when they come in to visit or observe science classes.)

Lots of things would be nice. But just because we don't have them doesn't mean they're a cost of high-stakes testing.

"Wouldn't you agree that testing has produced the worst and most destructive unforced errors of the last generation? "

No.

In fact, the ACT and the SAT are two of the most egalitarian tests ever created, so effective that they essentially sorted a generation.

As for school testing, absolutely not. The only real damage tests can do is lead to school closings, and I'm beginning to realize that these decisions are based far more on money than they are on test scores.

"I'd rank a lot of progressive educational theory & pedagogy up there"

I generally agree with this. However, it's worth remembering that progressive educational theory didn't cause the achievement gap, but rather became prevalent as a result of the achievement gap's refusal to close despite our expectations. At that point, we would have resolutely refused to believe that the achievement gap was genuine, so something moronic would have come along to try to mask the gap.

I don't know what you mean about terrible test questions. I spend a lot of time looking at tests of all sorts, and I have reviewed the questions in three subjects. I don't see a lot of stupid questions at all--particularly at the high school level.

Middle school science tests are sometimes of poor quality either because of bogus questions - until very recently one of the 8th grade science CST release questions had two correct answers - or - more often and with district tests - they cover the content very inconsistently or unpredictably. I don't know that it's a huge problem with the state tests, but it's a moderate inconvenience with district level tests. (Again, with middle school science.) I think science is just a lower priority so it doesn't get the sort of scrutiny you see in, say, math. (Oakland ended up cutting its science benchmarks altogether a couple of years ago.) The standards themselves might also be more difficult to interpret and therefore to make useful tests for (especially the 7th grade science standards).

A couple of points I believe I can address:

First, teacher bias against disadvantaged students. I prefer tests that I make myself because they assess the material I cover in class the way I cover it. I want to know if my students are learning my material, and if the students get the question right, they get it right. No bias there. Most teachers, including myself, do not even look at the name on the test at all while grading.

This worked fine for me for years. Then, a couple of years ago, school administration started giving the entire science department the tests we were to give. While there were (as John says) some wacky questions, my biggest problem was that the questions were very specific about details that were widely scattered across a broad range of topics. There was no way for me to cover all that material to the detail that seemed to be required, so I ended up just covering the questions themselves, which absolutely harmed learning. There was no central theme or concept to tie the material together, and this happened over and over.

Second point: is the backlash against testing the best thing happening in education? My answer is that it seems to be the best thing that is actually happening (and is a good thing) but it does not address the biggest problem I had in my classroom, which is lack of student effort and motivation. I absolutely blame the increased focus on "progressive education" because it took away most of the consequences and the tools I had as a teacher to enforce discipline. The increase in the amount and number of classroom disruptions parallels the increase in progressive education focus.

This is the biggest problem, and it is a problem that is getting worse. Where is the attention on this topic?

Cyberike,

I like you explanation that the backlash "is best thing that is actually happening." That is better than my explanation that testing is the worst thing we can actually control.

I also agree that progessivism's lack of interest in discipline remains its worst legacy.

I also agree that the lack of student motivation and effort is the biggest issue, and it is getting worst.

Testing is one contributer. There are many, many other contributers. That is an area where we can make things better through engaging instruction and other efforts. The overall phenomon, however, is more than an unforced error. Testing is.

Or to mix metaphors, testing has dug us into a hole and we could just stop digging.

The problems you cite are much more complicated.

John, thanks for your thoughtful reply. Here is what I am afraid of.

Educational policy makers cite the poor performance of our students as the rationale behind reforms. Eliminating the most onerous testing regimen will help, but because it does not address the real (and most serious) problem, student performance is likely to continue to get worse.

This result would strengthen the call for further "reform", with any attempt to slow down or mitigate these reforms met with the same immovability we see on the federal budget issue. The argument will be "you got what you wanted, it didn't work, now try it my way". To even more disastrous result.

Assessments are an important part of the education process.They are essential for measuring student's progress. I think both the methods of assessment has their pros and cons, a combination of both methods must be implemented.

The comments to this entry are closed.

Advertisement

Advertisement

Disclaimer: The opinions expressed in This Week In Education are strictly those of the author and do not reflect the opinions or endorsement of Scholastic, Inc.