About this blog Subscribe to this blog

THOMPSON: Out of the Mouths of Social Scientists

Science_Guy Unlike Gerald Bracey, I strongly support almost all of the Obama educational proposals, but Bracey gives Secretary Duncan a way out of his Race to the Top dilemma, writing "The President of the United States and his Secretary of Education are violating one of the most fundamental principles concerning test use: Tests should be used only for the purpose for which they were developed. If they are to be used for some other purpose, then careful attention must be paid to whether or not this purpose is appropriate. This position was developed jointly by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education."

Duncan should commit to science-based policies by issuing a regulation along the lines of "no district accepting federal funds may use a test data from a statistical model for the purposes of evaluation or tenure unless that model has been subjected to peer review consistent with established social science practice for use in conditions comparable to those encountered by the teacher under evaluation."

Free us from the improper use of test scores and growth models and educators could join in a golden age of data-informed decision-making, using test scores for diagnostic purposes, and building a culture of collaboration. Free us from invalid punitive measures, and we can use the miracles of the digital age to identify children before they are lost and to inform instruction. - John Thompson

Comments

TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00e54f8c25c988340115715663d7970c

Listed below are links to weblogs that reference THOMPSON: Out of the Mouths of Social Scientists:

Permalink

Permalink URL for this entry:
https://scholasticadministrator.typepad.com/thisweekineducation/2009/08/thompson-out-of-the-mouths-of-social-scientists.html

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

John,

I hear ya. But I’m not sure what you’re hoping for is possible. Case in point:

“Free us from the improper use of test scores and growth models and educators could join in a golden age of data-informed decision-making, using test scores for diagnostic purposes,…”

I don’t think improper use is the problem. I think the mere existence of testing and test scores is the problem.

I think even hard-nosed psychometricians agree that the kinds of tests we give kids in school don’t really measure very much. Often, there are few, if any, questions directed toward a given standard. And the margin of error when n=1 is horrendous. So I think we’ll always have a problem when we wonder “What did this test actually measure in regards to this kid?” Interestingly, a simple study of the famous TIMMS test showed that kids who filled out the egregiously long pre-test survey did better on the test than kids who did not. In fact, the rankings of countries could be predicted from the percentage of kids who filled out the entire survey. Malcolm Gladwell’s conclusion about this was that tests are more about stamina than anything else.

Now, on to our second problem. Let’s imagine that really good tests could be created. The data we get back is usually not finely grained enough to make important decisions. Let’s take, for example, the decision about whether to pass a kid along to a new grade. Any cutoff you pick will be arbitrary. And, with a large margin of error for individual test takers, you’re likely to make more than a few mistakes. Imagine how much more controversial this would be if the decision were about firing a teacher.

So we’ve got at least two problems with tests: we can’t be sure about what they measure; and what they measure, they don’t measure very accurately at the level of an individual test taker. This leads me to conclude that true data-driven decision-making is, and forever will be, a myth.

Except in extreme situations – like all students failing a math test – it’s difficult to predict what a principal or group of teachers could do with state test data. Knowing that 34% of your kids are “Nearing Proficiency” while 42% of your kids have “Attained Proficiency” simply tells you what you already know: most kids are in the middle. But, again, one doesn’t need data to figure this out either.

Data-driven decision-making comes from the world of manufacturing. And it’s really quite useful for churning out widgets. It’s so easy to adjust your supply chain when you have a solid estimate of Christmas orders you know you need to fill. But measuring learning isn’t at all like manufacturing widgets. And even the psychometricians know it. In fact, most will concede that standardized tests don’t really measure learning. They primarily measure one’s ability to complete a standardized test.

I’ll be the first person to eat my hat if anyone can show me that they’ve actually used test data to make a good decision for their kids – a decision they could not have made any other way. But so far, among the 200 or so schools I’ve worked with, I’ve never seen it. And I’ve never eaten a hat.

Cheers,

Steve

I think we will get to a golden age of data-informed decision-making in my lifetime, but not during my career.

Your point that test may just measure stamina is excellent. I've always had a more primitive theory. Tests measure the ability to read and morale. Do the kids bother to figure out the answer? If the students enjoy school and see a value in education, they'll make an effort. But we've taken advantage of their goodwill.

At Eduwonk, someone said he'd give his left one to see my Value Added Scores, but I didn't take the bait. But I remember a year when my NRT increases were unbelievable so I checked it out. A student who had previously had scores in the single digits turned in a score greater than 90%. The previous year, he explained, he got drunk, but this year he got high.

Well, you better get that info quickly to Arne Duncan so he can put that "test prep technique" into the requirements for "Race to the Top".

John - would this apply to homework, tests, final exams, and papers created and graded by teachers to evaluate students?

None of those things have been "has been subjected to peer review consistent with established social science practice" (your words).

OFFICIAL RUBRIC
XYZ PUBLIC SCHOOL SYSTEM LESSON PLAN

JOHN THOMPSON
TEACHER IDENTIFIER: 12345678910

STANDARD: na
OBJECTIVE: na

WARNING! WARNING! WARNING!
THE ABOVE ENTRIES ARE INVALID
RUBRIC CAN NOT BE POSTED WITHOUT
STANDARD AND OBJECTIVE
AND IDENTIFYING CODE: so sue me

WARNING! WARNING! WARNING!
INVALID RESPONSE

STUDENT: Charlie Barone STUDENT IDENTIFIER: 9876543210

ASSIGNMENT: Free response to the prompt which proposed that when governments utilize standardized tests and statistical models to evaluate teachers or grant tenure that those processes be evaluated according to professional and social science standards.

WARNING! WARNING! WARNING!
FREE RESPONSES QUESTIONS AND WRITING EXERCISES NOT PERMITTED UNDER SCOPE AND SEQUENCE

STUDENT’S THESIS: If a governmental agency must meet such obligations before terminating the property rights of teachers, then a teacher must do the same for all assignments for 140 students.

MODALITY OF EXPRESSION: Satire

WARNING! WARNING! WARNING!
SATIRE NOT AUTHORIZED IN SCOPE AND SEQUENCE

Was that a "yes" or a "no"?

You want me to reveal your grade? Let's just say it wasn't passing.

Or are you asking for a yes or no answer to your tongue-in-cheek question? If you are serious, then that's another example of the reasoning by analogy that has grown with NCLB. "Reformers" thinking that they know enough about classroom instruction because they know politics or finance is like the dynamic that Malcolm Gladwell described in "Cocksure," with investors trusting their Bridge card game skills more than the concrete facts of the economy that they didn't know.

You. like Duncan, should welcome these types of suggestions. Duncan's RttT would incentivize systems for teacher evaluation that would clearly be legal. It would also encourage evaluation systems that would clearly be illegal. You don't think the lawyers would know to bring expert witnesses to the stand? What not accept some constructive suggestions and head off litigations that would cost millions of dollars, untold disruptions to reform, and possibly wreck the momentum Obama education efforts?

The comments to this entry are closed.

Disclaimer: The opinions expressed in This Week In Education are strictly those of the author and do not reflect the opinions or endorsement of Scholastic, Inc.