This Week In Education: Test Scoring: The Elephant In The Room

Some people who tell all about the industry they worked in are greeted as brave whistleblowers and embraced by the media and lawmakers eager to make changes. Others are greeted with suspicion and unease.

Todd Farley is an example of the latter. His book Making The Grades has gotten surprisingly little attention given how harsh and complete is its insider's attack on standardized testing (and accountability). No hearings. No lawsuits. No changes in states' testing regimens or industry scoring practices.

Perhaps the humor throws readers off. Perhaps it's Farley's role, which makes him seem unsympathetic or unreliable. Perhaps it's the challenge his book represents to everyone who's involved in the testing and accountability world.

Read below for a new interview with Farley about the reaction to the book and what that tells us about where school reform stands.

Or just check out my blurb review of the book from a few weeks ago: A Scathing, Humorous Look Inside The Testing Industry

What do you think the book tells us that we need to know?

TF: I believe it proves beyond a shadow of a doubt why this country shouldn’t allow large-scale assessment and for-profit companies to have anything at all to do with American teachers and students.

Is this really new news?

TF: There's never been an insider view of the testing industry like the one I've written, and I think my revelations confirm every nightmare anyone's had about that business.

What's the reaction been to the book, over all?

TF: It seems to be being ignored.

Why did you think it would be more popular?

TF: I assumed that my real-life revelations of the testing industry’s shortcomings would resonate with people, but that doesn’t seem to be the case.

What have been the best and worst reactions you've gotten?

TF: The most rewarding reactions I’ve received have been from Jonathan Kozol, Charles Murray, Linda Perlstein, NEA President Dennis Van Roekel, and Alfie Kohn, people who have made education their life’s work.

The worst?

TF: I can’t say I’ve got too many bad reactions to the book, although I’ve been called an “unreliable narrator” once or twice.

Is that a fair criticism?

TF: It seems some people believe the fact I spent most of my time in standardized testing trying to make money (or live big on my expense account) made me “unreliable,” although most of the reason I was so honest in the book about my personal foibles is because I wanted people to know what it means when you trust for-profit companies (and for-profit people like I was) with making important decisions about American education.

What you described in the book almost sounded like malpractice, or outright deception -- what are the legal options for redress, if any?

TF: The industry itself hasn’t admitted any such thing. They continue to act as if testing is some kind of flawless science.

What about individual families and schools?

TF: It’s always possible when you get a test score back to ask to have it re-scored, even if that might cost you more money. It was because people asked to get their tests re-scored that many massive screw-ups have been uncovered (SAT 2006, Praxis 2004, Minnesota state assessment 2000). Those revelations sure didn’t happen because the testing industry came out and admitted their errors.

Any word from your former colleagues and employers?

TF: Obviously, any number of people I worked with in testing now hate my guts. Plenty others don’t, however. One guy recently told me he was surprised I hadn’t put more examples of “testing dysfunction” in the book, and another congratulated me (a little snidely) for writing the book “everyone he knew had long threatened to write.” That comment I find particularly telling, as it concedes anyone who’s ever been in a test-scoring center knows what a circus it really is. The testing companies have said not a word about my claims, which I find interesting.

Any impact from the book that you know of?

TF: A couple newspapers columnists have mentioned Making the Grades prominently when discussing state testing snafus in Florida, and another woman running for office in South Carolina is touting my book as a reason to eliminate state testing.

How about actual changes in state testing regimens?

TF: No one seems to care I’ve confessed that the testing companies do nothing other than produce numbers, random numbers that don’t necessarily reflect anything about American students or teachers.

You described yourself as an unsympathetic character -- how so?

TF: I portrayed myself (honestly) as guy who during his career was more interested in soccer & travel & expense account living than in the anything as philanthropic or exalted as “education.” I wanted people to know who they are entrusting decisions about American education to when they hire for-profit testing companies to do this work. I think my unapologetic description of myself may make me come across “unsympathetically” to people who care very much for schools and children.

So what are we supposed to do now -- go back to trusting teachers like we used to?

TF: I know beyond a shadow of a doubt that entrusting the testing industry to make tens of millions of decisions each spring about individual students and teachers is an absolutely absurd idea.

What should be happening?

TF: That governments want to keep paying millions of dollars a year to testing companies instead of buying books and hiring teachers is beyond my comprehension. As far as I’m concerned that is just throwing money away. There might be ways for testing to work, but I can't see how that would ever entail hiring short-term employees off the street to assess student responses. Perhaps the system would work if teachers were hired (and paid more) to score state tests, but entrusting bored temps to care about student learing for ten bucks an hour seems illogical to me.

Would we be better off going back to right and wrong (multiple choice) tests, since short response items seem to be the main culprit?

TF: There would be fewer problems with the scoring of those tests than there is with the open-ended, constructed response items that need to be scored by humans. Philosophically, I have to admit I am against the idea of large-scale assessment of that kind, too, but at least I don’t have 15 years of experience watching those tests get horribly mis-scored.

Are all the testing companies you worked for pretty much the same?

TF: ETS is a test development company and I think they do phenomenal work--I would happily entrust the education of my new son to all the people I knew there, especially the ones working on NAEP. NCS Pearson, however, is a test-scoring company, and while I never claimed it was company policy to cook the books in terms of reliability, validity, etc., the reality is those books get cooked all the time.