This Week In Education: THOMPSON: Survivor Bias

The Car Talk puzzler was "a tale worthy of John Le Carre" where "life-saving wisdom emerges from the inky shadows." Interviewing flight crews after a bombing mission, without recognizing "survivor bias" or the different perspectives of the men who did not return, can actually create dangerously false information.

"Reformers" in education, seeking to replicate "best practices" from more effective schools, make the same mistake by not listening to teens who have been driven out of urban schools. How many drop out due to violence or the chronic disruptions that wear down students’ will to learn? As Robert Pondiscio says, how much of the "achievement gap" is actually a "time-on-task gap" because of classroom disruptions?

Survivor bias can also illustrate the inherent contradictions between data-driven accountability and data-driven decision-making.

Engineers charted the damage from World War II bombing missions and recommended that areas in the wings which were hit with the most bullets should receive additional armor. A statistician, however, noticed the relative lack of damage in the cockpit and fuel tanks and realized that the engineers had reached "the exactly wrong conclusion." The engineers had studied survivable damage and not the big threats - from which planes did not return.

Under data-driven accountability, the frequently damaged areas would have been mistakenly up-armored. Who would risk their careers by not taking CYA action when the data seemed to be so clear? Data-driven decision-making, however, takes into account the watch dog that didn’t bark. Students in effective schools do not need to complain about the non-stop "drama" that undercuts engaging instruction. With or without the "no snitching" code, teens are reluctant to admit that fear governs their choices. Take time to listen and educators will hear poignant tales of students driven out of school not by "zero tolerance" policies but by the inability of adults to provide basic protections and a decent learning environment for poor children.

Under data-driven decision-making, we would pay much less attention to survivable flaws in professional development or a lack of curriculum alignment or imperfect differientiated instruction, or whatever, and concentrate on the real killer - toxic learning environments.

Accuracy alert. The above anecdote may be apocryphal. But that just proves my point. When is the last time an enthusiastic proponent of data-driven accountability issued a comparable disclaimer? - John Thompson