Thompson: The Story Behind Sharpstown High (& "Apollo 20")

PBS Frontline's Dropout Nation reported that Houston Superintendent Terry Grier had just been on the job for a few monthswhen he heard that four of the district's high schools were failing.  He heard about Roland Fryer’s ideas on school improvement and got in touch with him. Frontline reported that, "After a long phone conversation, Grier gathered a team and headed to Boston to hammer out a plan." It did not report on any effort by Grier to look into evidence for Fryer's hypothesis. 

Eventually, Grier gambled $61 million on his "Apollo 20" reforms. The first year he spent $6 million replacing 310  teachers and the principals of nine schools.  The school featured by PBS, Sharpstown, was not one of the worst of the Apollo 20 high schools, but 39 of the school's 78 teachers were replaced.  Based on Frontline and other coverage, however, it appears that the school benefitted from some of the best of the administrative hires. None of my complaints with Grier's quick-triggered judgment should be taken as a criticism of Sharpstowns' dedicated educators.

Grier still maintains his facile claim about the toughest schools - that we "know what to do with them."  But, his administrators at Sharpstown openly acknowledged their inability to overcome the worst legacies of trauma and generational poverty.  While Grier's spin was consistent with the cherry-picking of Roland Fryer in featuring the experiment's successes, the school administrators' candor was consistent with the data buried in the tables of Fryer's evaluation of Apollo 20. And, as PBS reported, the second year academic results were even more modest.

The most successful aspects of the experiment were the hundreds of highly-recruited math tutors and the counseling efforts of educators outside the classroom.  Each tutor taught two students. As with similar reforms, the first year brought significant increases in math performance, but meager results in reading. By the end of the second year, 79% of Sharpstown students scored Satisfactory or above on a math test.  By contrast, pass rates for reading and writing were 44% and 28%. Moreover, the number of students tested in those subjects was nearly 25% less than those tested in math. 

The district and Fryer have been remarkably opaque in the number of students that were served in the experiment.  An interim report issued in January 2011 reported that Apollo 20 served 7,385 students who were 86.6% low income.  The math results that Fryer showcased were based on 6,097 test takers.  But, the test-taking sample after the first year was 61% low income, raising questions about which students were unable to make it through the longer school year, with additional Saturday school, and the "No Excuses" culture. 

Similarly, Fryer used arcane language to estimate that Apollo 20 had an "attrition rate " of less than 4%. Frontline used plain English to explain how Sharpstown typically started with about 450 freshmen and graduated 275. Also, of Sharpstown's 166 school leavers, 32 supposedly left for private schools.  It's reports of students who left the country seem to be similarly questionable.

This lack of transparency is doubly troubling because Houston's experiment was like so many other "reforms" where non-educators leap before they look at evidence.  Now is the time to analyze what would be necessary to improve outcomes for the students who were not reached by Apollo 20.  Helping the kids who are most at risk of dropping out will require an equally comprehensive effort to coordinate the interventions that really show promise.  That will require us to wrestle with the same hard facts that "Dropout Nation" illuminated.-JT(@drjohnthompson) Image via. 

Comments

[Art]

61% FRL in the treatment schools and 63% in the comparison schools, so that much lines up at least. You're right, however, that Fryer should explain the 61% in the context of earlier statements that the district was 80% FRL. Asking him to clear that up would be responsible and collegial. Roland is a great scientist, but even great scientists can overlook things. Accusing him of cherry-picking his results is a serious charge, and, I believe, an irresponsible one given the flimsy evidence you presented.

[john]

I've asked him.

What's flimsy about the evidence? And if it flimsy, why can't he answer it? I promise I was real nice when I asked him. And, after all, he wrote his report. Nobody stopped him from saying clearly in his introduction how many he started with, how many stuck it out to testing, and what was the low income status before and after. Back in the day, that was considered standard operating procedure.

By the way, the 63% comparison group weakens his argument, by undermining his estimates of benefits. 63% is dang near average. If I had a magic wand and could make all schools close to avarage, what would it be worth?

[Art]

What flimsy about your evidence is that you have none.

And if the comparison schools are very similar to the treatment schools, that increases faith in the results because it reduces the likelihood that the advantage of the treatment schools is due to pre-existing differences among students. I say reduces, not eliminates, because as Fryer acknowledges, this was not a randomized trial.

I agree with you that the report should have been clearer about the characteristics of the analytic sample and the comparison sample at baseline and at testing. But, again, incomplete reporting does not justify the charge that Roland cherry-picked his results. That's a very serious charge against any scientist. That he failed to respond to your real nice inquiry says nothing about his study and does not at all support your charge that he cherry-picked his results.

You seem mad that Roland did not do the study you want. Instead of sniping about that, why not get some schools to try out your ideas?

[John]

You can argue that cherry picking is a serious charge against a scholar, but it is business as usual for an advocate. Clearly, Fryer is acting as an advocate. If he was just putting together a theory piece, his failure to ask the obvious questions in regard to practical policy-making would be no big deal. If he did not know he was leaving out the most important evidence for a policy piece, then that is huge.

I contacted him after hearing his presentation on CSPAN where he acknowledged the research (that you seem to believe is flimsy) on the Matthew Effect and which calls his model into question. He said he'd be looking into alternative explanations.

Surely, now at least, Fryer and other Apollo 20 advocates now know about Heckman, Tough, and the research on the failure of vams to address peer effects in high schools. The question is why did they push ahead with the mass replacement of teachers without first considering it, as well as the warnings in Mass Insight's The Turnaround Challenge.

[HISD former teacher]

Fryer was just one of the many vultures circling HISD trying to squeeze out as much money as they could for their own personal gain, while communities were completely disrupted. I promise you that those teachers who were let go, probably knew so much more about those students than the new people at the school.

Frontline didn't stick around to see the low morale in HISD and how teachers and administrators who can flee the district, do so, even if it's the middle of the year. It's every person for themselves and the last one out--be sure and turn off the light.

[HISD Apollo Teacher]

Let's talk about some of the things that were not noted in the documentary.

Roland Fryer was paid approximately $600,000 for about 6 months of work with the 9 Apollo schools (4 high schools, 5 middle schools) the second semester of the project's first year. Looking at the AYP data for HISD's high schools in 2008-2009, 2009-2010, you will see that the four schools chosen were not the four at the bottom of the pack in the school district. (Not quite sure how the schools were chosen.)

Sharpstown only lost approximately 50% of their teachers while other schools lost a much larger percent in addition to their administration. Any scientist knows that when performing an experiment that you can not have an excessive number of variables because you will not be able to measure the effect of any one. After some of these schools became a part of the Apollo 20 program, the school data plummeted.

While it is nice to highlight how schools are addressing issues such as dropouts, I think that using the Apollo program as a solution for school reform is ludicrous.

[TIRED]

HISD SKEWS DATA.... HISD makes data say whatever they want it to say from the top to the bottom they put their own spin on it to make Grier look good and it is all a sham. Someone needs to investigate how when Grier got here all of a sudden when have contractors for things that were being done by people who work for the district already from curriculum all the way down to lawn service...... Grier is a crook who helps him self and friends. Every audit has been done by a company he has ties to. Look at the AP mess he's on the board HISD will need someone who generaly cares for the students and the city Grier has no ties to Houston or Texas he has not even registered to vote here so what does that tell you??????