About this blog Subscribe to this blog

THOMPSON: Real Numbers

Statsignificant "A potential time bomb" threatens New York City’s graduation rate. The 30% of the city’s students who earned a Local Diploma in 2007  will not be able to graduate in 2012 without passing five Regents exams. So, Joel Klein’s Department of Education should have welcomed the audit of the city’s graduation rate.

The press has concentrated on Comptroller Bill Thompson's audit of 43,651 graduates, without noting that they represent only 40% of the students who were scheduled to graduate in 2007 under that year's lower standards. The study questioned whether 9.6% of the graduates met the graduation requirements in terms of earning credits and passing Regents exams (when required.) The audit also showed that 20% of the graduates did not meet the law’s attendance requirements, with graduates having  attendance rates as low as 64%. They originally found "little or no evidence" to justify the classifications of 17.5% of students as "discharges" rather than as possible dropouts. The transcripts of more than 50% of their sample of students categorized as "still enrolled" had been changed (in addition to 25% of the graduates). And they documented other loopholes for understating the dropout rate (students under the age of 17 were not counted as dropouts and 1/7th of the sample of the "still enrolled" appeared to have left school).

Pad the numbers by 9.6% here, 20% there, 17.5% over there, and 14% around the corner, and pretty soon we’re talking about real numbers.

Klein’s DOE submitted after-the-fact documentation regarding most of the 9.6% questioned transcripts, which the auditors accepted, and this is where the real drama lies. New York City uses "annualization," which it claims is "implicitly"allowed by State law to give a full year’s credit to a student who fails the first semester of a course but passes the second semester with a grades as low as 65. Each school can use its own discretion in awarding those credits. Up to 1/5th of graduates were granted multiple credits for passing the same course more than once; for instance granting credit for a senior class to a student who passed the freshman class twice. As the graduation date approached, the number of changes in transcripts, including changes for previous years, were increasingly recorded. One graduate had seven last-minute changes, including the addition of five Spanish credits during the last semester. The auditors also questioned changes to a graduate's summer school records, asking how the student could earn five credits in a short summer school session.

Undoubtedly, many changes were acts of compassion. I can’t imagine an urban educator who hasn’t repeatedly bent or broken the rules. But the audit was not questioning the good, bad, and ugly policies of the NYC School System. It was questioning a) the accuracy of its records and b) the district’s refusal to address its accuracy at a time when principals are under pressure to show results even if it means the inappropriate manipulation data. Klein should welcome an audit on "Credit Recovery" in order to avoid a disaster by 2012 when the 28 to 30% of students earning local diplomas must pass the Regents Exams.  And that does not include many many more, as shown by the audit, who must meet significantly higher standards. - John Thompson

UpdateGotham Schools has done great work on this issue and I can't wait until Jennifer Jennings tackles the latest on the "discharge rate."  And now the Huffington Post is on the "Enron of public education" story.  Given the interest of the National Journal on the subject and the controversies over the District of Columbia's actual performance, perhaps independent evaluators' reports will become a hot new genre.  Everyone should share the joy of being immersed in the methodology and prose of professional auditors.  And maybe Alexander can report on the sexiest practitioners of the green eye shades profession.

Correction.  As explained in the comments, I have changed a typo in the second paragraph.  Responding to David Cantor, I have made it explicit that the original numbers of 9.6% of questioned graduates and 17.5% of discharged were addressed subsequently.  I do not read the audit as determining that all but three of the questioned cases were determined to be "proper."  But that depends how you define the word proper, and you can read further in the comments.




TrackBack URL for this entry:

Listed below are links to weblogs that reference THOMPSON: Real Numbers:


Permalink URL for this entry:


Feed You can follow this conversation by subscribing to the comment feed for this post.

This is a sleazy and incompetent post.

The recent graduation rate audit was conducted by NYC Comptroller Billy Thompson. Mr. Thompson is running for mayor against incumbent mayor Michael Bloomberg, who controls the city's public school system.

The audit was not conducted by Ernst & Young. To the contrary, Ernst & Young has been retained in recent years by the city's Dept of Education to audit graduation and related rates in an effort to ensure the accuracy of the city's calculations. Ernst & Young does not merely audit the graduation rate, as John. Thompson suggests; it also audits discharges and students who remain enrolled past a fourth year--all the possible outcomes of the students in each year's class.

These are not arcane facts.

In his audit the Comptroller found that 195 of the 197 graduates whose records were sampled had the proper qualifications for graduation. Similarly, 79 of 80 students discharged were found to have been properly discharged. These are 99% accuracy rates.

The Comptroller initially identified a larger group of students as having problematic records -- 9.6% of graduates and 17.5% of discharges. John Thompson mischarcterizes these figures as the Comptroller's final determinations. They are not. As he says in his audit, we provided extant records that the Comptroller accepted as establishing the proper qualifications of all but two of the graduates and one of the discharges. He did not have to accept the documentation we provided, and did not do so in a few cases. Throughout our response to the Comptroller's earlier draft, (included as an addendum in his final report) we cite the fact that his findings demonstrated the accuracy of our calculations to 99%. If the Comptroller felt, as John Thompson appears to, that our conclusions were unsubstantiated, he had every opportunity to rebut or address them in his final report. He did not.

I can easily address every apparent anomaly John Thompson cites, from students purportedly attaining mulitiple credits for the same class (did not happen) to the student who gained five Spanish credits in one semester (she was fluent and worked through her entire language requirement with tutors in accordance with state guidelines.

John Thompson would know most of this if he'd read the audit--the DOE response is in there.

David Cantor
NYCDOE Press Secretary

Everyone should read the audit precisely and narrowly; what did the NYC schools document and when did they document it? Then reread the audit for insights into what’s really happening in schools under the gun to make numbers look good. As I indicated, the audit used samples of the 40% of the 2002 to 2007 student cohort and it raises the question of how far the DOE could jack up numbers with the combination of policies like “annualization,” the nonenforcement of the attendance policy, awarding multiple credits for the same course, repeatedly changing transcripts, playing with the definitions of “still enrolled,” and setting their own rules for “discharges,” not to mention “credit recovery.”

Cantor is right on one thing. Originally I intended to explain that we can better benefit from auditors like the Comptroller’s and Ernst and Young if we read precisely and then reread between the lines. Like an idiot, I deleted the word Comptroller instead of the words Ernst and Young.

I never caught the mistake because I was concentrating on the real gold of the audit which was found in the give and take between the auditors and the DOE and the evidence that the DOD seemed to claim had been informed by Ernst and Young’s search of records .

It was the DOE that argued, “In the particular cases where automated student records appear to be inaccurate or incomplete, Ernst and Young follows up at the school level to review the students’ cumulative achievement record.”

I apologize for my typo. I will now change it after acknowledging the mistake.

Getting back to the audit, since individual schools have the power to set or interpret policies, to change records after the fact, and claim that their actions were a part of those policies, the Comptroller’s audit was limited to citing examples where records were out of compliance and recounting the process by which the DOE responded. But that process was illuminating. In regard to the 9.6% of questioned records, the audit wrote, “we found insufficient evidence to justify the rejection of any of the documentation [subsequently] provided ...” But it concluded in bold underlined letters that “the DOE Does Not Require that Schools Ensure that Permanent Student Records are Accurate and Complete.”

My post concentrated on the DOD’s response in regard to those 9.6% of sampled students, and how their exchanges with the Comptroller’s auditors are the best evidence that something is terribly wrong. Cantor also says that the audit accepted them and the discharges as “proper qualifications” and in the narrowest sense he is correct. The issue in all of these cases is whether the NYC schools should have determined whether these students were properly qualified. Regarding the graduates, the audit and the DOD responses raise serious questions. In regard to the discharges, they are silent. I’ll wait until Jennifer Jennings gets on this issue.

Again, I cited questions on discharges, along with the dubious accounts of how students were determined to be qualified, along with the nonenforcement of absenteeism, along with the multiple credits for single classes, along with the repeated changes in records, and along with the charges of improper record-keeping.

Similarly, in regard to “still enrolled” students, the Comptroller’s audit acknowledged that the students could not be classified as dropouts but that was because the schools had not followed the processes that they were supposed to follow after a student is absent for 20 consecutive days. The DOE replied that schools were “expected to enter the dropout retroactively” during the next year. The Comptroller’s audit found no evidence of that expectation, and cited circumstantial evidence to the contrary.

Similarly the auditors could only recount the facts of a student whose summer school records were changed just before graduation to add three credit to the two that had been awarded in a timely manner, and then comment that it is “highly questionable that a student completed five classes during a short summer term.

As with other arcane issues Comptroller’s auditors concluded that “by increasing the incentives of schools to ensure that the students perform well, the risk that schools may take inappropriate actions is increased. However, it does not appear that the DOE acknowledges this reality.”

Throughout the audit, as with David Cantor’s response to my post, the DOE protests too much. How would Cantor respond to a supporter of world class standards citing a student who was already fluent in Spanish, who was awarded 11% of the credits required to graduate in one semester in Spanish 1,2,3,4, and 5, based on the claims of a tutor? What was the value added by the schooling? Even if this specific incident wasn’t inappropriate, how can NYC meet the challenge of the higher graduation requirements in 2012 if every principal can set his or her own bar?

As I wrote, any teacher understands that many of those changes, rightly or wrongly, were made out of compassion. The most damning evidence consistently came from the DOD’s responses, and that is the main story. I am less concerned about how inflated the claims of NYC are. I am more concerned about their refusal to seek more accurate data so they can honestly appraise the effectiveness of their efforts and address the challenge of 2012. That’s why the dry words of auditors is less important than their story of how the DOD tied itself into knots seeking to retroactively justify all of their numbers.

This is bad faith. I urge those who are curious to read the DOE's response to the Thompson audit, where it is appended as an addendum. Mr. Thompson's suggestion that the DOE manipulated data "after the fact" or "retroactively" presumes a.) the Comptroller's draft numbers were definitive and our responding to them was innately illegitimate, and b.) the Comptroller failed to make errors or mischaracterizations in his gathering and presentation of data. In fact, the Comptroller's failed to locate records in schools in many instances.

The DOE addendum responds to Mr. Thompson's many allegations of impropriety. (Not only were no students given multiple credits for the same class, btw, none of those alleged to have done so needed the credits to graduate).

Mr. Thompson skirts interesting issues involving standards and the value of a course credit. He doesn't appear inclined to discuss them reasonably in the context of New York City's experience, however.

David Cantor
NYCDOE Press Secretary

If we keep this exchange up long enough, David might get around to presenting the evidence that he says the DOE has, and which he indicates was included in the audit, but which was not. Or, he might continue the pattern of the DOE shifting its arguments as the debate unfolds. In fact, the audit said that that was the reason why one record was rejected. They received two different explanations from the DOE and didn’t know which to believe.

In fact, that is usually what a careful reader would take from the audit and the responses. It would be ridiculous for me to conclude that all of the DOE’s explanations were “innately illegitimate,” just like it would be unbelievable that all but three were proper. In most cases I can’t gauge the accuracies of the audit’s or the DOE’s evidence and “he said, she said” narratives. Enough of the DOE’s explanations were not credible, however, that it is fair to describe the patterns.

And again, the main pattern was that the “combination of policies like ‘annualization,’ the nonenforcement of the attendance policy, awarding multiple credits for the same course, repeatedly changing transcripts, playing with the definitions of ‘still enrolled,’ and setting their own rules for “discharges,” not to mention ‘credit recovery’” indicates that something is “terribly wrong.”

Yes, the DOE responds to the charges that 39 students in the sample received multiple credits. But the DOE’s responses in the audit were different than the DOE’s response in Cantor’s comment. The DOE said that majority of the multiple credits involved PE, band, yearbook, or advisory, (which the audit says is false) but it only responded specifically in three cases. Only one was mentioned as having more credits than necessary. The DOE also argued that the actual courses weren’t repeated but the course’s codes were misunderstood. The Comptroller replied, that the “DOE provided no evidence to support its claim that these instances were repeated course codes rather than repeats of the same course.”

In cases about the process where it was the auditor’s statements versus the DOE’s without either offering definitive evidence, I remained silent. I’ll admit that the auditors’ dry “nothing but the facts” presentation seemed more credible than the DOE’s repeated attacks on the auditors’ integrity. I know that the Comptroller is a candidate for mayor, but that doesn’t mean that his professional staff compromised their integrity. And I sure don’t believe that I’ve compromised mine.

But in the case of the multiple changes, (as with the obvious issues of annualization, absenteeism and dropout records) I’m confident in my appraisal.

I won’t repeat the whole account of dropouts except as it illustrates the reason why I felt confident to make the specific appraisal that I made. When students are over the age on 17 and miss all but a few days in the spring semester, then the schools are supposed to make the outreach interventions required by law, and document them, and then declare the student as a dropout. Presumably, the school can then engage in recovery efforts and document them. Following mandated procedures does not mean that schools have to give up on the student.

The DOE said that the outreach interventions were made in the spring of 2006-07, but the audit said that no documentation of those interventions was made. The DOE said that some students were dropped on the first day of the fall of 2007-08, also after making the interventions and documenting them. The audit questioned how the interventions could be done when the students were dropped on the first day of school, it denied that evidence was presented regarding interventions in the fall, and it provided circumstantial evidence that there was not an expectation that dropouts would be reported retroactively as claimed by the DOE.

For these and other reasons, I’m sticking with my words of “after the fact,” and “retroactively.” The audit, for instance, said that at the time of their school visits 14 of the 80 discharges did not have documentation, and that the subsequent information came afterwards. Also, the fall semester comes “after” the spring, and changes in grades that were made were just before graduation were made “after”grades that were assigned originally. Exculpatory information was provided “after” the draft was written. And, the DOE also indicated that retroactive changes in transcripts were good because it shows that administrators were working hard to help students. The DOE, itself, wrote “the school is expected to enter the dropout retroactively.”

Lastly, the point of my post and comments was not to show fraud, but to show how the NYC administration has a) compromised standards, and b) been hypocritical about it. Joel Klein and others can be awfully strident when criticizing urban teachers for not having high expectations and lowering our standards. But when the best reading of the evidence is that his schools have done the same thing, and have not kept the records required for data-driven accountability to work its miracles, the DOE attacks the integrity of its critics.

The comments to this entry are closed.

Disclaimer: The opinions expressed in This Week In Education are strictly those of the author and do not reflect the opinions or endorsement of Scholastic, Inc.