Bruno: Why the NAEP Science Scores Are Good News

The NAEP released the grade 8 science test results for 2011 last week, and I've been a little puzzled by the negativity with which they've been greeted. For example:

"There is no cause for optimism," Gerry Wheeler, interim director of the National Science Teachers Association wrote in a statement, because "the results show miniscule gains in student achievement." He lamented that "the majority of our eighth-grade level students still fall below the proficiency level," calling the scores "simply unacceptable."

I don't think anybody would deny that we should want our students to be scoring much higher on the NAEP, but we knew that scores were low after the 2009 test administration. The fact that scores are still too low in absolute terms isn't news, then.

And, yes, average scores for all students increased "only" 2 points since 2009. Just as with the 8th grade math and reading scores, however, that "top-line" number is somewhat misleading because demographic shifts can mask substantial improvements for student subgroups. It turns out that since 2009 we've seen statistically significant score gains for black students, Hispanic students, students of two or more races, and low-income students. (There were also gains that did not rise to the level of statistical significance for American Indian students, students with disabilities, and English learners.) I obviously would have preferred to see, for instance, our Hispanic students gain even more than 5 points, but exactly how much improvement is it reasonable to expect over a two year period?

Ultimately we knew we were starting from weak position in 2009, but we seem to have made some very real gains since then. Viewed in that light, the 2011 NAEP results seem like generally good news. - PB (@MrPABruno) (image source)

Comments

[Bruce]

On first blush they are good news; the New York Times presented them that way. But looking a little more deeply, one sees that the largest single identified group is still that of those scoring below basic in their achievement. So I suppose the disappointment is related to the relatively slow pace of improvement. Indeed, when one looks at the entire panoply of American achievement and attainment data from comparative and historical perspectives, the overall impression is one of improvement over the last few decades; but our rate of improvement has been much slower than that of a number of our economic competitors, and given the high costs of American labor, the competitiveness of our recent graduates in the job market is at extreme peril, and will too often lead to dim futures.

[Paul Bruno]

@Bruce - No doubt faster improvement would be better. The question is, how much more improvement would it be reasonable to expect in a two year period?

[Sarah]

As a fan of mathematics, I’m surprised critics find an average increase of 2 points in two years to be something to, well, critique.

Over such a comparatively short time span, to see an entire population improve at all indicates (what else?) improvement. You can’t expect American students’ scores to achieve perfection overnight. Any upward trend here is a good thing.

[Bruce]

Paul, I wish I could answer your question about reasonable expectations. Unfortunately, after browsing through the technical notes appended to report, I am still unable to determine what two extra points means in terms of grade-level expectations; in reading and math you can usually determine that x number of extra points implies y number of extra months (or years) of advancement, but you can't do that with these science numbers, as far as I can tell. So the numbers shifted a bit in the right direction, but I can't extrapolate any trends from these that might enable us to compare our rate of improvement with that of countries that have been falling and those that are really spectacularly rising, where American manufacturing jobs have been flocking.

[James]

Bruce says: "[T]he largest single identified group is still that of those scoring below basic in their achievement."

The NAEP report says: "Sixty-five percent of eighth-graders performed at or above Basic in 2011."

Seems like the largest single identified group is actually those scoring at or above Basic.

[Paul]

To be clear,the the achievement level breakdown in 2011/2009:

Below basic - 35%/37%
Basic - 31%/31%
Proficient - 32%/30%
Advanced - 2%/2%

[Bruce]

Correct, we still have more students scoring below basic than either, basic, proficient, or (among the tiny number of students considered) advanced; but if you add together the top three of the four categories NAEP uses, combined they outweigh the large number of students still below basic in their achievement. The average (mean and median, though not mode) eighth grader was rated as a basic achiever in both 2009 and 2011.