DPS '08-09 EOG test numbers look better through "new math" lens of re-tests

One of the nagging questions about this year's end-of-grade test score improvements -- which saw a dramatic increase in most NC school districts' performance -- has been just how much a critical change in methodology in this year's test administration impacted the final results on school performance.

And there's been plenty of reason to suspect that much of the improvement had to do with a chance to allow students a second chance to pass their end of grade (EOG) exams.

Not that the test score improvements haven't been celebrated; the Herald-Sun described school board chair Minnie Forte-Brown as having "practically danced in a school administrative building hallway" when this year's Adequate Yearly Progress (AYP) reports were released.

DPS tripled the number of schools meeting the Federal AYP standard versus its 2007-08 performance -- though only one of twelve high-schools made the standard, versus more than half of elementary schools.

Why such a disparity? BCR's assumption has been that much of the delta here can be chalked up to the fact that elementary schools this year could count a first re-test given to students who didn't pass their EOG tests the first time around -- a rule change that doesn't take effect at the high school level until next year.

We noted here earlier this month that we had asked DPS for the data on what test scores would have been if a first re-test hadn't been allowed to count... that is, an apples-to-apples comparison of scores last year and this year.

The District has now provided those data.

And the bottom line? Except for fifth graders -- where scores seem to have intrinsically increased -- it's hard not to chalk up the largest portion of this year's increase to the re-testing policy.

Here's the reading-math composite scores for 3rd, 5th and 8th graders based on the data DPS released in their 2008-09 "State of the System" performance report, much ballyhooed in the press last week -- and showing, in a comparison of reading and math average passage scores between '07-08 and '08-09, what looks to be some significant improvement:

Now, what do the test scores look like if you discount the re-test? Based on the data we received last week from the District, they look a lot flatter -- a lot less improvement, that is, than you'd think just based on these top-line numbers:

Still some improvement, of course -- notably in fifth grade reading and math, where even on first-testing there's a nice rise. But it's still less dramatic than the data appear without counting the re-test.

Now, this isn't to say that a single re-test shouldn't count; as the N&O noted (in their dive into the giant "re-test" asterisk hanging over this year's EOG scores), plenty of high schoolers take the SAT more than once, and plenty of professional licensure exams see adults go for more than one round.

And there are others -- plenty of them -- who'll point out that part of the trouble in public schools these days is the incessant "teaching-to-the-test" present in school systems desperate to avoid falling afoul of strict No Child Left Behind rules.

The school system can't force its students to have stable home lives, or parents who spend time making education a priority for students. It also can't circumvent a national educational testing mantra whose louder-and-louder drumbeat is leading to endless focus on standardized testing preparation -- and the secondary outfalls like that, such as the increasingly organized and angry group of parents upset about dictated programs like "Reading Street" that get adopted by school districts that feel that poor test scores give them no alternative.

All that said, you can't change the methodology on a test and without wanting to see the data both ways. After all, how can you compare one year's data to another if you don't have like-like grounds for comparison?

Naturally, this isn't a Durham-only phenomenon.

There are probably charts like this floating around in every one of the 106 or so school districts in the state -- all of which should have benefited from this change in the state's approach to calculating these scores.

And, in fact, BCR understands that these data -- the numbers of what the District's scores on reading and math would have been without the re-test option -- were provided to the school board and leadership.

Still, it's hard to discount the PR importance for DPS and other districts of being able to crow about this year's improvement in AYP... even if it came at the hands of a Roger Maris-sized asterisk.

DPS' own scorecard lists EOG goals as having been achieved in all three of these grade levels, comparing the '08-09 goals and results to the '07-08 results -- without a mention that the goalposts have been moved... or that, without the change to the re-test threshold, DPS would have missed its targets for 3rd and 8th grade:

Of course, DPS might well note that they picked their goals for '08-09 knowing that the retesting policy would be in place. Fair enough. But if that was the case, were this year's goals aggressive enough? (Important caveat here: DPS' '08-09 goals also incorporate science tests for at least some of the grades, so the goals as listed here aren't directly comparable to the no-retest, reading/math-only data I've provided above. See the article's end for more on this.)

For what it's worth, the DPS report does note -- in exactly one sentence, on page five, second paragraph -- that re-tests are counted this year.

There's that asterisk for you. And it hangs over this year's performance report every bit as much as it does the AYP scores that derive from end-of-grade testing, in Durham and in the rest of the state.

.....

(One caveat on the original composite number for 2008-09 shown here and referenced in the DPS scorecard, by the way.

DPS noted that the 2008-09 composites on the Performance Report were not directly comparable to the '08-09 no-retest numbers the District provided, since the data that ignore re-tests include only math and reading tests, while the composite scores districtwide include science tests this year.

For comparability purposes, we at BCR used the raw EOG math and reading scores for '08 and re-calculated the total grade composite, leaving science out. A check of the DPS report seemed to suggest that the total grade composite score is a simple average calculation, based on a cross-checking of DPS' '07-08 and '08-09 calculations. DPS' assistant superintendent for research, development and accountability, David Sneed, validated a review set of recalculated '08-09 data excluding science before this article was published.)

Comments

[Matt Drew]

Now if we could only see some historical data and do some trend analysis to see if these scores are part of a trend or are merely outliers. The revised numbers are certainly nothing to be ashamed of.

I find it fascinating, and rather disconcerting, that the only people crunching these numbers for public display is BCR. I assume DPS can do the statistics, which then begs the question of why they don't release results year-over-year. Maybe the tests change too much to get any accurate data?

[TH]

Kevin, thanks for being the only outlet in town that seems to have taken the time to publicly analyze the numbers and report on the background. It's EXTREMELY frustrating that the state and the district put so much emphasize on these scores, yet the NC Report Card's Website and DPS's site STILL have 07-08 scores up... 4 whole months after the 08-09 test were administered.