DPS briefing on value-added and pay for performance models illuminates research -- and different board perspectives
Monday night's Durham Public Schools special 5:15pm board meeting featured a discussion of research-based evidence on programs for increasing teacher accountability and incentives, with Duke professor Jacob Vigdor sharing his perspectives on what the latest research had to say about programs that are at the forefront of American education discussions.
Board vice-chair Heidi Carter noted that she hoped the discussion could help to shed some light on what experts in the field were saying on subjects like performance-based pay and bonuses and on value-added growth -- a measure hoped but not proven to gauge a teacher's net impact on a student's learning.
Forty-five minutes later, Vigdor's words may have shed light on those subjects. And the reaction to them -- and between board members -- also shed light on what appears to be at least a small schism among board members to the topics, along with the district administration's interest in the subject.
Not that the differences held any animosity -- or that there's any surprise in them cropping up. After all, it's a board with differing perspectives and experiences.
Many board members, including board chair Minnie Forte-Brown, have been engaged in programs and trainings focused on American school reform, and the district has partnered with foundations funded by philanthropists like Bill and Melinda Gates and Eli Broad for everything from training on school board effectiveness to support for experimental small high school and performance learning projects.
And topics like value-added growth and pay for performance are key precepts in some schools of education reform, as districts increasingly measured and criticized by standardized test performance and bottom-line, results-oriented, data-centric analysis face pressure from the public and state and federal governments alike to meet institutional goals.
At the same time, the board now has two members -- teacher Nancy Cox, and parent Natalie Beyer, whose candidacy earlier this year found a core base of support in Durham's DARE organization, a confab of parents that organized after last year's Reading Street controversy and tends to raise a skeptical eye towards what they perceive as testing-centric approaches to education.
Monday night's discussion, which included a summation by Vigdor of recent findings in the field that cast at least some skepticism on pay for performance models, served to draw nodding agreement from the latter crowd -- and looks of concern at times, it seemed, from Brown, plus silence from several other board members.
~ ~ ~ ~ ~
Vigdor -- a Harvard-trained economist, member of the prominent National Bureau of Economic Research (NBER) and, he noted, the parent of two Durham Public Schools students -- noted that pay for performance programs could be looked at as logical or controversial, depending on the approach taken to the question.
There is little to disagree with, Vigdor said, in the concept of wanting to say "that we want to recognize good teaching, that teachers that do an excellent job of what we want them to do should be recognized, should be rewarded" through recognition or possibly monetary rewards. And he said that the idea that ineffective teachers should face incentives to improve also gets little argument.
"Essentially what we're talking about is a world in which we recognize differences among teachers and treat them differently on the basis of those teachers," Vigdor said.
But he noted that it's difficult to get from this general concept to a specific approach to how to measure who is, and who is not, a successful teacher.
"What makes it a little bit difficult in practice is that if you want to recognize good teaching, you have to know when it's there," Vigdor told the board.
Value-added growth models are a popular way of calculating a teacher's impact, he said; these systems compare the test performance of students at the end of an academic year with those students' scores when they entered the classroom, and measure growth not on the final average level of achievement -- a measure that would seem biased against teachers whose students were low-performing when they entered the classroom -- but on the relative growth in test score or school year equivalency over that year.
It's a measure, in other words, that looks at the growth in student performance over a year and imputes that growth (or lack thereof) to the work of the teacher.
"What a lot of people have seized on is this idea that these tests, which were originally used to evaluate students, can also be used to evaluate teachers in the classroom."
The test scores do provide at least an objective measure, not the subjective measurement of a principal or outside observer, he noted; and end of grade testing is already mandated by No Child Left Behind-era rulemaking, meaning there's little cost to collect the data.
But the concept runs into problems in execution, Vigdor said. Many students spend time divided between different teachers, for instance; in other cases, some students enter from grades that lack state-required end of grades testing.
And he noted that fundamentally, value-added growth models use test scores because these are the data they have; the tests were designed for measuring student achievement, not teacher achievement, though Vigdor conceded that one could view the end result of demonstrated achievement as the end on which teachers' work is driven.
"Value added -- maybe it measures what the teacher's doing, but maybe it's measuring luck," he said. "We're not really sure that reading and math tests capture everything that a student is supposed to accomplish in a year.
Vigdor cited a just-released US Department of Education-funded Vanderbilt University study out of the Nashville, Tennessee (Davidson Co.) school district, where teachers could receive bonuses of up to $15,000 based on their students' value-added growth performance.
"They ran it like a clinical trial; they randomly assigned some teachers to participate in the bonus program, and then other teachers were in a control group," Vigdor said. "And the question at the end of the day was, whether the kids who were being served by these teachers who were enrolled in the bonus program did any better than the children who were" in the control group.
"The answer is no. They found no noteworthy differences between the children in the bonus regime versus the control group," Vigdor said, noting that at least under this design of a pay-for-performance system, the outcomes seemed to not be promising.
Vigdor added that the Vanderbilt researchers spoke to teachers to get a qualitative opinion on how effective such measure were. "They felt like whether the kids raised their test scores or not was something that was to a large extent beyond [the teacher's] control. If you perceive something is not within your control… then it's not too surprising that we don't convince you to change your behavior all that much."
He did concede that the Vanderbilt study was populated by teachers who volunteered to participate -- a seeming selection problem in that those teachers may have been more likely to have been strong teachers to begin with (in reality, or at least in their own minds.)
The professor warned against designing performance measurement systems that create moral hazards, like the finding in a Chicago study that teachers who were rewarded primarily based on how many students they push above a proficiency threshold. That goal, Vigdor said, was found to incentivize teachers to pay disproportionate attention to students just on the cusp of the goal threshold -- leaving high-performing and perceived-'hopeless' very low performing students with less time and support from teachers in the study.
~ ~ ~ ~ ~
At the wrap-up of Vigdor's presentation and initial board member comments, new DPS superintendent Eric Becoats opened his remarks with a comment that seemed to clearly stake out the new administration's position on the subject.
"I think it would be very helpful for us to have some of the individuals who develop value-added [systems] to also come and speak to us, because I think we need some balance in this discussion," the superintendent said after Vigdor's somewhat pessimistic findings.
Becoats noted that pay for performance is "one of the items" he had put before the board, and added that test result improvements alone were not the only rationale for looking at such programs, adding that they can be used to reduce turnover or improving staffing in hard-to-hire-for schools or subject areas. And he added that he understood the need to look at the test scores of any teacher's pupils for a multi-year period to smooth out variations within a small sample size.
But the superintendent, noting "lots of different sides" on the issue of pay for performance, noted that end-of-grade and end-of-course tests are, "whether we like it or not," the measurements for success -- and that it's DPS's role to use these tests to assess how well students are performing.
And Becoats was clear in wanting to ensure at least some linkage between student success and teacher performance.
"I'm a firm believer that the child is not the reason why… the child is successful or not successful," Becoats said. "If children are to be successful in the classroom, I suspect that the teacher has to take some responsibility for that, because that is the person who we look to for the instructional program."
Board chair Forte-Brown spoke up to back Becoats' position on the subject.
Forte-Brown noted that there are many issues that contribute to a student's academic success or failure, including poverty, parental literacy levels and the home life.
But she advocated a course for the district that focused on the district's role as the sole entity held responsible for children's academic performance, regardless of their home circumstances -- and exhorted her colleagues to focus on ensuring that extrinsic home factors not factor in as acceptable reasons for low performance.
"We are the only place that's being held accountable for teaching children to read and calculate," Forte-Brown said. "We have to do that job regardless of where they come from, regardless of what their behavior is, because that's our job."
And the value of a teacher, at some level, comes down to how well they do that job, Forte-Brown said.
Other board members seemed to have more concerns on the subject of pay for performance and value-added modeling, and on the question of the relative responsibility of teachers for educational outcomes.
Nancy Cox joined Heidi Carter in asking about systems that would include composite measures of in-classroom evaluation in addition to any test-score measures. Besides test scores, Cox said, as a teacher she "also need to be evaluated on how effective I am as an instructor."
While Cox's point on mixing observations with any quantitative metrics on teacher performance drew no disagreement from the group -- "I haven't heard anybody say test scores should be used and that's it," Vigdor said, noting that that seemed to be a consensus among board members -- Cox did express a concern on the role of the district versus extrinsic factors in the responsibility for educational outcomes.
If classroom observations found that a teacher was doing things right -- running an organized classroom, managing the students, and so forth -- "and the kids still aren't learning… you do have to look at some of those outside influences," presumably those that Forte-Brown said could not be excuses for low performance.
Beyer's question to Vigdor, meanwhile, sought to reconfirm that standardized tests that get repurposed for value-added growth assessment were designed for measuring students, not teachers.
Both Beyer and Cox seemed to nod thoughtfully, even enthusiastically, when Vigdor raised research-based concerns on pay for performance systems' effectiveness -- though Forte-Brown and some other board members looked much more stoic on the matter.
~ ~ ~ ~ ~
All of which points to what could be an interesting debate in the months to come from Durham's school leaders.
National educational reformers have been buoyed by the attention paid to films like "Waiting for 'Superman'," the charter school paean that expresses concerns with teachers unions for resisting reform and accountability. (The film gets a Durham screening sponsored by the national and Durham chambers of commerce next week.)
Yet they've also seen backlash from parents groups like DARE locally, and from many in the research community -- and, in Washington, DC, from the recent loss of school-reformist Mayor Adrian Fenty and the subsequent resignation of controversial DC schools chancellor Michelle Rhee.
The questions at play in Monday's meeting, while technical in origin, ultimately play out along the same dimension: to what extent will Durham Public School seek to leverage standardized test scores as a bird-in-hand measure for teacher evaluation?
No one, including Becoats or his hand-picked number-two man Lewis Ferebee, argued for making them a sole metric, though it was clear from both of their comments that they seem aligned with making it at least one very key touchstone.
(Ferebee's comment, which noted that value-added tests could be helpful if broken down by subgroups of students in identifying teachers who might thrive with high-performing and low-performing students if not overall, showed a seemingly striking similarity with the subject matter.)
Meanwhile, the board's discussions weren't playing out solely among themselves.
The audience, as Carter acknowledged at meeting's end, was filled with school principals.
All of whom are, along with activist parent groups, waiting interestedly to see what direction the district takes under the in-the-works strategic plan -- and what level of board support reform will enjoy.
(See also the Herald-Sun's reporting on the subject from Tuesday's paper.)