BCR's Daily Fishwrap Report for November 3, 2010
Win two tickets to the Nasher Museum of Art's fifth-anniversary gala event after-party

DPS briefing on value-added and pay for performance models illuminates research -- and different board perspectives

Monday night's Durham Public Schools special 5:15pm board meeting featured a discussion of research-based evidence on programs for increasing teacher accountability and incentives, with Duke professor Jacob Vigdor sharing his perspectives on what the latest research had to say about programs that are at the forefront of American education discussions.

Board vice-chair Heidi Carter noted that she hoped the discussion could help to shed some light on what experts in the field were saying on subjects like performance-based pay and bonuses and on value-added growth -- a measure hoped but not proven to gauge a teacher's net impact on a student's learning.

Forty-five minutes later, Vigdor's words may have shed light on those subjects. And the reaction to them -- and between board members -- also shed light on what appears to be at least a small schism among board members to the topics, along with the district administration's interest in the subject.

Not that the differences held any animosity -- or that there's any surprise in them cropping up.  After all, it's a board with differing perspectives and experiences.

Many board members, including board chair Minnie Forte-Brown, have been engaged in programs and trainings focused on American school reform, and the district has partnered with foundations funded by philanthropists like Bill and Melinda Gates and Eli Broad for everything from training on school board effectiveness to support for experimental small high school and performance learning projects.

And topics like value-added growth and pay for performance are key precepts in some schools of education reform, as districts increasingly measured and criticized by standardized test performance and bottom-line, results-oriented, data-centric analysis face pressure from the public and state and federal governments alike to meet institutional goals.

At the same time, the board now has two members -- teacher Nancy Cox, and parent Natalie Beyer, whose candidacy earlier this year found a core base of support in Durham's DARE organization, a confab of parents that organized after last year's Reading Street controversy and tends to raise a skeptical eye towards what they perceive as testing-centric approaches to education.

Monday night's discussion, which included a summation by Vigdor of recent findings in the field that cast at least some skepticism on pay for performance models, served to draw nodding agreement from the latter crowd -- and looks of concern at times, it seemed, from Brown, plus silence from several other board members.

~ ~ ~ ~ ~

Vigdor -- a Harvard-trained economist, member of the prominent National Bureau of Economic Research (NBER) and, he noted, the parent of two Durham Public Schools students -- noted that pay for performance programs could be looked at as logical or controversial, depending on the approach taken to the question.

There is little to disagree with, Vigdor said, in the concept of wanting to say "that we want to recognize good teaching, that teachers that do an excellent job of what we want them to do should be recognized, should be rewarded" through recognition or possibly monetary rewards. And he said that the idea that ineffective teachers should face incentives to improve also gets little argument.

"Essentially what we're talking about is a world in which we recognize differences among teachers and treat them differently on the basis of those teachers," Vigdor said.

But he noted that it's difficult to get from this general concept to a specific approach to how to measure who is, and who is not, a successful teacher.

"What makes it a little bit difficult in practice is that if you want to recognize good teaching, you have to know when it's there," Vigdor told the board. 

Value-added growth models are a popular way of calculating a teacher's impact, he said; these systems compare the test performance of students at the end of an academic year with those students' scores when they entered the classroom, and measure growth not on the final average level of achievement -- a measure that would seem biased against teachers whose students were low-performing when they entered the classroom -- but on the relative growth in test score or school year equivalency over that year.

It's a measure, in other words, that looks at the growth in student performance over a year and imputes that growth (or lack thereof) to the work of the teacher. 

"What a lot of people have seized on is this idea that these tests, which were originally used to evaluate students, can also be used to evaluate teachers in the classroom." 

The test scores do provide at least an objective measure, not the subjective measurement of a principal or outside observer, he noted; and end of grade testing is already mandated by No Child Left Behind-era rulemaking, meaning there's little cost to collect the data.

But the concept runs into problems in execution, Vigdor said. Many students spend time divided between different teachers, for instance; in other cases, some students enter from grades that lack state-required end of grades testing.

And he noted that fundamentally, value-added growth models use test scores because these are the data they have; the tests were designed for measuring student achievement, not teacher achievement, though Vigdor conceded that one could view the end result of demonstrated achievement as the end on which teachers' work is driven.

"Value added -- maybe it measures what the teacher's doing, but maybe it's measuring luck," he said.  "We're not really sure that reading and math tests capture everything that a student is supposed to accomplish in a year. 

Vigdor cited a just-released US Department of Education-funded Vanderbilt University study out of the Nashville, Tennessee (Davidson Co.) school district, where teachers could receive bonuses of up to $15,000 based on their students' value-added growth performance.

"They ran it like a clinical trial; they randomly assigned some teachers to participate in the bonus program, and then other teachers were in a control group," Vigdor said. "And the question at the end of the day was, whether the kids who were being served by these teachers who were enrolled in the bonus program did any better than the children who were" in the control group.

"The answer is no. They found no noteworthy differences between the children in the bonus regime versus the control group," Vigdor said, noting that at least under this design of a pay-for-performance system, the outcomes seemed to not be promising.

Vigdor added that the Vanderbilt researchers spoke to teachers to get a qualitative opinion on how effective such measure were. "They felt like whether the kids raised their test scores or not was something that was to a large extent beyond [the teacher's] control. If you perceive something is not within your control… then it's not too surprising that we don't convince you to change your behavior all that much."

He did concede that the Vanderbilt study was populated by teachers who volunteered to participate -- a seeming selection problem in that those teachers may have been more likely to have been strong teachers to begin with (in reality, or at least in their own minds.)

The professor warned against designing performance measurement systems that create moral hazards, like the finding in a Chicago study that teachers who were rewarded primarily based on how many students they push above a proficiency threshold. That goal, Vigdor said, was found to incentivize teachers to pay disproportionate attention to students just on the cusp of the goal threshold -- leaving high-performing and perceived-'hopeless' very low performing students with less time and support from teachers in the study.

~ ~ ~ ~ ~ 

At the wrap-up of Vigdor's presentation and initial board member comments, new DPS superintendent Eric Becoats opened his remarks with a comment that seemed to clearly stake out the new administration's position on the subject.

"I think it would be very helpful for us to have some of the individuals who develop value-added [systems] to also come and speak to us, because I think we need some balance in this discussion," the superintendent said after Vigdor's somewhat pessimistic findings.

Becoats noted that pay for performance is "one of the items" he had put before the board, and added that test result improvements alone were not the only rationale for looking at such programs, adding that they can be used to reduce turnover or improving staffing in hard-to-hire-for schools or subject areas.  And he added that he understood the need to look at the test scores of any teacher's pupils for a multi-year period to smooth out variations within a small sample size.

But the superintendent, noting "lots of different sides" on the issue of pay for performance, noted that end-of-grade and end-of-course tests are, "whether we like it or not," the measurements for success -- and that it's DPS's role to use these tests to assess how well students are performing. 

And Becoats was clear in wanting to ensure at least some linkage between student success and teacher performance.

"I'm a firm believer that the child is not the reason why… the child is successful or not successful," Becoats said. "If children are to be successful in the classroom, I suspect that the teacher has to take some responsibility for that, because that is the person who we look to for the instructional program."

Board chair Forte-Brown spoke up to back Becoats' position on the subject. 

Forte-Brown noted that there are many issues that contribute to a student's academic success or failure, including poverty, parental literacy levels and the home life.

But she advocated a course for the district that focused on the district's role as the sole entity held responsible for children's academic performance, regardless of their home circumstances -- and exhorted her colleagues to focus on ensuring that extrinsic home factors not factor in as acceptable reasons for low performance. 

"We are the only place that's being held accountable for teaching children to read and calculate," Forte-Brown said. "We have to do that job regardless of where they come from, regardless of what their behavior is, because that's our job."

And the value of a teacher, at some level, comes down to how well they do that job, Forte-Brown said.

Other board members seemed to have more concerns on the subject of pay for performance and value-added modeling, and on the question of the relative responsibility of teachers for educational outcomes.

Nancy Cox joined Heidi Carter in asking about systems that would include composite measures of in-classroom evaluation in addition to any test-score measures. Besides test scores, Cox said, as a teacher she "also need to be evaluated on how effective I am as an instructor."

While Cox's point on mixing observations with any quantitative metrics on teacher performance drew no disagreement from the group -- "I haven't heard anybody say test scores should be used and that's it," Vigdor said, noting that that seemed to be a consensus among board members -- Cox did express a concern on the role of the district versus extrinsic factors in the responsibility for educational outcomes.

If classroom observations found that a teacher was doing things right -- running an organized classroom, managing the students, and so forth -- "and the kids still aren't learning… you do have to look at some of those outside influences," presumably those that Forte-Brown said could not be excuses for low performance.

Beyer's question to Vigdor, meanwhile, sought to reconfirm that standardized tests that get repurposed for value-added growth assessment were designed for measuring students, not teachers.

Both Beyer and Cox seemed to nod thoughtfully, even enthusiastically, when Vigdor raised research-based concerns on pay for performance systems' effectiveness -- though Forte-Brown and some other board members looked much more stoic on the matter.

~ ~ ~ ~ ~

All of which points to what could be an interesting debate in the months to come from Durham's school leaders.

National educational reformers have been buoyed by the attention paid to films like "Waiting for 'Superman'," the charter school paean that expresses concerns with teachers unions for resisting reform and accountability. (The film gets a Durham screening sponsored by the national and Durham chambers of commerce next week.)

Yet they've also seen backlash from parents groups like DARE locally, and from many in the research community -- and, in Washington, DC, from the recent loss of school-reformist Mayor Adrian Fenty and the subsequent resignation of controversial DC schools chancellor Michelle Rhee.

The questions at play in Monday's meeting, while technical in origin, ultimately play out along the same dimension: to what extent will Durham Public School seek to leverage standardized test scores as a bird-in-hand measure for teacher evaluation?

No one, including Becoats or his hand-picked number-two man Lewis Ferebee, argued for making them a sole metric, though it was clear from both of their comments that they seem aligned with making it at least one very key touchstone.

(Ferebee's comment, which noted that value-added tests could be helpful if broken down by subgroups of students in identifying teachers who might thrive with high-performing and low-performing students if not overall, showed a seemingly striking similarity with the subject matter.)

Meanwhile, the board's discussions weren't playing out solely among themselves.

The audience, as Carter acknowledged at meeting's end, was filled with school principals.

All of whom are, along with activist parent groups, waiting interestedly to see what direction the district takes under the in-the-works strategic plan -- and what level of board support reform will enjoy.

(See also the Herald-Sun's reporting on the subject from Tuesday's paper.)

 

Comments

Emily Geizer

Doug Lemov has done some interesting work in this area and recently published the book, Teach Like a Champion. In it he describes 49 specific techniques that excellent teachers use.

Before evaluating all teachers, it would be wonderful to see these skillful techniques taught first and then measured for implementation and efficacy.

Rodrigo "El Justiciero" Dorfman

If administrative jobs were also linked to student performances I think we would be having quite a different discussion. The teachers are in the front lines but the rug is being constantly pulled from under their feet by top down decisions from administrators who act on "models", "charts"and "statistics".

And to declare, like Brown does that the home environment of the child should not play a role in assessing his or her abilities is short sighted and ignores the 800 pound gorilla in the room: social class.

Teachers need more training, better pay across the board, different tools to assess their students and a seat at the table.

One could ask: do we have an "achievement gap"? or a "testing gap?".
Until we figure out a way to evaluate the WHOLE child, judging a teacher with narrow minded tests like EOG's is a mistake.

Elizabeth Tolman

Emily Geizer, I agree 100%! Lemov's book is amazing. I would love to see DPS invite him to Durham to lead DPS teachers in a series of workshops and book discussions. He gives pointed techniques that all teachers can implement, whether they teach art to elementary school kids, history to high school students, or Spanish to graduate students (that would be me).

Julie Maxwell

Thank you to Jacob Vidgor for so thoughtfully and thoroughly addressing this subject with the board. There seem to be many nefarious influences swarming around the issue of US education "reform."

Durham is very fortunate to have independent press coverage and knowledgable involved citizens like Jacob. Perhaps we can slow down on adopting popular but unproven edu strategies and avoid undermining our public school system attempting to conform to the corporate model favored, financed and promoted by education philanthrocapitalists like Eli Broad (KB Homes, AIG insurance), Sam Walton (Wal-Mart), and Bill Gates.

Tina

So last year the Central Administration told elementary teachers how they must teach literacy and provided teachers with very little REAL flexibility. Now the Central Administration wants to hold teachers accountable for the effectiveness of the outcome of this mandate?? Am I the only one who sees this as crazy? And know that this sort of mandate can easily trickle from Elementary to Middle to High School.

By admitting that social class and support systems effect student's learning does NOT mean teachers are giving up on students, as it seems that some board members have implied. How can we really reach the child unless we understand ALL the problems. Let's find out who needs more support at home, whose parents need some literacy training themselves, who goes to bed hungry, AND then we can find REAL solutions... finding grants, finding community organizations to work with these families IN ADDITION to telling a teacher it's your sole responsibility to save this student from failure. Teachers must believe in their students and be properly trained to meet the needs of students BUT they can't do this all on their own. IT TAKES A VILLAGE!!

And if we're not careful all the teachers are going to up and leave for a village where they feel supported and where THEIR students who they toil over and care about daily get REAL support from their communities instead of just scapegoating the teachers!

DRR

As a DPS teacher, I can say that I do not dislike the idea of accountability, but I am often concerned by how it is implemented. I think that performance-based pay can be done correctly, but there are many ways it can be done incorrectly.

Potential issues that I see:

1. The testing for this would potentially be overwhelming for students. In order to evaluate student growth, there would need to be testing for all subjects in all grades.

2. How are we going to evaluate art teachers? PE?

3. Can this be done in a non-competitive way? Will my pay be based on my students doing better than other students? If so, doesn't that discourage me from working with and helping other teachers? If my pay is not based on the success of other teachers, what happens if all teachers do really well and deserve a raise? Where does that money come from?

Between the budget cuts and layoffs of the previous years and now this discussion teacher morale is lower than I have ever seen it around here. I have been hearing more and more good, hard working teachers talking about leaving the classroom and it really does concern me. I am not going to stand on a soapbox and demand huge pay increases or anything like that, but I think that there would be a lot to be gained if the powers that be took the time to praise the successful teachers we have, rather than pass to us the blame for every problem that we fight against everyday.

Kirsten Kainz

In this response I am neither advocating for using state tests to evaluate teachers nor for incenting teachers through performance bonuses. However, I am advocating that we start this conversation and respectfully consider the pros and cons of such efforts so that we can use our deliberations to improve opportunties for all students, but especially opportunities for our economically disadvantaged students. Several commenters on this page seem to be pointing us in a useful direction.

DRR raises some exceptionally important points. Even if using state tests to evaluate teachers were a reliable practice, it would only provide feedback on a fraction of the educational program in DPS. We need more information and from more sources to ensure all children have consitently high quailty education opportunities in DPS.

Related to DRR's request for recognition for successful teachers are the comments of Ms. Geizer and Ms. Tolman who direct our attention to published accounts of quality instruction. One reason individuals and institutions can be attracted to test scores as indicators of teacher quality is that it is hard to generate consensus around what makes a good teacher. Some people are comfortable saying that they know a good teacher when they see one, but I think if our community is going to make good on its desire to improve education for all students we will have to align our definition of quality teaching with quailty learning.

This brings me to a final point. I deeply appreciate Ms. Forte-Brown's willingness to make us uncomfortable with the fact that in Durham (like the rest of the nation) our poor students, our African-Amreican students, and our Latino students do not enjoy markers of high academic achievement at the same rate as our European-American and affluent students do. These markers include test scores, grades, participation in advanced courses, participation in enrichment activities, and high school completion. So, this is not merely a testing gap that we are witnessing. It is an achievement gap. And this achievement gap is related (though perhaps not causally) to a lifestyle gap defined by disparities in income, wealth, safe neighborhoods and housing, and health. To make us uncomfortable about this fact is not the same as saying that it shouldn't be a consideration in how we educate children. That is not the proper interpretation of Ms. Forte-Brown's comments. Rather, I interpret Ms. Forte-Brown's comments as a strong invitation to acknowledge the extreme discomfort that accompanies our recognition of these achievement and lifestyle gaps. DRR's comments compliment those of Ms. Forte-Brown's in that DRR reminds us that we shouldn't rush to blame teachers when we confront this discomfort.

Chana

What's disturbing to me about this whole trend, in addition to the continual scapegoating of teachers and attempts to mess with their already inadequate pay, is that researchers have been saying for decades that standardized tests are biased against poor people, and people of color. I'm a GED teacher, and I see this first-hand every day. I've had students who've gotten "reading" questions wrong because they didn't know who Marilyn Monroe was, or they weren't sure what kind of articles "Field and Stream" usually publishes, or they've never heard the expression "a watched pot never boils." I'd say a good 25% of that test is essentially a test on middle class culture, not reading comprehension-and that's in addition to the questions that are just poorly constructed, and could trip up any kid (I've seen quite a few questions where I didn't think any of the answers were right, as well as questions where more than one could have been). How can we be using these tests to decide anyone's grade status or pay????

JC

This is a good conversation here, and I especially appreciate Prof. Kainz's comments.

I don't believe anyone disagrees with the concept of assessing performance. We absolutely must gauge whether our schools are effectively educating our children. Surely, no one disagrees with that. And likewise, I'm sure that all would agree that excellent teaching should be rewarded, that teachers who are struggling should get help and opportunities to become more effective, and teachers who are chronically ineffective should be removed from the system. I'm pretty confident there would be little argument from from anyone on those points.

Thus, the disagreement comes with respect to the tools we are using to assess student performance and teacher performance.

My point is that, let's not argue about whether we need to ensure our schools are getting the job done. If you want to argue about the assessments, fine, I think there is a legitimate debate to be had there.

I don't think anyone involved in this conversation is trying to scapegoat teachers. Teachers are incredibly important components in the education of our children, but if there is one message that comes out of Geoffrey Canada's work, and I believe out of Waiting for Superman, it's that programs such as the Harlem Children's Zone and Kipp schools succeed because they work very hard to address those issues around disparities in income and health and lifestyle.

Steve

Before touting Waiting for Superman too much, you might read the thoughts of historian Diane Ravitch's here: http://www.nybooks.com/articles/archives/2010/nov/11/myth-charter-schools/?pagination=false . Oh wait, you might also read her blog post The Problem With Value-Added Assessment here: http://blogs.edweek.org/edweek/Bridging-Differences/2010/10/dear_deborah_you_asked_what.html .

Meanwhile, I'll go back to just being a teacher who can't wait to see how he's going to be evaluated this week and who will point the finger of blame this time. Sorry JC, if you don't think teachers aren't being walloped across the nation, you need to get out more.

P.S. How the heck did all those excellent teachers from when I was in school ever manage to teach without knowing Lemov's techniques or before the creation of value-added assessment?

JC

Steve's response to my comment is a great example of why this debate is so frustrating. Emotion and hyperbole dominate, and people fail to see that they agree on the core issues.

My whole point was that we can and should be arguing over whether the current assessments are good measures of performance, and we should be working together to identify great ways to effectively measure performance and achievement.

Are we effectively educating all populations? Are there areas where we need to improve? Are we giving teachers the support they need and not burdening them with useless bureaucracy? Those are questions we are all interested in understanding.

Or maybe not.... Perhaps we should just leave the educators and the administrators and elected officials alone and let them do their job. That's how things operated when I was a teacher, and yes, there were some terrific teachers. But there were huge populations of students not being adequately served. Expectations were low for minority and underprivileged students, the worst or newest teachers were assigned to the lowest-performing students. And in the past 20 years, I don't think things have improved -- correct me if I am wrong.

We have to figure out how to fix our broken system, and I think all engaged in the debate are engaging with the best of intentions.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)