Earlier this summer, a team of researchers from the Santa Monica, Calif.-based RAND Corporation examined issues around pay for performance initiatives in five different industries: transportation, child care, education, emergency response, and healthcare. The researchers wanted to find out what issues, potential, and pitfalls, the various industries had around performance-based payment of some sort.

A number of interesting findings emerged. Overall, the researchers found limited effectiveness for all kinds of performance-based payment methodologies across all five industries studied. They did find that performance-based payment can under certain circumstances be successful. The six elements that contribute to success, they found, were the following: “a goal that is widely shared among all stakeholders; measures that are unambiguous and easy to observe; incentives that apply to individuals or organizations with control over the relevant inputs and processes; incentives that are meaningful to those being incentivized; few competing interests or requirements; [and] adequate resources to design, implement, modify, and operate” performance-based reward systems, or what researchers called “performance-based accountability systems” (PBASs). When it comes to healthcare, the researchers found overall that pay for performance programs for hospitals and physicians involved incentive payments that were generally too small, relative to overall reimbursements, to spur the kinds of behavior changes that payers and purchasers were hoping for.

Brian Stecher, Ph.D., Acting Director, RAND Education, at the RAND Corporation, led the pay for performance study. He spoke recently with HCI Editor-in-Chief Mark Hagland about the study’s findings.

Healthcare Informatics: What was your methodology for the study?

Brian M. Stecher, Ph.D.: We began with a literature review: We looked at historical work on performance accountability, and in particular, empirical evidence from the five sectors from the past decade: child care, healthcare, education, transportation, and public health emergency preparedness.

HCI: So, how a performance accountability program is designed is key to its effectiveness?

Stecher: One of the main things we found is that you can’t transplant something that works in one context and necessarily have it work in another. So the kind of model developed in child care isn’t necessarily going to work in transportation. And the elements that seem to require the most care include getting the quantitative measures to be fair to the people affected; getting the right incentives in place that motivate and improve performance without corrupting behavior, and that don’t cost more than the improvements that they yield; and getting an agreement about the valued goals that you’re trying to encourage.

HCI: Does healthcare have particularities that make it truly unique?

Stecher: We found that each of the sectors was unique. At the same time, there were enough commonalities there that we were able to do this comparison. Healthcare is not the only industry in which there is joint production going on; that’s also true in education. The situation in education is that while there’s a primary teacher who is ‘first on deck,’ as a primary care physician would be in healthcare, at the same time, the teacher’s performance will be affected by tutors and many others involved in the educational process. And in child care, you have pooled care, and they often work in teams. And maybe in those cases, the right unit of accountability is not the individual worker, but perhaps a group.

HCI: Healthcare is riven with charges of poor data. Is that true in the other industries you studied?

Stecher: Yes; in fact, healthcare and transportation are better off than some of the other industries, in that respect. In healthcare, the issue is cost; if you wanted to do chart review, you could get really granular data. And in transportation, there’s a wealth of data. But there’s much less in child care and in emergency preparedness. So the problems persist everywhere. But in a lot of cases, you take the best data available, which may not be optimal. In child care, you want to know whether a kid is doing better after two years and is prepared for the next level [such as the next level of preschool, or kindergarten]. But in a lot of cases, you have only pupil-to-teacher ratios and staff credentials to go on.

HCI: In all cases, you make recommendations, among them, that leaders of these PBASs should create the right units of accountability, make the rewards or penalties big enough to matter without exceeding the value of improved performance, and integrate the PBASs with existing performance databases and accounting and personnel systems. What might that mean, for example, in the healthcare context?

Stecher: The recommendation that comes first to my mind is that these systems shouldn’t be static; that the accountability system itself needs to be live, and that you need a mechanism in place for revising and updating it. Because if it’s successful in changing behavior, people will top out on the [performance] measures under the best circumstances, and won’t have any incentives to further improve. So you need an annual or more-frequently-than-annual mechanism for modifying the system, and then need to modify it as a result. So you can’t really think of this as a turnkey kind of thing. It’s hard, though, to get policy consensus; and nobody wants to go back and revisit it. One of the things we noticed in healthcare is that in a lot of these pay for performance systems, the incentives are relatively modest compared to overall reimbursement; and as a result, the changes are relatively modest. So it would seem that there’s not enough there to really grab attention.

HCI: Can you think of any lessons from other industries that healthcare leaders might want to consider?

Stecher: I can’t point to a specific case where I’d say, go look at how the transportation industry handles things differently. But I would say that more money is put at risk in the transportation and construction industry relative to overall payment. I could point to a negative comparison with education, where a lot of main incentives in No Child Left Behind are sanctions and interventions, and those have been shown to be less than effective, because they’re negative. They’re beginning to be effective in eliminating poor performers, but it’s not clear that we can replace them with better performers.

HCI: Any overall thoughts on the data and IT challenges involved?

Stecher: We noted that there are times when the accountability system itself might do a better job if it focuses on inputs rather than outcomes. And IT is a good example. If you think medical records are important, then to get people to invest in them, you might have to build incentives for them to do that, and of course, that has now taken place in healthcare [via the HITECH Act]. And you see some of that in public emergency preparedness, because you don’t have very many cases, and it’s hard to predict what will occur; but you can measure stockpiles of supplies, and you can do the tabletop or live simulations of exercises. So they’re you’re essentially, rather than measuring outcomes, measuring inputs or processes.

HCI: Any other thoughts?

Stecher: I think a continuing conversation across sectors might provide potential value. Right now, we haven’t figured out a forum for doing that. But those of us who participated thought it was remarkably insightful. We did a conference, and maybe we should do another.