Skip to content

What Counts as Success in Energy Efficiency Programs?

Most countries’ plans for reducing greenhouse gases rely heavily on energy efficiency programs. So, even if you aren’t an energy efficiency specialist, it’s important to understand how, and how well, those savings get counted. When it comes to measuring the impacts of energy efficiency programs, this means disentangling which energy consumption changes can be credited to the program, and which would have happened anyway. For those of you who aren’t steeped in the intricacies of this energy efficiency accounting exercise, let me provide a bit of background.

My husband often compares electricity market regulations to the rules for Dungeons & Dragons, a 1980s gaming craze that captivated some – mostly pre-teen boys – and bemused everyone else. It involved multiple 100+ page rulebooks that explained things like how many “experience points” a character might get for slaying a certain monster.

Energy efficiency policy can be as complex as Dungeons & Dragons. This isn’t a criticism. It’s endemic to energy efficiency policy for one fundamental reason: it’s really difficult to measure the savings from energy efficiency programs. There’s no meter that runs backwards to measure those savings. As a result, there are lots of discussions and evolving rules on how to measure them.

To take an example, imagine that your local utility offers rebates for homeowners who buy energy efficient hot water heaters. The task for the regulators is then to assess how much less energy homeowners will consume because of the rebate program.

One of the core issues is figuring out how many homeowners would have bought an efficient hot water heater even without the utility rebate. Those who would have are labeled “free-riders” – they get paid the rebate for doing something they would have done anyway.

A standard method for measuring savings from a rebate program is to first develop an “engineering estimate” of the savings associated with each hot water heater replacement and then add those savings up across everyone who applied for a rebate. This represents the “gross” savings.

The next step is to figure out what share of homeowners are free-riders and then take that share out in order to estimate the “net” savings. Sometimes, regulators ask consultants to survey homeowners about whether they would have bought the efficient hot water heater absent the rebates. This can be an inherently difficult process – effectively asking people to construct a counterfactual world for their past selves.

Colleagues and I have argued elsewhere that there are better ways to develop estimates of savings from energy efficiency programs, including randomized controlled trials and other quasi-experimental approaches. One of the advantages of these approaches is that they provide an estimate of the net-to-gross ratio. (They can also highlight other shortcomings of the engineering estimate approach I described above, for instance if the engineering estimates are too optimistic.)

For instance, a few years ago, Judd Boomhower and Lucas Davis applied a quasi-experimental technique to evaluate a “cash-for-coolers” rebate program in Mexico that subsidized refrigerator and air conditioner replacements. They found that even without the rebate about half of the participants would have replaced their inefficient appliances with a more efficient one, and many would have done so for a lower rebate than they were paid.

Despite this sort of evidence, I’ve heard several energy efficiency advocates, recognizing the difficulty in calculating net-to-gross ratios, assert that it doesn’t really matter. They say things like, “The climate doesn’t care about the difference between net and gross savings.” The implication seems to be that we should pat ourselves on the back for achieving the gross savings.

But, focusing on gross savings is problematic for two reasons:

  • We will not solve climate change by focusing only on U.S. energy customers. (This point has come up repeatedly on this blog, such as here and here.) One implication of this is that we need to export the right policies to the rest of the world. So, it’s essential to figure out whether an energy efficiency program causes people to take certain steps or whether they would have taken them anyway.

Imagine, for example, that the hypothetical utility rebate program for hot water heaters has a low net to gross ratio (i.e., a lot of people would have bought the efficient hot water heater even without the rebate). Maybe they would have bought the efficient hot water heater because of another energy efficiency policy, such as efficiency standards. We need to know if standards are doing all the heavy lifting  so that other parts of the world will adopt the most effective policies.

And, it’s really true that other countries are watching what we’re doing carefully. A couple months ago, I spoke to a deputy energy minister from the Pakistani province of Punjab, who was more up-to-date on California net-metering policy than me.

  • It’s a waste of money. Paying people to do something they would have done anyways is not a good use of taxpayer, ratepayer or anyone’s money. If there aren’t very many cases (i.e., if the net to gross ratio is high) and it’s the inevitable outcome of an otherwise very successful program, we can live with it. But it’s important to know the net to gross ratio and not just focus on gross savings, so that we know how much money is going to waste.

Beyond this, there could be distributional implications. For example, if low-income consumers are less likely to participate in energy efficiency programs, then their rates are being used to subsidize higher income customers to do something they would have done anyway. That’s not just inefficient use of funds, it’s unfair to the neediest in our society.

So, we need to continue to sharpen our pencils and apply state-of-the-art measurement techniques to energy efficiency programs. It’s critical to know how much bang we get for every energy efficiency buck, especially in a time when the national political winds are likely to push back against government energy efficiency programs. The rest of the world is watching!

Catherine Wolfram View All

Catherine Wolfram is Associate Dean for Academic Affairs and the Cora Jane Flood Professor of Business Administration at the Haas School of Business, University of California, Berkeley. ​She is the Program Director of the National Bureau of Economic Research's Environment and Energy Economics Program, Faculty Director of The E2e Project, a research organization focused on energy efficiency and a research affiliate at the Energy Institute at Haas. She is also an affiliated faculty member of in the Agriculture and Resource Economics department and the Energy and Resources Group at Berkeley.

Wolfram has published extensively on the economics of energy markets. Her work has analyzed rural electrification programs in the developing world, energy efficiency programs in the US, the effects of environmental regulation on energy markets and the impact of privatization and restructuring in the US and UK. She is currently implementing several randomized controlled trials to evaluate energy programs in the U.S., Ghana, and Kenya.

She received a PhD in Economics from MIT in 1996 and an AB from Harvard in 1989. Before joining the faculty at UC Berkeley, she was an Assistant Professor of Economics at Harvard.

19 thoughts on “What Counts as Success in Energy Efficiency Programs? Leave a comment

  1. Another approach, at least in buildings with a split incentive problem, is to avoid the complexity of using incentives, and just use the value of the savings, like has happened in Seattle, using MEETS.
    You can find the report here:

    All this hand wringing over what is “cost effective” and who is a “free rider” has been traditionally necessary, because we are incentivizing EE by using other ratepayers’ money. We need standards to properly spend other people’s money.

    Things get a lot simpler when we don’t use other people’s money. MEETS allows that and gets us much deeper in buildings, buy eliminating the split incentive issue.

  2. I welcome this article. It skirts the edges of the “Cheshire Cat” effect but is no less welcome for that. Like the Cheshire Cat, all that we see (measure) is the actual use of energy (the smile) from which we infer the savings (the body). I am all in favor of randomized control trials but I assume that these are prospective. In the meantime, while we await the results, what is badly needed are retrospective statistical analyses of participant and non-participants in energy efficiency programs. What we have now is an unholy alliance of utilities, consultants and advocacy groups. The Estimation, Measurement and Verification (EM&V) consultants confirm that the estimated savings of participants actually occurred and the utilities are made whole for “lost revenue”. (The advocacy groups get to claim success which ensures continued support from the various regulators.) The losers are the customers who pay the bill for the whole charade and taxpayers, in cases where the programs are funded, in whole or part, by government. (Of course, in general, they are the same people.) The “savings” from energy efficiency cannot be measured but only inferred from “baseline” consumption. Part of the problem is undoubtedly consumer behavior in relation to idealized engineering assumptions about specific technologies. An appropriate retrospective statistical analysis would pick up such behavioral effects. I have yet to find any retrospective studies and there are thousands of so-called EM&V Reports. I was an advocate of energy efficiency well before it became popular and still think it has a big role to play but we need realistic estimates of what can be achieved, not rubber-stamping of “savings” based on “baseline” assumptions.

  3. Studying what people claim they “might” do without a rebate is a very inexact science. We have many examples when rebates are ended what happens. For instance, when electric vehicle rebates ended recently in Georgia, electric vehicle sales dropped from over 4,000 per year to 300. So I understand the intent, but to use “intent” surveys for public policy is dubious. One of the other failures of this kind of research is timing. When surveyed if gasoline prices are high or there was recent electricity rate increase, it will provoke an answer of actions without incentives. But a year later, when the decision is actually to be implemented, the disposable income of the consumer, or memory of the increase, or just the extra time to take the extra effort, the actual acquisition of an energy efficient appliance is shunted aside. So intellectually, I enjoyed the article, but as a tool for guiding public policy, I am very dubious. – Scott Sklar, Adjunct Professor, The George Washington University (GWU)

  4. The measurement problem you highlight needs to identify and differentiate between behavioral and infrastructure (appliances and building design) efficiency measures. What you’ve focused on in this post is infrastructure (e.g. water heaters) which actually incorporate both design and behavioral issues. One of the problems complicating your post is the time period selected for measurement. Assuming that behavior does not change, the savings for a more efficient water heater over its expected life is basically an engineering calculation. While expected lifetimes can vary, over large populations estimates should be reasonable. However, accounting for changes in behavior even for a water heater are very difficult because usage can be impacted by electricity rate design and price, designs that provide more reliability, cultural norms, and family demographics. Doesn’t your post really focus on the behavioral issues?

  5. The energy efficiency of infrastructure is relevant to society only as it relates to its climate and toxic emissions. Purchase of new infrastructure, whether a water heater, car, house, or factory, is where policy could have the greatest leverage on GHG emissions, because that new infrastructure will be emitting throughout its useful life. The party responsible for deciding to add a long-term emitter should be required to pay the social costs of the expected lifetime emissions from that infrastructure. Such fees could be returned as rebates for similar products with zero or significantly lower emissions (feebates).

    Policymakers would have to decide which economist’s estimate of social costs to adopt, but even $40/MTCO2e on future emissions would for example imply a fee of about $3000 on a new 20 mph vehicle with a 150,000 mile lifetime (or ~$28,000 by the SCAR analysis). Then the initial buyer would feel the impact of his/her decision, instead of a low-income family paying the social costs of their gasoline when they can’t afford a more efficient car. Such fees only make sense for products with alternatives that have significantly lower emissions–don’t penalize a farmer who needs a new pickup now. But there are five 200-mile EV models in the $35,000 range that are, or will soon be, appropriate ZEV alternatives. Or heat pump water heaters or space heating vs burning gas, or a net-zero building vs one that barely meets code. (Yes, emissions are often a function of the expected grid emissions, but that can be reasonably estimated.)

  6. In California we will by design face a carbon profile of generation with continued decreasing marginal emissions. The current 50% RPS, coupled with the behind the meter carbon free generation will reduce the carbon abated by EE significantly over the coming decade or so. The recently proposed 100% RPS would obviously remove marginal carbon completely from the electric sector, but for potential imports and behind the meter non-carbon free CHP. If this was previously a complex issue, it’s about to change even more.

    • The next target becomes electrifying other carbon emitting sources in buildings (gas) and transportation (oil). Reducing electricity load frees up GHG-free generation capacity to serve these other markets at less cost.

  7. Catherine,

    One of the issues we see in the commercial lighting space (which has heavily benefited from energy efficiency rebates) is the repeated dosing that rebates create for a customer.

    The idea of rebates is that you bring them in when a technology is immature to help offset the economics then withdraw them as the underlying economics become sustainable. However, customers get hooked on the concept of rebates and adjust the payback expectations downward accordingly. The here today / gone tomorrow nature of many of the prescriptive programs makes this worse as much like a teenager waiting for game to go “on sale” at Bestbuy they wait out the utilities until they come to a budgetary cycle and rebates suddenly get more attractive.

    My sense is that there are a number of problems in this space that mostly relate to principle-agent problems between the PUC – Utilities – Distribution – OEMs. These are inherently difficult to solve but given the amount of money being applied it seems a shame that there is not more of an effort to do so.


  8. Nice explanation–sent this to my wife who was just asking about this issue.

    BTW, on D&D, it really took off in the mid-70s, and I’m not sure I ever met anyone younger than 14 who played.

  9. What about commercial building owners and industry and even our utility companies. They also consume a lot of natural gas, and vent into the atmosphere a lot of hot wasted energy and CO2 and water. With the technology of Condensing Flue Gas Heat Recovery they can be bringing up the efficiency of their boilers to well over 90%. Instead of hot exhaust, cool exhaust will be vented into the atmosphere.
    For every 1 million Btu’s of heat energy that is recovered from this combusted natural gas exhaust and is utilized, 117 lbs of CO2 will not be put into the atmosphere. How many light bulbs have to be changed hourly to keep up with this number?
    In every 1 million Btu’s of natural gas that is combusted are 5 gallons of recoverable distilled water. This is something that California should be paying attention to. They are allowing all this water to be vented into the jet streams where it will come down as rain in the Midwest or on the east coast.

%d bloggers like this: