Do Residential Energy Efficiency Investments Deliver?
Today’s post is co-authored by Michael Greenstone (University of Chicago) and Catherine Wolfram
We recently released a paper presenting the findings of a first-of-its kind, randomized controlled evaluation of the returns to some common residential energy efficiency investments. The study’ s context is the nation’s largest residential energy efficiency program, the Weatherization Assistance Program (WAP). You can read media coverage of the paper here, here, here, and here.
For those who haven’t read about the paper, between 2011 and 2014, we administered a randomized controlled trial (RCT)—considered the gold standard in evidence—on a sample of more than 30,000 WAP-eligible households in the state of Michigan in order to shed some light on a critical question: Do investments in important residential energy efficiency measures (improved insulation, air sealing, weather-stripping, window replacement, furnace replacement, etc.) deliver the energy savings they promise?
The research revealed five main results: (1) The energy efficiency measures undertaken by households in the study reduced their energy consumption by between 10 and 20 percent on average; (2) However, these savings were just 39 percent of the average savings predicted by engineering models; (3) There is no evidence that the shortfall in savings is the result of rebound—households did not turn up their thermostats after the investments were made; (4) While the investments cost roughly $4,580 on average, our best estimate of the energy savings was about half of these costs[1]; and (5) The costs also greatly exceeded the benefits when the monetary value of pollution reductions are added to the energy savings to calculate benefits. While the WAP program has a number of goals, when measured by the energy savings and emissions benefits, these efficiency upgrades were not a good investment.
The urgency of the climate challenge means that it is critical to identify cost-effective strategies that will deliver real greenhouse gas emissions reductions. Energy efficiency is a crucial component of most climate change mitigation plans, underscoring the importance of developing a body of credible evidence on the real-world—versus projected—returns on energy efficiency investments in the residential sector and beyond.
Such a process will undoubtedly uncover some gems, but in some instances it will also be necessary to update our beliefs. When seemingly inconvenient evidence comes to light that challenges our beliefs—as we have uncovered with this analysis—that data should not be undermined and ignored. Instead, it should be used to inform our strategy to confront climate change. The magnitude of the climate challenge requires that we ruthlessly pursue the most cost effective mitigation options.
Our paper has generated some strong reactions and important questions, some the result of misconceptions about what exactly we evaluated and how the evaluation was conducted. In the remainder of this blog, we respond to the most common criticisms of our study and its findings.
* * *
Reaction 1: This is just one study and scores of other studies have opposite findings.
Some critics have cited prior evaluations showing that residential energy efficiency programs are good investments and that our study is an anomaly. Many of these evaluations, however, are based on savings projections that- as we found- can significantly overestimate the savings when applied in the real world. Other studies use real- world data, but analyze these data using methods that can confuse the effects of energy efficiency improvements with other factors that drive changes in energy consumption.
Our study is different. It represents a first-of-its-kind evaluation using a randomized controlled trial, the gold standard for rigorous evaluation. Society routinely relies on this methodology to assess the efficacy of new drugs, treatments, and other interventions. This approach is increasingly used in the social sciences, including criminology, education, development economics, and energy economics. In many instances, the application of randomized control trials has changed the conventional wisdom. Our application of this approach to residential energy efficiency measures is therefore an important departure from, and improvement upon, previous analyses.
Reaction 2: The study unfairly paints WAP as an ineffective program.
WAP has multiple goals and improving the living standards of its recipients is clearly a central and worthy one. Our study does not claim to provide a comprehensive evaluation of WAP, nor would it be appropriate to do so.
Rather, the study’s purpose is to measure the real-world energy savings resulting from WAP-funded energy efficiency improvements. We then compare them to both the investment costs and the projected energy savings generated from detailed energy audits.
In interpreting the results, it is important to bear in mind that for a measure to be implemented under WAP, federal regulations require that it pass a cost-benefit analysis—that is, the projected cumulative energy savings must be greater than the investment costs. This cost-benefit analysis is based on an in-home energy audit conducted using an engineering model, in this case the National Energy Audit Tool (NEAT).
For the households we studied, NEAT-driven audits projected that the WAP measures would reduce annual energy consumption by 43.7 million British thermal units (MMBtu). Yet, when we observed the energy bills of households that received WAP measures, the actual energy savings were just 17.2 MMBtu. In other words, the model systematically over-predicted energy savings by a factor of 2.5.
This is an important finding. The investments in efficiency in our study underperformed relative to projected values and in a way that the program was expressly designed to avoid. Homeowners, program managers, and taxpayers only received 39 percent of the projected savings. According to the Department of Energy, the NEAT model is used by approximately 700 state and local Weatherization Assistance Program subgrantees in more than 30 states.
Broader program objectives notwithstanding, WAP is a compelling setting to learn about the returns to energy efficiency investments. WAP is the nation’s largest residential energy efficiency program. According to the Department of Energy, which administers the program, more than 7 million homes have participated in the program since its inception in 1976. If one is attempting to assess the performance of commonplace residential energy efficiency investments on a large scale, there may be no better option.
Reaction 3: The study’s calculations of costs and benefits are inaccurate.
Here again, it is important to note that we recognize WAP has benefits beyond saving energy. But, the intent of our study was focused solely on evaluating the energy-related (and associated emissions) costs and benefits. We never claim to evaluate the other benefits of these upgrades, as that is beyond the scope of our study. It is also important to note that, no matter how one decides to evaluate monetary costs and benefits, a central finding of our analysis remains unaffected: efficiency upgrades delivered just 39 percent of the energy savings they promised. It is therefore challenging to find a set of assumptions (e.g., about lifespans and discount rates) that would cause these efficiency investments to have energy savings and emissions benefits that exceed their costs.
To drill down a bit more, here are some of the criticisms of our calculation of the costs and benefits and our responses:
Costs
Some have argued that it is inappropriate to factor in costs that don’t directly lead to energy savings. As anyone who has done home repairs knows, once you start down the path to do something like lay new insulation, additional costs are necessarily incurred. For example, weatherization can reduce indoor air quality by tightly sealing a house, so additional costs may be required to maintain indoor air quality. Separating what’s required to lay the insulation from what’s completely separate is not easy. The average household in our sample received approximately $4,600 in energy efficiency upgrades, which includes roughly $800 in costs required to make installation of the weatherization measures safe and functional, such as wiring upgrades. Our judgment is that the most reasonable assumption is to include all of these installation and materials costs. It is worth noting, however, that if we take the polar opposite view and exclude all costs that do not directly result in energy savings, the average cost per household still significantly exceeds our central estimate energy savings.
Moreover, there are other costs associated with these retrofits that are not reflected in our cost-benefit comparison. For example, we do not include any program overhead or administrative costs. Nor do we account for the hassle and effort that households expend to implement a weatherization retrofit, even one with zero out of pocket costs. An earlier blog makes the point that these process costs can be large (we found that it cost $1,050 per weatherized household to encourage take up of these measures). Accounting for these additional expenses would of course widen the gap between costs and savings.
Benefits
We measure benefits by calculating the net present value of annual energy savings using a range of discount rates (3, 6, and 10 percent) and investment lifespans (10, 16, and 20 years). Our estimate of the benefits also includes an estimated upper bound on the benefits households derive from increased warmth (based on our analysis of “rebound” in demand for heat in the winter). In no case does the present value of energy savings reach parity with actual costs, even if we ignore the indirect efficiency-related improvements.
In calculating program benefits, we used real 2013 residential energy prices for electricity and natural gas in Michigan and assumed that these figures would increase at the rate of inflation over the lifetime of the investments. While some have criticized this as too conservative, it is standard to use current energy prices as a predictor of future energy prices.
Reaction 4: The results cannot be generalized because they only relate to one part of Michigan, to one program, and to one subset of the population.
We study a subset of low income households in Michigan undertaking a particular set of residential efficiency measures recommended by NEAT. However, minimizing the significance of our findings on account of this context ignores the ubiquity of the measures we analyze and of the reliance on audit tools like NEAT.
As noted above, the households in the sample we studied were subjected to the same measurement tool that is used by residential weatherization programs throughout the country to gauge which upgrades are the most cost effective; and all implemented measures had to pass the same cost-benefit analysis. The types of upgrades installed at the WAP households in our sample (e.g., furnace upgrades, improved insulation, and weather stripping) are commonplace for home retrofits for all income groups.
Drawing implications from a study is not an all-or-nothing proposition. For example, the results of a randomized controlled trial studying the effectiveness of a given drug or treatment on middle-aged men will in some instances tell us everything we need to know about its effectiveness on young women. Of course, in other instances, less decisive conclusions are warranted until further research is conducted.
Our study tells us that a common set of efficiency measures installed in the low-income households we studied in Michigan did not deliver the expected energy savings, and that investment costs significantly exceeded these savings. Given similarities between the setting we evaluate and other efficiency applications, these findings likely generalize to a broader set of residential efficiency investments. There is logic behind this implication, while also acknowledging the need for further experiments on the returns to energy efficiency investments in other contexts (Indeed, we have already begun to do them and, in at least one case, our preliminary results are qualitatively similar).
Reaction 5: The study period covered a time when the program experienced a significant increase in funding that led to poor results (e.g., inexperienced contractors).
The time period we studied included an unprecedented number of weatherizations because the American Recovery and Reinvestment Act (ARRA) increased the amount of money allocated to the program dramatically. As a result, some say new, inexperienced contractors were called in to do weatherizations and their work may not represent the norm.
To investigate this possibility, we compared savings at homes where the contractors were experienced to homes where the contractors weren’t experienced and found no difference in the average energy savings. Consequently, we find no evidence that inexperience during the time period played a role in explaining the lower-than-expected savings.
[1] This blog focuses on a subset of the numbers and results reported in the paper. Here we emphasize our preferred estimates from the randomized controlled trial that estimates average impacts for the subset of households whose participation in weatherization was the result of random assignment to our experimental intervention. These households are associated with somewhat lower average costs ($4,580) as compared to the larger sample of recipient households from whom we collected data for our quasi-experimental analysis ($5,150).
Categories
Energy Institute Blog 
I was struck by this passage: “While some have criticized this as too conservative, it is standard to use current energy prices as a predictor of future energy prices.” As a practicing energy economist for 30 years, I am unaware of this “standard.” It’s a simplistic approach to the problem, but the uncertainty of energy prices is one of the justifications for energy savings programs. The fixation on using single-point forecasts is at the root of this mistaken assumption. A more thoughtful approach would have used some sort of scenario or risk analysis to measure the value of the savings.
I studied building science with a focus on building simulation and energy policy with a focus on efficiency as a climate mitigation strategy. Through my studies, I have come to know Wolfram and Fowlie professionally and deeply respect their research. I believe in the need for more empirical efficiency program evaluations and recognize that the urgency of climate mitigation requires an unsentimental re-evaluation of our efficiency program priorities. I applaud the use of energy consumption data in this study and support the use of randomized controlled trials to evaluate programs (but statistical power issues are a very real concern, not all programs lend themselves to such design, and “gold standard” is a heavily freighted and disputed label for them). My intuition, experience studying building efficiency, and technical training all lead me to accept this study’s energy savings estimates.
So in theory I should be very pleased by the release of this study. However, I am distressed to see that the technically competent savings estimates have been mixed with a remarkably naive interpretation of the complex and politically determined mission of the Weatherization Assistance Program (i.e. that it is reasonable to assign 100% of its costs to efficiency and that its generosity of assistance is representative of efficiency programs in general), a straw man dismissal of existing program evaluations and their lessons (they have been telling us for years that models are consistently used to over-estimate savings), and a surprising lack of interest in the incentives for over-estimating savings that cost effectiveness criteria provide (the models just simulate physics, but the users of the models get paid based on their results – remember econ 101: incentives matter) to generate a very misleading round of press coverage.
In brief, the “gold standard” only applies to the energy savings measurements (there is room there for a critique of the sample quality, but the quality of this study’s execution was quite high). The cost effectiveness claims and generalization beyond low income programs that were the headline generating aspects of this project are backed only by some very heavily freighted assumptions that reasonable people could easily disagree about. For example, when you replace an old, malfunctioning furnace with a new and more efficient one, should 100% of the cost of the new furnace be allocated to the efficiency gains (what this study assumes) or should at least some of the cost be considered a necessary payment in the service of continued access to heat? Should a contractor intervening for an efficiency project in a home leave cancer-causing asbestos duct insulation in place because replacing it costs money but doesn’t save energy?
Such questions do not have objective and precise answers, but can lead to dramatically different views on what the underlying efficiency gains cost. In their widely publicized conclusions, the authors have taken a maximalist view: 100% of program costs are efficiency costs. What can not be disputed is that among efficiency programs, only WAP pays the full costs of furnaces and asbestos abatement. This is the primary reason efficiency experts are tying themselves in knots to underscore that WAP programs are not representative of all efficiency programs on costs.
It is one thing to have your research turned into a set of click bait articles with misleading headlines questioning the legitimacy of efficiency as a public investment or climate strategy by an intellectually lazy press. It is another thing entirely to cultivate that press with a policy brief and authoritative public statements that lean very heavily on assumptions about cost allocation and generalizability that are so debatable. To do this based on a study that has not even been peer reviewed and at the expense of a program that is providing valuable non-energy services to low income people is not up to the standards of academic practice that the authors should hold themselves to.