Are the Benefits to the Weatherization Assistance Program’s Energy Efficiency Investments Four Times the Costs?
Today’s post is co-authored by Michael Greenstone (University of Chicago) and Catherine Wolfram (UC Berkeley) Faculty Directors of The E2e Project.
As featured in the New York Times For Government That Works, Call In the Auditors
The urgency of climate change demands solutions that work. And so, a clear-eyed assessment of how well existing policies and programs are performing is critical, no matter the results. Using rigorous evidence to inform policy design and implementation is the only way to ensure that we effectively confront climate change. The need for objective evidence is particularly important in the case of energy efficiency investments because they play a central role in virtually every single climate change mitigation plan.
Recently, we conducted a first-of-its-kind, randomized controlled trial of the nation’s largest residential energy efficiency program, the federal Weatherization Assistance Program (WAP). The trial consisted of a sample of more than 30,000 WAP-eligible households in the state of Michigan. Our research revealed that investments in residential energy efficiency upgrades among weatherized households in our study cost about double what these households will save on their energy bills (efficiency retrofits cost about $4,600 per household and estimated energy savings are only $2,400). You can read more about our results here.
These findings contradicted commonly held beliefs about the benefits of energy efficiency investments and generated a heated debate. Our previous blog post responded to some of the key concerns.
The Department of Energy (DOE) has since released 36 documents (here and here), totaling 4,446 pages, evaluating WAP. The boldly stated conclusion of the evaluation is that the benefits of the investments are four times the costs.
We have spent many hours poring over these opaque documents. Our judgment is that many of the DOE’s conclusions are based on dubious assumptions, invalid extrapolations, the invention of a new formula to measure benefits that does not produce meaningful results, and no effort to evaluate statistical significance. Using the DOE’s findings, we show below that costs exceed energy savings by a significant margin. We also document major problems with the valuation of non-energy benefits, which comprise the vast majority of estimated program benefits.
Overall, the poor quality of the DOE’s analysis fails to provide a credible basis for the conclusion that the benefits of energy efficiency investments conducted under WAP substantially outweigh the costs. This blog summarizes our assessment of the DOE’s analysis for a general audience. We provide a more technical discussion, using the important example of benefits relating to thermal stress, here.
I. Energy Efficiency Investment Costs Exceed Energy Savings by a Significant Margin.
The typical approach to assessing energy efficiency investments is to test whether the energy savings exceed upfront costs. Using standard economic assumptions, our study and the DOE study agree that costs incurred under the Weatherization Assistance Program significantly exceed energy savings. Here we focus on the ARRA period, but the basic conclusions from outside of ARRA are similar (we explain this here).
What does the DOE find? They collect energy consumption data from approximately 16,000 households. They then compare energy consumption at weatherized households before and after weatherization with energy consumption in a comparison group. Using this basic approach, the DOE finds that the average annual savings for households is $223 per year during the ARRA period. This is the primary result on energy savings.
For most, if not all, households, the decision to pursue weatherization assistance is purposeful. Households may choose to invest the effort of applying for weatherization following – or in anticipation of – a change in energy consumption or their ability to pay energy bills. In other words, there could be important differences in energy consumption patterns across groups of households receiving weatherization in different time periods. The report provides little evidence that the comparison group provides a credible estimate of the energy consumption patterns that would have been observed among weatherized households had they not received weatherization retrofits. The consequence is that we cannot know whether the measured differences in energy consumption are due to the energy efficiency investments or the pre-existing differences between weatherized households and the comparison group.
In contrast, our study is based on a randomized controlled trial (RCT) precisely because we wanted to improve the quality of evidence on the returns to energy efficiency investments. The power of RCTs is that they deliver the true effects, obviating concerns about differences in the types of households that get weatherized and the timing of those decisions. Indeed, this is precisely why society has relied on RCTs to test the efficacy of new drugs for decades and is increasingly using them to evaluate social policy.
Setting aside concerns about the estimation strategy, it is straightforward to convert the estimate of average annual energy savings reported in the evaluation, $223 per year, into a present value of energy savings over the lifetimes of the measures installed by making several assumptions. The following table describes the present value of lifetime savings under different assumptions about the discount rate and the lifespans of the measures. Notably, the table follows the standard assumption in the economics literature and assumes energy prices will remain constant, after adjustment for inflation. It also generously assumes that there is no depreciation in the savings created by the measures.
| Net Present Value of Energy Savings from DOE Study of ARRA Period | |||
| Time Horizon | Discount Rate | ||
| 3 Percent | 6 Percent | 10 Percent | |
| 10 years | $2,003 | $1,641 | $1,370 |
| 16 years | $3,028 | $2,254 | $1,745 |
| 20 years | $3,646 | $2,558 | $1,899 |
| Note: The table reports the net present value of energy savings (in dollars) implied by the DOE’s household-level energy savings estimates ($223) using a range of discount rates and assumed time horizons. Reductions in energy bills associated with the estimates are assumed to accrue over the life of the measure. | |||
These “present value” energy savings can be compared against the average cost per weatherized household. The DOE reports a wide variety of costs, and it is difficult to judge which should be compared to the savings. The most straightforward is the average total expenditure per weatherized household: $6,812 during the ARRA period. The DOE reports incurring costs per household of $5,926. When comparing household benefits and costs, the DOE evaluation uses a much lower $3,745 per household, although it is unclear what cost components are omitted to arrive at this low number.
Using a 6 percent discount rate that seems a reasonable approximation of households’ borrowing costs, a 16-year time horizon (the average projected lifespan of efficiency measures in our data), and assuming no depreciation in annual energy savings (likely generous), the DOE estimate of annual energy savings implies a present value of $2,254 per household. This is less than half of the DOE expenditures per household. It also falls significantly below the lowest reported costs of $3,777.
To compute the average discounted value of energy savings, the DOE uses a longer time horizon, a lower discount rate, and assumes that energy prices will increase in real terms over time – all assumptions that make the savings look larger. These assumptions imply a higher present value of $3,190 per household. This higher value notwithstanding, WAP program costs still exceed estimated savings by a significant margin.
The conclusion that the program is remarkably cost effective cannot be justified on energy savings alone. The DOE’s claims rely heavily on the possibility of substantial non-energy benefits. We explore the credibility of these estimated benefits next.
II. Unreliable Approaches to Exploring Whether there are Non-Energy Benefits.
The DOE evaluation reports a program-wide benefit to cost ratio of 4 to 1. To arrive at this conclusion, the report points to a long list of “non-energy” benefits. They include health-related benefits such as reductions in thermal stress and asthma (valued at $6,870 per household) and improved productivity at home and at work due to improved sleep (valued at $3,142 per household), safety-related benefits such as reduced fire risk and reduced carbon monoxide levels (valued at $985 per household), and other benefits such as reduced reliance on food and energy assistance programs (valued at $1,641 per household). The rationale for relating energy efficiency investments to some of these outcomes (e.g., improved productivity due to better sleep) is not well articulated, but nevertheless we examined the basis for the claims.
A close look at these claimed benefits uncovers significant problems with the methods used and assumptions invoked. We highlight three general concerns here.
- Non-energy outcomes are not directly measured. And the path from imputed impacts to benefits estimates is generally dubious.
In contrast to energy consumption, which is measured directly using household energy bills, most of the non-energy benefit estimates are based on survey responses from program participants. Connecting the dots between survey responses and non-energy benefits involves many steps. The resulting benefit values are extraordinarily sensitive to many key assumptions, several of which are not well explained or substantiated.
The significant benefits attributed to reduced hospitalization rates after weatherization provide one example. DOE surveyors asked a select set of households the following question: “In the past 12 months, has anyone in the household needed medical attention because your home was too cold (hot)?” Using a formula they invented that has never appeared in any textbook (more on this below), the DOE deduced from the survey responses that weatherization assistance reduced the share of households needing medical attention by 1.4 percent.
The path from the response to this survey question to monetized benefits is long and winding. The nearly $5,000 per household of mortality reduction benefits are particularly illuminating. Since no data on mortality was collected, the path to claimed reductions in death begins with the above question about medical attention. Of course, not all instances of medical attention lead to death, so the DOE analysis assumes that some proportion of these instances lead to hospitalizations, doctor visits, or emergency room visits. The next step in this circuitous path is to assume that some fraction of the assumed hospitalizations lead to death. Consequently, responses to a survey question about “medical attention” turn into $5,000 worth of benefits per weatherized household.
This approach to calculating benefits is applied throughout the analysis of non-energy benefits. For example, responses to questions about sleepless nights are converted into productivity benefits that exceed $3,000 per household. The general point is that the failure to directly measure the outcomes of interest leads down a long and winding road that seems far removed from solid footing.
There are three other noteworthy points about the imputation of health-related benefits
- It is natural to assume that the starting point of benefits associated with reduced thermal stress would be a substantial change in indoor temperatures, making it warmer in the winter and cooler in the summer.In the only report where indoor temperature is directly measured, DOE finds that average indoor temperature increased by 0.3 F in weatherized homes as compared to a control group. This very small change calls into question the basis for the claimed health effects due to reductions in thermal stress.
- Notably, in cases where DOE analysts were able to directly measure outcomes with a documented connection to health, these data were not systematically used in the benefits valuation exercise. For example, DOE researchers conducted a field study of indoor air quality parameters in weatherized homes, including carbon monoxide (CO), radon, formaldehyde, and temperature. This study found no significant changes in carbon monoxide following weatherization. Yet, the claimed benefits include reductions in CO poisoning.
- In contrast, weatherization was found to increase radon and formaldehyde levels. In the accounting of costs and benefits, however, no effort was made to quantify the potential health costs from the increase in these potential health risks. Thus, at least in the case of the direct indicators of air quality, the DOE study assigns benefits in cases when direct measures do not change and fails to evaluate costs when the indicators worsened. This selective accounting is a cause for concern.
- The invention of a new statistical technique to estimate benefits.
A very standard approach to estimating the impacts of an intervention (such as weatherization assistance) on an outcome, absent a randomized controlled trial, involves taking the difference in the pre-post change in outcomes in the treated group and then subtracting the pre-post difference in a comparison group. This is the basic approach that the DOE used to calculate energy savings summarized above. If outcomes observed in the comparison group provide a good proxy of what would have been observed in the treated group absent the treatment, this approach can provide a credible estimate of the effect of a program.
To estimate impacts on key non-energy outcomes (such as health and productivity), the DOE introduces a very non-standard and non-intuitive approach. Specifically, DOE analysts compute the average of the observed pre-to-post change in outcomes in the treated group and the observed difference in outcomes between the control group and the treatment group in the pre-period. This approach takes an average of two differences, rather than subtracting one from the other. In doing so, it undermines the entire rationale for the difference-in-difference approach (namely to control for the effect of potentially confounding changes over time that are not caused by the weatherization).
To the best of our knowledge, this approach has never been used in any textbook or research paper previously. Additionally, DOE researchers provide no rationale. While the DOE’s analysis of energy savings requires some non-trivial assumptions that are not required for randomized controlled trials, it is at least a recognizable approach to estimating the impacts of the program. In contrast, the approach to estimating non-energy benefits is unrecognizable, and we believe the resulting estimates have no meaningful interpretation.
- No attempt to separate a signal from the noise.
People’s lives change from day to day for many reasons. For example, the number of work days missed, or prescriptions filled, can vary over time for a variety of reasons most of which cannot be measured. The standard practice is to test whether a program’s impacts are statistically significant or just due to random chance. The evaluation does not follow this important convention. Indeed, the little information that is provided in footnotes about the variability of outcomes in treatment and comparison groups suggests that many of the key differences documented could well be due to random chance versus a causal impact of weatherization.
In summary, non-energy benefits comprise a very significant share of the estimated benefits of the energy efficiency investments that were made through the WAP program. The process of assigning dollar values to non-energy costs and benefits is inevitably complicated. Estimated values can be highly sensitive to underlying assumptions and measurement choices.
We find the DOE’s evaluation of non-energy benefits is flawed in several respects. While it is certainly possible that the people who received weatherization improvements benefitted in other ways, our judgment is that the DOE analysis fails to provide evidence that can credibly confirm or contradict this possibility.
Conclusion
The DOE has gone to great lengths to collect data on what happened to households after they received weatherization assistance. For this, they should be commended. However, the data were not analyzed in transparent and standard ways; this undermines the credibility of the results.
A seriousness of purpose in achieving policy goals requires finding out what works. A commitment to independent evaluation is the critical first step, and that requires high-level political support. The Obama Administration has supported evidence-based policy making in other realms; there are great opportunities to extend these efforts.
Categories
Energy Institute Blog 
Roger appears to assume that this WAP evaluation did not include all of the costs, which appropriately should be compared with all of the benefits. The nature of low-income WAP is that all of the costs ARE provided as program support, with no consumer cash contribution. Unless this particular WAP did not pay the full retrofit costs (highly unusual) it IS appropriate to compare these costs to the sum of the energy and non-energy benefits that result.
The WSU evaluation of the Washington State Low-Income Weatherization program (another similar Stimulus Act funded program) did exactly that. I referenced that in my earlier post. This format does not allow pasting in of tables, so I will show only the “mid” case results, not the low and high. What is important, however, is that the energy benefit is LOWER than the program cost, but the TOTAL BENEFIT is much higher than the total cost. Excluding more than half of the benefits, as the Chicago study did, can be expected to produce the inferior results that study reported.
The full report can be reviewed at:
Click to access HIP-Weatherization-2010-Final-Wx_Eval.pdf
Present Value Mid
Emissions Benefit $380
Economic Benefit $1,310
Utility Benefit $340
Participant Benefit $2,270
Total Non-Energy $4,300
Energy Benefit $4,840
Total Benefit $9,140
Total Cost $6,070
Benefit-Cost Ratio 1.5
While the net present value expansion of the benefits is appropriate why is there no attempt to also address the opportunity cost associated with the actual cost of (investment in) efficiency measures. Expanding the present value of benefits without also simultaneously expanding the costs creates an asymmetric comparison. Customer and government provided tax dollars that fund efficiency measures could be used to fund other potentially more productive investments in retirement, health care, education or other lifestyle areas. There appears to be an implicit assumption that these efficiency dollars are free.
Despite the discord, both WAP (national, just released, and regional reports) and E2E reports find that the savings is about 12-20% of energy usage on average. Part of the (buried) big story is that RCT diff-in-diff and the pre-post bill (weather-adjusted, with or without a control group) comes to the same savings conclusion.
The remainder is the details of discount rates, what costs to count (include or don’t include health and safety), what benefits to count and how (health, other pollutants, CO2). These are important details which can give a benefit cost ratio of 0.25 (E2E) to 1.5 or more (WAP). And these details should be worked out: arguments presented, viewpoints considered, and ultimately the funders (gov’t, Congress) decide what yardstick to use.
But since this argument is over 30 years old (http://www.jstor.org/stable/41322467?seq=1#page_scan_tab_contents), my hope for a negotiated solution is small.
Transparency and side by side comparisons to show the sensitivity of the results would be very useful here.
Technical questions. First, I’m not sure I understand this: “Specifically, DOE analysts compute the average of the observed pre-to-post change in outcomes in the treated group and the observed difference in outcomes between the control group and the treatment group in the pre-period. This approach takes an average of two differences, rather than subtracting one from the other.” Is this written out as a formula somewhere?
Second, adding weatherization to a household is going to provide a combination of benefits. If the occupants do NOT adjust their thermostats, they will save money. But there will be approximately zero impact on indoor temperature, and therefore all the claimed benefits of better temperature control will be zero. Or, the occupants might choose to adjust their thermostats so that they are spending the same amount as before, and getting better indoor temperature control. In this case, they will get the non-energy benefits.
But the analysis seems to be ADDING the two benefits, instead of ORing them. No household can get full benefits of both kinds. What am I missing here?
Re: your 2nd question — are you saying that it’s not possible to get both cost savings and comfort benefits at the same time?
I am saying that the total benefit is a convex combination of the two. You can go all the way for cost savings, with zero comfort, by keeping your thermostat at the same temperature. You can go all the way to max comfort, with zero cost savings. Or, you can choose a location somewhere in between, by adjusting your thermostat an intermediate amount. The analysis seems to be ignoring this.
Actually, it’s a little better than this. Even if you keep the thermostat at the same level to “maximize energy savings” the work done on reducing air leaks will make the place more comfortable. The thermostat is normally on an interior wall, and the insulation and air leakage work will make it more comfortable at the perimeter, even with the same overall indoor air temperature.
One night in drafty motel room at the Oregon coast in January will remind you of this. It’s cozy by the fire, but miserable by the windor.
To answer SW, the clear conclusion from the piece is that Randomized Controlled Trials are needed to assess benefits. A second conclusion is that health benefit analysis is often pulled out of thin air. We are 20 years into these projects. Why isn’t there better data and better analysis?
Randomized Controlled Trials are not that hard to do in programs with big subsidies. Since the budgets are limited anyway, you add one more step to the process for selecting who is going to receive the subsidy.
I agree with art2science,
We have been doing these assessments since the 1980s – and doing them badly! Given how much money is being spend on energy efficiency programs It’s high time that they be done in a more disciplined manner. Yes, they should take account of external benefits (and also costs) but let’s use defensible quantitative estimates of these externalities, not just speculative notions.
And, please, leave out the job creation argument. How much are those jobs costing tax payers? There are many ways to create jobs through investments in projects that are defensible based on quantifiable, internalized costs and benefits.
I could not agree more with the premise that our energy efficiency programs have to be as effective as possible, and that we need to perform very careful analysis. My problems with the Chicago Berkeley study and with this blog entry are as follows:
First, the authors continue to suggest that their very limited study of a low income energy efficiency program in one state tells us something transferable about the overall effectiveness of energy efficiency programs. While this assertion may have headline value, I fear that it does not add significantly to our understanding of what’s going on. Second, while the blog entry disparages the Department of Energy’s effort to calculate non-energy benefits, the authors continue to ignore such benefits themselves. This reflects either a misunderstanding about the nature of low income energy-efficiency programs or a failure to acknowledge that those differences exist.
Third, the authors criticize the Department of Energy’s analysis by raising a concern that there might be a differential between energy needs of some of some households before undertaking weatherization improvements and their energy needs afterward. The problem is, the authors have done nothing to demonstrate that their concern is a valid one. If not prepared to undertake that analysis themselves, can they cite another study to make that point?
Perhaps my biggest misgiving is that while the authors attempt to throw a wrench into the energy efficiency policy machine, they don’t offer a better way. Are they suggesting that no energy-efficiency effort can be cost-effective and that therefore the programs should be abandoned? Are they suggesting that there are specific programs that work well and should be replicated in other places, or is this just all about being critical and raising a fuss? If the answer is that there is much more work to do before being able to provide those answers, then perhaps it makes sense to stop making such a big deal out of what are at best preliminary findings.
I think the answer is that not all energy efficiency programs are good ones to pursue. Unfortunately our policy makers seem to have never seen a program that they didn’t fall love with. Look at the CFL debacle in California as one example. We should demand that rigorous analytic tools be applied to the choosing the right measures. Right now that’s clearly not the case.
The other point of this blog post is to show how a distorted analysis looks like its been constructed to justify this continued love affair with every program rather than really cutting through which ones are good ones to pursue.
In California, we make this even worse by approving a portfolio of programs that on the whole must exceed the 1:1 benefit cost ratio. The obvious mistake here that no one seems to raise is that means that 50% of the measures must be below the 1:1 ratio! Why are we wasting our resources on these decisions. (Especially since there are many opportunities above 1:1 that are underfunded.)
Your results do not surprise me. The cost benefit calculations used to justify energy efficiency were flawed back in the mid-1980s and do not appear to be any better today.
One big problem is ignoring price elasticity which gives rise to the “snapback effect” i.e, energy efficiency effectively reduces the unit cost of the energy product (e.g., space heating or cooling) which gives the consumer an incentive to use more of the product. A second problem is using a low discount rate – often the riskless interest rate – to increase the present value of the downstream benefits. Doing so ignores the fact that the future benefit stream is a function of future energy prices which are uncertain, therefore risky. The risk associated with oil and natural gas prices justifies a real discount rate of at least 5 percent.
One could legitimately add the value of greenhouse gas reduction, which would pull the marginal projects across the goal line but that would only justify a small fraction of the projects being funded today.
Energy efficiency deserves to be seen for what it is: a sacred cow that deserves to be slaughtered (however, I don’t eat beef).
Have you considered one or more bodyguards? 🙂
DOE was correct to include a broad range of non-energy benefits in the analysis. The omission of these by the Chicago paper is a fundamental flaw, and the primary reason it has been largely disregarded by the efficiency sciences community.
One must consider the entire layer cake of benefits when evaluating investments. A narrow approach produces an incomplete and biased result. There are utility system benefits, participant benefits, and societal benefits. It’s important to address overlap, but not to exclude benefits. See: Recognizing the Full Value of Energy Efficiency, Regulatory Assistance Project, 2013. http://www.raponline.org/document/download/id/6739
The most dramatic example of this was from the New Zealand home weatherization evaluation. As a stimulus project, New Zealand decided to retrofit ALL low-income housing (180,000 units). The evaluation showed that the energy savings were barely cost-effective, but the health benefits were much larger. The occupants of the treated homes showed a 43% reduction in hospital admissions due to respiratory ailments, a 39% reduction in days lost at work, and a 23% reduction in days lost at school. These are immense economic benefits that must be properly accounted for, and which the Chicago report ignored. http://www.healthyhousing.org.nz/research/current-research/evaluation-of-warm-up-new-zealand-heat-smart/
Perhaps most important, however, is the fact that these stimulus programs had as their objective re-employment of the building trades at least as much as the objective of cost-effective energy savings. These employment benefits — keeping people from sliding into destitution — are significant and relevant to the context of the recovery act.
For a reasonably complete analysis of the cost-effectiveness of residential weatherization of low-income households, it’s more appropriate to look at an experience energy program evaluator’s perspective. WSU’s evaluation of the Washington State program during the stimulus era is a good example of this.
Click to access HIP-Weatherization-2012-Evaluation-Executive-Summary.pdf
While including other benefits is useful, including the WRONG benefits in inexcusable. For example, increasing evidence shows that asthma is not tied directly to air quality, but rather to childhood conditions. It would be interesting to conduct the NZ analysis–DOE should have been able to do this rather than using the survey method. (Interestingly, the NZ also shows a 4:1 ratio–was the DOE study constructed to arrive at that result?)
Also, using employment benefits to justify a program that is not otherwise cost-effective is not appropriate. It ignores the question of whether making a more cost-effective investment could have both generated the same added employment AND added net economic benefits. Otherwise we fall into the absurd trap that Keynes mistakenly laid out in The General Theory: “If the Treasury were to fill old bottles with banknotes, bury them at suitable depths in disused coalmines which are then filled up to the surface with town rubbish, and leave it to private enterprise on well-tried principles of laissez-faire to dig the notes up again (the right to do so being obtained, of course, by tendering for leases of the note-bearing territory), there need be no more unemployment and, with the help of the repercussions, the real income of the community, and its capital wealth also, would probably become a good deal greater than it actually is. It would, indeed, be more sensible to build houses and the like; but if there are political and practical difficulties in the way of this, the above would be better than nothing.”
Maybe we should focus on a comprehensive evaluation of recovery-act weatherization results and the 2008 program, pre-ARRA. One was published by Oak Ridge National Labs, covering all 50 states. It’s consists of 21 separate reports, accessible at:
http://weatherization.ornl.gov/evaluation_nr.shtml
There is a high-level numerical summary for both the 2008 program (pre-ARRA) and the 2010 program (ARRA) at:
Click to access WAP_NationalEvaluation_WxWorks_v14_blue_8%205%2015.pdf
The total cost per household was $6,812; the portion of this attributable to energy measures was $3,545. On an energy-only basis (ratio of energy-related costs to energy-related benefits), the benefit:cost ratio was 0.9.
On a total cost and total benefit basis, the results were quite different. The total benefits were $13,167 per house, producing a total benefit:cost ratio of 1.93. The non-energy benefits include health and safety, avoided criteria emissions, avoided carbon pollution, reduced water and wastewater costs, and macroeconomic benefits.
By comparison, the evaluation of the 2008 program carried a total benefit:cost ratio of 4.1; the “urgency” of the recovery act deployment obviously caused a loss of leverage of the investment. Part of this was higher labor costs, and workers needed retraining; part was less cream-skimming, because more customers were eligible; part was undoubtedly overhead costs of a hasty deployment.
BUT, the evaluation shows that the program supported 28,000 jobs, up from 8,500 jobs for the 2008 pre-ARRA program. Since a major purpose of the ARRA weatherization program was to employ building trades workers left unemployed by the housing collapse, this is a very important evaluation result.