Evaluating Evaluations – Energy Efficiency in California
Last year, Governor Jerry Brown signed a law, Senate Bill 350, that sets out to double energy efficiency savings by 2030. Last week at the Democratic National Convention, Governor Brown focused his remarks on the importance of policies such as this to tackle climate change.
The precise energy efficiency targets haven’t been finalized, but they will be ambitious.
Meeting these targets will require an expansion of energy efficiency policymaking. Policymakers need to understand which programs work in energy efficiency and which don’t.
This is a daunting task. The California Public Utilities Commission’s (CPUC’s) energy efficiency efforts fund roughly 200 programs. The California Energy Commission (CEC) is regularly introducing new appliance and building standards. The evaluations of these activities are made public, but they can be hard to find and difficult to interpret. Additionally, policymakers may not have the time or training to critically assess the methodologies being used.
As a result, individual programs may not be getting enough scrutiny.
Many people working on energy efficiency may think the last thing we need is MORE evaluation. Energy efficiency is heavily evaluated.
I disagree. Today we have an opportunity to step up our game. We have access to more data and more rigorous evaluation techniques than ever before. It’s time for more evaluation, not less. In particular, it’s time to evaluate the evaluations.
To illustrate what I’m talking about, let’s look at an example from another heavily evaluated sector, criminal justice. The context is quite different, but the basic lessons are instructive.
In the 1980s many US states enacted stricter laws to reduce domestic violence. Rather than putting every offender in jail, courts began to mandate that offenders go through batterer intervention programs (BIPs). The initial evaluations of these programs found they were highly effective. These evaluations contributed to the justice system’s growing reliance on BIPs. In a 2009 report, the Family Violence Prevention Fund and US government’s National Institute of Justice estimated that between 1,500 and 2,500 such programs were operating.
As the cumulative number of evaluations grew, researchers began to undertake reviews that evaluated the evaluations, referred to as meta-analyses or systematic reviews. What they found was disappointing.
Many of the past evaluations that showed positive effects had methodological shortcomings. While some men completed a BIP and did not reoffend, others failed to complete court-mandated BIPs. Many men also became difficult to track down for surveys. The positive evaluations left out these populations, who were the people most likely to re-offend. More recently, careful studies that recognized the systematic differences between men who stuck with the programs and those that didn’t found that mandating the programs had a small or no effect.
There is disagreement on what to do next. Some researchers and practitioners have argued that BIPs could still be effective for some people. What is needed is better targeting and tailoring of the BIPs, coupled with evaluation. Others have taken the position that policymakers should stop relying on these programs because they waste valuable resources and create a false sense of security for women who think their batterer will be reformed through the programs. This is a really important evidence-based debate that should result in more effective policy.
This example is not unique. Evaluations of evaluations, known as systematic reviews, are becoming prevalent in many sectors including medicine, international development, education and crime and justice.
The way a systematic review works is that a team of reviewers focuses on a specific policy intervention. The reviewers do an exhaustive search for all the evaluations on the intervention. This includes academic and consultant evaluations, and includes other geographies. Then the reviewers carefully consider each study. They particularly focus on how carefully each study considered what would have happened in the absence of the intervention—the counterfactual – and whether there is a risk that the results may be skewed one way or another.
The systematic review report discusses each study’s risk of bias and then reaches a conclusion about the intervention based on the studies with the lowest risk of bias. In some cases a systematic review may conclude that a program is effective, or that it is not. In other cases a review finds that there is insufficient evidence to reach a conclusion. In these cases the review recommends how evaluations should be performed in the future to reach a firmer conclusion.
There are several reasons why now is the time to begin doing systematic reviews of energy efficiency evaluations. First, a very large number of evaluations have been completed across the country and world. There is value in reviewing and synthesizing these evaluations so that policymakers everywhere have access to the best evidence. Second, new statistical approaches are taking hold in energy, fueled in part by smart meter data. Systematic reviews can help policymakers make sense of the diversity of approaches. Third, energy efficiency is taking on increasing importance, as reflected in ambitious goals and growing spending. The evidence base needs to be strong to ensure the resources are being used effectively.
Research conducted at The E2e Project points to questions that systematic reviews could help answer. When are ground-up engineering estimates most appropriate to use? How important is the rebound effect? What considerations are most important when embedding evaluations into program design? What can interval smart meter data tell us about the effectiveness of programs that other approaches cannot?
Several of these were highlighted by agency staff at an energy efficiency workshop held by the CEC last month.
California produces only 1% of global greenhouse gas emissions. Given that, as Severin emphasized in a prior blog, the state’s policies can’t possibly have a meaningful direct impact on climate change. Instead, the way California can best address the climate change challenge is through invention and learning, then exporting the knowledge to the world.
In the case of energy efficiency, California should focus on finding which policy interventions are most effective and sharing the findings. Policymakers should take a look at systematic reviews as a tool to accomplish this.
Categories
Andrew G Campbell View All
Andrew Campbell is the Executive Director of the Energy Institute at Haas at the University of California, Berkeley. At the Energy Institute, Campbell serves as a bridge between the research community, and business and policy leaders on energy economics and policy.
Energy Institute Blog 
Evaluation resources are for a variety of needs–and more need to be dedicated towards providing information that can be used for future interventions. An evaluation of evaluations can be most useful if it highlights where methods and efforts need to be adjusted/changed to meet future information needs. The past is past–and not much can be done to change that. Evaluation also needs to focus more on harder to estimate and currently undervalued aspects of the energy efficiency programs–for example, how the interventions accelerate the maturity of energy efficiency markets, and the non-energy benefits that ensue from these interventions.
I now understand why the conservatives think CA is a mess. WHY must the state evaluate evaluations. Energy is a COST for ALL businesses; let them choose WHATEVER efficiency mechanism they want to save the most for their bottomline. Academics can write all the papers they want and do evaluations of evaluations to be used by businesses in making decisions — the state has no business in setting these mechanisms – eat in so far as they set standards for state entities.
The herd of elephants in the room is the rapid convergence of energy, technology and broadband: The Internet of Things,
A good point that California needs to take its role at the global laboratory seriously. We have more implementation experience than anyone AND we have the same difficult circumstances that are common everywhere.
It’s amazing that Increasing Natural Gas Energy Efficiency has not yet been realized by the State. California’s goals to reduce Global Warming and CO2 Emissions and the need to Conserve Water all fit together when increasing natural gas energy efficiency.
California has a lot of industries that are venting a lot of Hot combusted natural gas exhaust into the atmosphere – and electricity producing power plants, so we can power the EV’s.
This heat energy with the technology of Condensing Flue Gas Heat Recovery will recover the heat energy and make it available to yet be utilized. Being vented into the atmosphere will be Cool Exhaust.
For every 1 million Btu’s of heat energy that is recovered and utilized, 117 lbs of CO2 will Not be put into the atmosphere.
In every 1 million Btu’s of natural gas that is combusted are 5 gallons or recoverable distilled water. If a power plant recovers 20% of the water from it’s exhaust it can be self sufficient. If it recovers more it can become a water supplier to it’s district.
There is much more that can be done to help the State to meet it’s goals, if they truly want it.
This is an interesting post. While you’ve highlighted many of the reasons that evaluation can be a useful tool, you haven’t exposed a critical flaw in that the current EM&V framework is not a useful policy tool because its far too slow to be useful in modifying policy in time to successfully support the goals of SB350. This in addition to being highly subjective and often contested – not least because energy efficiency technologies and services evolve and appear more quickly than most utility policy environments can handle. While current version of EM&V 2.0 might speed things up they will converge to a point where evaluation will mostly serve as an attribution mechanism of what happened in the past rather than as a policy tool to drive the future.
Nick, Thanks for the comment. I agree policy can’t always wait for evaluations. I think policy should move forward with the best evidence on hand when the decision needs to be made. However, if the evidence is shaky, then we should build further evaluations into the program. Systematic reviews could help identify which programs need better evidence. Then as we learn more, we can revise our plans for the future.