Vehicle Emissions Testing Where the Rubber Hits the Road

What do pro cycling, education reform, and vehicle emissions standards all have in common?

goodhart  source ,  source , source

Cheating scandals is the flip answer. But a more fundamental connection is summed up by “Goodhart’s law” which predicts that, once a measure becomes a proxy for the purpose of enforcing a policy, it will cease to provide a good measure. Goodhart (an economist) had monetary policy in mind. But look around and you’ll find lots of other examples.

Let’s start with the greatest sport of all time: pro cycling. Doping has dogged this sport for decades. In the late 1990’s, regulators tried to crack down by requiring competitors to stay within acceptable limits defined in terms of things that proxy for doping (such as red blood cell counts). Once rules were in place, it appeared as though cyclists were cleaning up and staying within (err.. suspiciously close to) these limits. But we have since learned of the crazy things pro cyclists have been doing to dope up without testing positive.

Public education reform offers another unfortunate example. No Child Left Behind relies on standardized test scores to hold public schools accountable for student achievement. Initially, the news was good- many schools were rallying to hit test-based targets. But many of the changes made to meet these targets involve “teaching to the test” versus fundamental improvements in learning.  And the pressure to improve test scores lead to cheating by some teachers and administrators.

Of course, a similar saga is unfolding in the world of vehicle testing.  Automakers must comply with EPA tail pipe emissions standards and federal fuel economy standards if they want to sell new cars in the United States. To assess compliance, regulators rely heavily on measurements of vehicle performance collected in controlled testing environments. These tests are systematic and predictable. Predictably, automakers have learned to tune their vehicles to pass the test…. or cheat.

With his trademark Auffhammerian wit, Max recently blogged about the VW cheating scandal that broke in September. The story reads with all the opprobrium of a Tour de France doping expose. VW, while promoting its diesel vehicles as clean and green, was behind the scenes installing “defeat devices” that could detect when these cars were being tested and reduce emissions accordingly.

vwSource

These vehicles performed beautifully during the laboratory testing. But when real drivers get behind the wheel, emissions are up to 40 times the limit. VW has since admitted to installing devices in approximately 11 million diesel cars worldwide. As of last week, the scope of the scandal was still widening.

Vehicle emissions testing appears broken

11 million diesel cars and counting is a big deal. But the VW scandal is just one part of a larger problem. This is not the first time a major automaker has been caught cheating on these tests. Less scandalous -but also troubling- are the superficial adjustments that manufacturers can use to pass these tests. Because testing protocols are well understood by automakers well in advance, manufacturers can design cars that perform better in test-mode, as compared to real world driving conditions.

As emissions and fuel economy standards get more stringent, the gap between test results and reality appears to be widening. With regards to fuel efficiency, research by David Greene and colleagues suggests that the shortfall between test cycle estimates (used to measure compliance with regulations) and in-use estimates has been increasing since 2005. This Mind the Gap report estimates that the gap between CO2 emissions rates measured in European vehicle testing and on road performance has increased from 8 percent in 2001 to 40 percent in 2014.

So where does this leave us? Depressed if you care about air quality and fuel economy. Some of the gains we think we’ve made in improving the environmental performance of our vehicles are illusory. Confused and misinformed if you are in the market for a new car. Consumers cannot hope to make informed choices among vehicles with different emissions/environmental attributes when posted performance measures provide distorted/noisy signals.

Broken but fixable?

From my blogger’s armchair, vehicle emissions testing looks broken but fixable. Particularly when compared to other manifestations of the Goodhart problem.

First, think about the outcomes we are trying to target and improve upon. In public education, these outcomes are hard to define (let alone quantify): improvements in critical thinking, deep understanding, moral character. No chance of measuring these directly with a bubble sheet and a No. 2 pencil, so we rely on crude proxies. In contrast, the environmental performance of new cars driving around on the roads is directly measurable (in principle). To overcome Goodhart’s law, why not focus more directly on what we are targeting?

Relatedly, think about the incentives to cheat. In the case of pro cycling, the regulator’s problem is greatly complicated by the fact that she is inevitably stuck monitoring (read drawing the blood from) athletes who have a strong incentive to cheat and a strong incentive to evade detection.  Cynically, the same can be said of vehicle manufacturers. But we can (in principle) put more distance between the manufacturers with an incentive to cheat and the vehicles we seek to test.

Finally, new cars and measurement technologies are getting really smart. Max’s car can park itself. My neighbor’s car gives her a friendly warning when there is pedestrian or cyclist nearby. It seems relatively elementary, then, to ask our cars to log emissions and fuel efficiency information.  It is estimated that over 60 percent of new cars will have “connected capabilities” by 2017, allowing them to exchange data digitally with other cars or infrastructure. Couldn’t this interconnectedness be leveraged to collect rich data on real-world performance?

If you recoil at the thought of inviting the EPA into your personal tailpipe space, or if you are concerned that smart car technologies could also be designed to outsmart emissions monitoring, there are other direct-measurement alternatives. Several states (including California) are using roadside devices to scan exhaust emissions (and license plates) from thousands of vehicles per day and then match these data with vehicle make, model, and year via vehicle registration data. These data can be used to estimate model-specific measures of the average emissions performance of new cars on the road.  Why not use these measures in lieu of – or in conjunction with- test data to determine compliance with new vehicle standards?

I get it that there are plenty of complications that crop up when we start to think seriously about using data collected under noisy real-world conditions to monitor compliance directly (rather than relying on test-generated proxies). But recent events have highlighted the troubling limitations of current testing approaches (perfectly predictable a la Goodhart).  The EPA has recently indicated that it plans to expand random, on-road emissions testing (albeit under controlled protocols) to provide a reality check for lab tests. If recent developments ultimately move vehicle emissions testing closer to where the rubber hits the road, there may be a silver/green lining in this cheating scandal cloud.

 

This entry was posted in Uncategorized and tagged . Bookmark the permalink.

9 Responses to Vehicle Emissions Testing Where the Rubber Hits the Road

  1. Pingback: Vehicle Emissions Testing Where the Rubber Hits the Road - Berkeley-Haas Insights

  2. Bill Julian says:

    Meredith — Thank you. How about the proxy of renewable energy for reduced carbon emissions ? Bill Julian

  3. Pingback: Vehicle Emissions Testing Where the Rubber Hits the Road | Renewable Electron

  4. Dan Sperling says:

    This blog is largely misinformed. First, only very limited cheating has been found (VW diesel cars). There is no evidence there is a industry-wide or systematic cheating. The temptation to cheat is minimal in the US because the US and Calif conduct random testing of in-use vehicles (by “borrowing” cars from private owners). There was some cheating on some models by two car companies a few years ago on fuel economy, but the differences were probably too small to have been detected by on-road testing. Second, the fact that the tests aren’t accurate measures of real-world emissions means that the test procedures are not well matched to real-world driving, not that the tests are incapable of accurate measurements. In Europe the tests are well known to greatly under-measure real-world NOx emissions and moderately under-measure real-world CO2 emissions, but it is because the tests are purposely designed to under-estimate. The solution is to fix the tests. The only reason they are not fixed is because of politics (automakers intervene with national gov’ts that intervene with the European Commission)–which means the problem is not a test procedure issue but a lack of political commitment. Third, on-road emissions testing can be used for detecting possible problems, but it can never be used for compliance purposes. One needs standardized tests and test protocols in order to create a fair, replicable, enforcable test for all automakers and all cars. On-road tests that find a car with high emissions are not robust enough to determine if all cars of that model are out of compliance. The high emissions might be due to hard acceleration at that moment, tampering with the devices, and other one-off events and circumstances. There would have to a vast network of monitors and a vast and ongoing analysis of second-by-second data for the vast number of vehicle models to collect enough that to create an enforcable compliance program. That is impractical and highly expensive.

  5. Paco D'Exponere says:

    After reading the claim that professional cycling is the greatest sport of all time, one is tempted to dismiss any further claims by the author. However, even leaving the ad hominem fallacy behind, much of this entry seems to miss a clear solution.

    To be sure, Goodhart’s Law seems particularly relevant. Then again there is one measure that is much more difficult to avoid. Simply stated, if policy is allowed to assume that all purchased fuel will be burned then it becomes a relatively straightforward exercise to appropriately assign the environmental burden to the fuel purchaser.

    Clearly opposition in the form of CEO visits to the governor’s office, busloads of refinery workers slamming ARB hearings and other forms of political opposition stand in the way. But if the country were to implement only those policies that faced no opposition then far fewer laws would exist. Shock of all shocks, some policies are appropriate in spite of well organized opposition!

    So, back to the idea of imposing the environmental burden on the fuel buyer. Fuel is purchased and then it is burned. Funny thing; none of the global climate models incorporate any parameter based on the efficiency with which a fuel was used. The atmosphere really sees no difference between a car that burns a gallon of fuel after driving 50 miles and one that burns a gallon of fuel driving 20 miles. Assigning the responsibility for the environmental burden to the buyer of the fuel puts the clear and direct burden on the shoulders of the agent who, by his choice, is burning the fuel.

    Once done, any emissions testing shenanigans will then become an issue to be addressed solely between the vehicle maker and the vehicle purchaser. The source of the environmental burden, that being the decision to burn fuel will have been identified and the burden assessed. What a concept.

  6. Pingback: Making Lemonade Out Of VW Lemons | Ethan Elkind

  7. Robert Sawyer says:

    Excellent review of vehicle emissions cheating and the problem of real world emissions exceeding certification test emissions. This problem has existed since the beginning of motor vehicle emissions regulation. An early crude version was to turn the emission control off at a time greater than the certification test duration. Yes, since the objective is to limit real world emissions, it is the real world emissions that should be regulated. Because of the dominant effect of driver behavior (such things as driving up hills, which is not in the test procedure; failure to maintain tire pressure; strong acceleration; high speed leaving; the windows down; air conditioner use; vehicle maintenance; and more), devising a laboratory test that captures real world driving is intractable.

    Measuring real world emissions is not so simple, but the technology is evolving. Current automobiles measure and record fuel consumption. This information could be part of in-use fuel consumption and carbon emissions regulations. Of course, the possibility of cheating through biasing the measurements remains to be checked. Roadside remote monitoring provides useful information, but because of the effect of driver behavior, is not suited for vehicle regulation.

    The technology for inexpensive, accurate, on-board measurement (PEMS, portable emissions monitoring systems) of other pollutants, NOx, CO, HCs, and particulates, while evolving, is not ready for regulatory deployment but certainly will be. An interim solution would be to require a fraction of vehicles to be built with PEMS technology as standard equipment. Deploying this regulatory technology first in the heavy-duty vehicle fleet would make sense because the cost would be more easily accepted and the disparity of real-world and test emissions is great.

    New car emissions are only part of the problem. Dealing with emissions from the existing fleet of aging vehicles, which can emit orders of magnitude greater emissions than new cars, remains.

    Robert Sawyer, Professor of Energy Emeritus

  8. You bring up a very good topic, i like the way you linked everything together that is similarly linked to get your point across. It is great to see that some more drastic measure are coming to get the standards for emissions more in line.

  9. Alan Meier says:

    Hi Meredith,
    Your proposal goes by several names including “energy reporting”. It’s an idea whose time has come, but lots of bugs still need to be worked out. For example, there’s already a grey market in “performance chips” that enhance your vehicle’s acceleration, often at the cost of increased emissions and fuel consumption. One new feature might be “energy mis-reporting”.
    By the way, circumvention goes beyond VW and beyond cars. More about that at:
    http://www.eceee.org/all-news/columnists/Alan_Meier/circumvention-through-the-years

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s