JP Morgan and Market Complexity
JP Morgan and Market Complexity
I’ve actually started to write this blog post several times over the last 9 months, but other things come up and I put it on the shelf. Fortunately (or not) the JP Morgan power trading story just keeps coming back, .. and back, so I keep having opportunities to finish this blog. For those not following this story, Morgan has recently settled charges with FERC that its “creative” bidding practices represented market abuse (or manipulation) for just over $400 million.
News of these kinds of abuses, whether legal or not, always brings on search for a deeper meaning in the direction of the electricity industry. Usually this debate centers upon whether we should have deregulated power generation or not. That is a very complex question that is important, but I don’t think the Morgan story adds much weight on one side of that or the other. Even if the industry went back to the organization of the 1980s, there would still be lots of non-utility generation, and undoubtedly lots of power trading. Utilities, even regulated ones, still need to buy their power from somewhere and as they figured out about 30 years ago it usually doesn’t make sense for them to generate it all themselves. Indeed, within the Midwest ISO footprint, one of the playgrounds for Morgan’s shenanigans, there are many old-style vertically integrated utilities co-existing with other forms of ownership.
What the Morgan story may say something about is the trend in US markets for increasingly complex price-setting processes. For example, the California market today is way more complex than the one Enron made infamous 12 years ago. Back then, the California power exchange took bids for price and quantity, but not much else. Today, an offer to sell power from a generation unit can contain all sorts of parameters describing what the plant can and cannot do, from the “ramp rates” to a minimum running time. All these parameters are fed into optimization routines such as mixed integer programs in which the system operator theoretically finds the “best” solution for everyone, taking all these operating constraints as given.
The problem is, the “best” solution when everyone is telling the truth about their costs and capabilities can be very different than the solution when firms are strategically bidding those parameters. One aspect of this is the concept of “make-whole” payments. These are intended as a means to compensate inflexible units who may get stuck in the wrong position in order to help the market in small number of hours – like a plant that gets switched on to help meet demand in one hour but can’t be (or says it can’t be) turned off very quickly. Because it is not flexible, that plant may end up still operating (at a loss) even after the price crashes back down again. Make-whole payments are supposed to allow plants to recover these costs if market revenues are not sufficient, to avoid discouraging them from participating during the hours when they are really needed.
But as the Morgan case has demonstrated, there are lots of ways strategic firms can twist the good intentions of a market operator. By overstating their costs, or their inflexibility, while at the same time “forcing” themselves into the market through low energy price bids or simply running the plant in a way the operator doesn’t expect, plants can grab lots of revenues from these payments.
How did we get here? Unlike say, complex financial securities, these rules are not the concoction of investment bankers with physics training. In other words, the traders themselves did not take the lead on these designs – although they certainly put in their opinions.
This is largely the work of engineers working with the market operators themselves. There was a very legitimate debate about 10 years ago about the pricing of transmission congestion. This was one of the simplifications of early electricity markets. It has been pretty widely acknowledged now that market prices needed to better reflect congestion caused by usage of a network. This is a classic “externality” (congestion imposed on other users) that was not being properly reflected by the old market rules. Fine. We fixed that.
However, at the same time all sorts of smart computer scientists, electrical engineers, and operations researchers went a little crazy trying to solve every other complexity involved in generating electricity. Plants can’t just ramp up and down instantly. They can’t shut down and start up again right away. Working these kinds of costs into the pricing would lead to more “feasible” solutions that better reflect reality. The problem with this logic is that these costs, unlike transmission congestion, are all “internal” to the owners of a plant. The fact that a price solution coming out of a computer program didn’t explicitly model them didn’t mean mean that the costs were not represented in the prices. Its just that they were the problem of the plant owners, who had to figure out how to best operate their plants to meet the sales orders being produced by the market.
It’s worth pointing out that, while making electricity is complicated, and marginal costs don’t look like a nice upward sloping line, this is true of a lot of other stuff also. Refining gasoline is complicated. Running an airline network is complicated. But we don’t run a single optimization program that simultaneously tries to clear the market and solve everyone’s production schedule for them. These markets run the way power markets used to. If a generator got a sale it couldn’t meet, it bought replacement power out of a spot market. If a plant ended up running in a way that lost money over the course of a day, it would change its offer price the next day so that didn’t happen again.
The market software does wondrous things, and solves tremendously complicated problems. It’s probably true that having a group of plant owners try to manage these complexities in a decentralized way creates some inefficiencies. But its still garbage in – garbage out. If firms monkey with the complicated parameters that these programs are trying to accommodate, strange things can happen to prices, and especially to the types of side-payments earned by JP Morgan. The “best” solution is not always the most complicated one.
Categories
Energy Institute Blog 
“The answer for the ISO markets may be to have bilateral agreements that essentially turn over power plant operations for short term dispatch to the ISO, with prespecified unchanging physical bounds.”
The ISOs effectively do this today. In California, they do it badly, largely because the mathematical formulations they rely on don’t handle dispatch and payment very well. Refining uses a different approach, which is to rely on prices for inputs, prices for outputs, and their knowledge of the operating characteristics of their refinery. I suspect if combined cycle plant operators did the same, there would be less fussing over how to dispatch and pay them.
My point is that the CAISO pays operators a different amount at different times based on varying bids from the operators. What I suggest instead is 1) an invariant investment recovery charge–essentially buying the hardware on the installment plan, 2) paying for 100% of the labor force (or using IOU employees), and 3) paying for fuel costs as incurred. No hourly bids, commitment pricing, etc. Simply a turnkey operation.
We could, but then we could also dismantle most of the so-called market operations of the ISO, return to fixed, cost-of-service rates, and create a whole class of unemployed economists and traders. I’m not prepared to go quite that far just yet, but it’s where we may well end up.
Actually, I am proposing going almost that far. The cost advantage that merchant owners offer is in the construction and maintenance of the power plants. Rate of return regulation is what led to restructuring in the first place. But if we had either 100% turn-key and/or tolling agreements, we could get rid of most of the market mechanisms. We’d only need to have inter-control area markets which could be run between system operators instead of involving generators. Yes, the traders would be gone. I think us economists would still have a job. ;^)
Policymakers have to decide (actually, for the most part they have): do they want competition, in which case electricity needs to be traded in a market as Stephen suggests, or do they want to continue to control the value chain? In the US, I’m convinced it’s the latter case rather than the former. An optimization is not a market. It’s an operations research solution.
Having worked in a power plant and talked with many grid operators over the years, I have to respectfully disagree with both Matt’s and Ross’s points regarding the ability of centralized optimizations to take account of technology characteristics, especially as combined cycle plants supplant conventional fossil-fired steam boilers. The California ISO’s multi-stage generation functionality, which is designed to address the mathematically challenging problem of centrally dispatching combined cycle plants, has been plagued by problems. Moreover, the centralized optimizations have (and always have had) difficulty with the “optimal” dispatch of pumped hydro plants.
The right way to go about this is to have all market actors operate in a bilateral market with forward prices at varying time intervals ahead of delivery. It’s much easier for a combined cycle plant to determine how it should operate based on prices than it is for the grid operator to determine how that plant should be operated and paid in the absence of forward prices, What I propose is no different than the method used by oil refiners to determine how best to maximize their profits when confronted with a variety of feedstocks and refined products, all of which have different prices and different technical characteristics. After all, there is no analogue of the grid operator for the refining industry, yet they only shortages I can ever remember were the result of panic buying.
What’s interesting about the refining industry is that it operates on an even thinner capacity margin than the electricity industry yet we see fewer large scale disruptions. Yes refined product can be stored, but not for extraordinarily long periods.
The answer for the ISO markets may be to have bilateral agreements that essentially turn over power plant operations for short term dispatch to the ISO, with prespecified unchanging physical bounds. This is in contrast to having hourly or subhourly markets on these services. This is essentially how utility networks ran pre 1996.
One, but not the only, lens to understand the increased complexity of information in offers is the better representation of technological characteristics into the systematic dispatch optimization. As Matt Barmack indicates, such technological characteristics have always been considered by operators to ensure that dispatch is feasible. The evolution of US markets has involved better reflection of these characteristics into the explicit information that is provided in the offer and in the parameters, such as ramp-rate limits, associated with the offer. The rationale is that by systematically optimizing the dispatch based on this information there will be more efficient outcomes than from out-of-market actions by operators. As Jim Bushnell indicates, however, strategic offers can defeat this purpose.
Stephen Littlechild asks for the moral of the story. To me, it is that the rules should be designed to reflect underlying technological characteristics, but not provide significantly more flexibility in the specification of offers than is justified by the technology. An obvious example is that there is little justification from technological fundamentals for allowing hour-by-hour changes in offers from thermal generators in either the day-ahead or real-time markets, or between the day-ahead and real-time markets, without an associated change in fuel costs or change in the operational status of the generator. Similarly, it is hard to see why a generating unit would be constrained, by technology or contract, to be repeatedly bouncing between a self-scheduled generation level and the flexibility implied by an offer. Nevertheless, some markets provide significant flexibility to change offers hour-by-hour or change the status of the generator from scheduled to offered, (while others do not). As a general observation, the more parameters that can be adjusted in the offer, particularly in the case of having significantly more parameters than justified by the underlying technological characteristics, the greater can a market participant exercise market power. Conversely, limiting the flexibility in offers to what is justified by technological characteristics will tend to limit the exercise of market power. The moral: understand the technological underpinning of the issues being represented in the offer and represent that issue parsimoniously.
Jim
I like your analysis, but what is the moral of the story? That market operators should go back to the rules and algorithms they used ten years ago and refuse to listen to engineers and traders proposing changes?
Surely this isn’t going to remove the scope for exploiting market rules. Companies have millions of dollars at stake and they naturally pay their traders and plant managers well to secure the best outcome.
Let me suggest an alternative approach. The New Electricity Trading Arrangements were put in place in England and Wales a dozen years ago. They have been criticised as not being correct or efficient in theory. But I am not aware of any significant complaint about market manipulation in practice. The kinds of cases one sees repeatedly all over the US simply have not been an issue for a dozen years. Why is this? Perhaps the significant degree of vertical integration has played a part: the buyers are also sellers and less likely to complain.
But I think there is another very significant factor. Almost all the prices are determined by bilateral negotiation and agreement. They are not determined by a market operator using an algorithm subject to manipulation by skilful and highly motivated traders. They are determined by negotiation between those same traders on both sides, all highly skilled and motivated, all well able to assess the market and to propose and to evaluate novel forms of contract. There simply is not the scope for market manipulation that exists when an algorithm sets “the market price”.
An algorithm is indeed used in the balancing mechanism, which applies to at most 5% of power supply. It has been a constant source of debate. Ofgem’s latest proposal is (inter alia) to set the balancing price equal to the marginal purchase cost rather than the average purchasing cost, on the basis that this will provide a more accurate price signal. By the same token, however, it will increase the scope for gaming and manipulation that your blog describes.
In my one and only appearance for my High School Dramatic Society, in G B Shaw’s Pygmalion (later My Fair Lady), I played the part of Mrs Higgins, mother of Professor Higgins who teaches Eliza Doolittle. The only line that I still remember was to Eliza’s father, who was worried about the wedding. “But my dear Mr Doolittle, you don’t have to suffer all this if you are really in earnest.” It seems to me that US electricity markets don’t have to suffer all this agony about market manipulation if they are really in earnest about competition: just let the market players negotiate the prices for themselves instead of designing algorithms to set market prices for them.
Thanks Stephen, I agree wholeheartedly on the forward contract point. This is a big reason why the latest round of trading abuses involve “only” hundreds of millions rather than several Billion dollars.
I wondered why in 1996 that FERC was so dead set on the idea that establishing hourly markets, and only hourly markets, was going to be the wondrous solution to bringing competition to the marketplace. And California swallowed this philosophy whole much to its later chagrin.
The hybrid market exists in most places where some players are able to recover their capital investments and some incidental; costs through the backdoor in rates, and others being fully exposed to volatile market prices. The real savings from the marketplace is not in the few cents per MWH in “more efficient” system dispatch, but rather in the billions of dollars in plant investment. Combined cycle plants came about because of market competition. If we relied much more on that type of competition,through turnkey construction or tolling agreements, we could get the real benefits of competition without the ridiculous market complexity, or the alternative of the ominous OOMs.
It’s not the market’s job to make sure that individual participants are made whole.
I agree that markets have become more complicated. I would quibble with the claim that, “The fact that a price solution coming out of a computer program didn’t explicitly model them didn’t mean mean that the costs were not represented in the prices. Its just that they were the problem of the plant owners, who had to figure out how to best operate their plants to meet the sales orders being produced by the market.” Many of the constraints that are now reflected in market-clearing algorithms and prices were considered by system operators in the past, but were addressed through out-of-market actions, sometimes known as Exceptional Dispatch in California. The regime change has been not from a case in which important constraints on certain resources were ignored by operators to one in which they are considered, but from a regime in which constraints were addressed through ad hoc operator actions to one in which there are specific market rules addressing the constraints. In addition, because the costs of out-of-market actions to address the types of constraints that you describe typically were recovered through exactly the types of “make-whole” payments that you suggest are increasingly significant under the currenet regime, the costs of many constraints generally was not reflected in clearing prices, as you suggest that they were. Under some of the new and more complicated rules, additional constraints are reflected in clearing prices.
All good points Matt. I would think a “less complex” direction involves careful consideration of which constraints are under the control of individual plants (internal) and those that are truly external to a plant and best managed by the market operator. Ancillary service products (rather than exceptional dispatch) would
have to be utilized to fill any remaining gaps.
Jim,
This is a pretty significant statement given your position and your insights as a member of the California ISO’s Market Surveillance Committee. It’s probably no surprise that I agree with much of what you say, though I would go a bit further in pointing out that the optimization algorithms used by electricity market operators do not necessarily provide an optimal solution and the results they produce are probably no less prone to manipulation than the single-part bids some of us have been advocating for many years. Moreover, the combination of make-whole payments and numerous other charges that don’t show up during the auction but do eventually get billed to consumers means prices out of the optimizations are not quite what they seem.
Assuming policymakers (or anyone else for that matter) really cares, the key question is, how do we fix this?