“We’re fifteen to twenty years out of date in how we think about renewables,” said Dr. Eric Martinot to an audience at the first Pathways to 100% Renewables Conference held April 16 in San Francisco. “It’s not 1990 anymore.”

Dr. Martinot and his team recently compiled their 2013 Renewables Global Futures reportfrom two years of research in which they conducted interviews with 170 experts and policymakers from fifteen countries, including local city officials and stakeholders from more than twenty cities. They also reviewed more than 50 recently published scenarios by credible international organizations, energy companies, and research institutes, along with government policy targets for renewable energy, and various corporate reports and energy literature.

The report observes that “[t]he history of energy scenarios is full of similar projections for renewable energy that proved too low by a factor of 10, or were achieved a decade earlier than expected.” For example, the International Energy Agency’s 2000 estimate for wind power in 2010 was 34 gigawatts, while the actual level was 200 gigawatts. The World Bank’s 1996 estimate for China was 9 gigawatts of wind and 0.5 gigawatts for solar PV by 2020, but by 2011 the country had already achieved 62 gigawatts of wind and 3 gigawatts of PV.

Dr. Martinot’s conclusion from this exhaustive survey? “The conservative scenarios are simply no longer credible.”

There is now a yawning gap between “conservative” scenarios and more optimistic ones, as illustrated in this chart contrasting scenarios published in 2012 by entities like the IEA and ExxonMobil with those offered by groups like the International Institute for Applied Systems Analysis (an international scientific policy research organization), Greenpeace, and the World Wildlife Fund.

At the low end, ExxonMobil expects renewable energy to have just a 16 percent share of electricity by 2040, while BP projects a 25 percent share. At the high end, groups like Greenpeace and WWF/Ecofys think renewables could nearly or completely overtake electricity production by 2050.

Renewable generation targets in the OECD far surpass the more modest outlooks, with the European Union, Germany, and Brazil reaching for at least 60 percent renewable electricity by 2030, while Denmark and Munich aim to hit 100 percent by 2030.

And those targets might actually be achievable. For the past seven to eight years, the annual growth rate of global wind capacity has been 25 percent to 30 percent, while that of solar has been 50 percent to 60 percent, Dr. Martinot notes. Collectively, renewables are growing at rates of more than 20 percent. More money has been invested globally in renewable capacity than in fossil-fueled generation capacity since 2010, led by China, the U.S., Germany, Italy, and India.

Martinot hastened to banish another point of conventional wisdom: that the grid can’t handle high penetration rates of renewables. “It’s not a question of technology or economics,” Martinot asserted. “It’s finance, business models, [and] market development. We have all the technology we need already. The economics are there, or coming soon.”

Further innovation would help, but isn’t necessary. “Research and development (both public and private) for new technologies, especially driven by new materials, will certainly make the future come easier and faster,” Martinot wrote in the report. “But we don’t need to wait for those breakthroughs — we have enough at our disposal already.”

Commercial battery storage is arriving faster than most people realize, and pumped hydro can be greatly expanded to buffer the grid. Even so, Martinot dismissed as more conventional wisdom the notion that we need to wait for better storage options, because there are now numerous ways to manage grid variability, such as demand response and better forecasting software. “Storage is just a small part of it,” he told the conference. “An 80 percent share [from renewables] is possible in most scenarios without large amounts of storage.”

Before 2020, renewables will become economically competitive. With the cost of solar PV now under $1 per watt, and grid parity within reach over the next five years in much of the developed world, it’s just a matter of time. Indeed, Martinot noted that in Germany, the feed-in tariff (FIT) incentive price is already below the grid price, so they could eliminate the FIT now and just proceed with net metering.

Then the conversation will turn to how integration can be managed, not whether it will happen. “A new generation of policy is ahead,” Martinot told the conference. “It’s a question of how, not cost- or price-related anymore.” New business models, including third-party players, will be needed — a point that Rob Day made last month.

Building the new capacity will take significant amounts of capital, but here again things are turning in favor of renewables. In an environment of low — even negative — interest rates, many wealth managers are looking to new sources of finance. Pension funds, insurance funds, securities funds like mortgage-backed securities, community funds, and sovereign wealth funds are discovering that renewable energy is the lowest-risk option, even lower than other conventional power sources.

In his personal observations at the end of the report, Martinot says that the prospects are excellent for the world to be 80 percent to 90 percent fueled by renewable energy by 2040-2050. Getting to 100 percent might be a useful social and political aspiration, but in practice it’s probably best to meet some remnant transportation needs with conventional liquid fossil fuels, and use some natural gas to balance out grid variability.

The demand for petroleum fuels will fall as better transportation options — vehicle-to-grid technology, biofuels, different vehicle sizes and types, and different modes of ownership and operation — are integrated. Likewise, the demand for energy in the built environment will fall with increasing integration of technologies like passive and zero-energy design, solar thermal (for cooling and heating), and geothermal energy.

Martinot’s vision for the renewable-powered grid is hugely bullish. “In the coming years, there will be an explosion of solar PV rooftops across the world, big and small,” Martinot wrote. “Fifteen or twenty years from now, a ‘bare’ rooftop will seem very strange to us, and most new construction will include PV as routine practice. This will lead to a parallel explosion in micro-grids (both residential and commercial), community-scale power systems, and autonomous-home systems. The grid will become a much more complex hybrid of centralized and distributed power, with a much greater variety of contractual models between suppliers and consumers. For bulk power supply and industry, the ‘big grid’ resources — wind, solar thermal power (CSP), and geothermal — will predominate.”

This may be an unwelcome prospect to the private utility incumbents in the U.S., as I have discussed previously in pieces published by Greentech Media (see “Can the Utility Industry Survive the Energy Transition?” and “Adapt or Die? Private Utilities and the Distributed Energy Juggernaut“). Or perhaps the utility industry will seize the opportunity to lead that transition, as Peter Kind asserted in his response to my article.

Martinot definitely sees big industry playing important roles, including utilities, oil companies, automakers, IT companies, and industrial giants.

Last week I reviewed some of the disruptive challenges that the private, investor-owned utility (IOU) sector is facing in the transition to distributed renewable energy production, and they are serious. Last month, NRG Energy’s CEO David Crane called distributed solar “a mortal threat” to the IOU’s business.

The question we all must grapple with is how the utility industry will be transformed by these disruptions, and what the effect will be on consumers.

The first issue is price. The IOUs have warned that more distributed power generation will drive up grid power prices for consumers. That may be true, to a limited extent, but the accounting is complex and the result is anything but straightforward. New coal, nuclear, and gas power plants are expensive too. As I explained last week, within a decade — less than half the expected lifetime of these long-lived capital assets — they are likely to be more expensive than renewables.

In fact, price isn’t much of a leg for the IOUs to stand on. According to EIA data on all 3,186 utilities in the U.S., helpfully compiled by Richard Caperton of the Center for American Progress, IOUs offer the most expensive grid power in the nation, with an average price of $0.1003 per kilowatt-hour. All other types of utilities — electric co-ops, municipal utilities, federal utilities, and power marketers — sell power for less. On average, municipal utilities sell power for roughly half a cent ($0.0046 per kilowatt-hour) less than the IOUs.

It’s not hard to see why. In January, the Houston Chronicle detailed how the average CEO pay at major utilities rose from $2.7 million in 2000, just after Texas approved deregulation, to over $7.5 million a decade later. In addition to paying executives more, IOUs must turn profits for shareholders, pay taxes, sponsor legislators, and occasionally fight with consumers and town governments in courts and on ballots, where their non-IOU competitors do not. All of those costs add up to higher grid power prices.

Further, as the New York Times noted in March, nonprofit municipal utilities can obtain tax-exempt financing for capital projects, rather than higher corporate rates. Along with having fewer expenses, this enables municipal utilities to invest more money in grid maintenance and outage prevention, which makes their service more reliable. After Hurricane Irene in 2011, municipal utilities in Massachusetts restored power to their customers in a day or two, while the IOUs took about a week. And in Winter Park, Florida, which converted from an IOU to a municipal utility in 2005, customers now enjoy more reliable service at competitive rates. After taking some short-term losses while it made capital improvements that the IOU had deferred, the utility is now turning a profit and investing in underground cables.

This doesn’t mean that municipal utilities are always more responsive, however. The much-publicized failure of the Long Island Power Authority (LIPA) to get power restored after Hurricane Sandy prompted a commission handpicked by Gov. Cuomo to recommend that the public authority be privatized.

Perhaps the grass is simply always greener on the other side of the public-private fence. But where public support for renewables is strong, it seems clear that municipal utilities are more likely to accommodate them than IOUs.

IOU: Intransigent, obstinate, and unwilling

Faced with the prospect of having their revenue streams from generation, transmission and distribution slowly leak away as more distributed renewable power joins the grid, it appears most of the IOUs would rather fight than switch.

PG&E, one of two California IOU monopolies, has a long record of fighting this trend even as it touts itself as a progressive utility. As I detailed in 2010, when Community Choice Aggregation (CCA) entities began to appear in response to residents’ demands for more renewable power than their utilities would offer, PG&E fought them vigorously. First it sponsored a public relations campaign under the guise of a “Common Sense Coalition.” When that failed, it put $35 million into a ballot proposition disguised as an appeal to protect the voters’ “right to choose.” The proposition would have foreclosed on the possibility of any further CCAs being created in the state, and prevented the one CCA that did exist from expanding its service area. It also failed.

Now, according to a recent E&E story, PG&E is trying to weaken California’s net metering law, which requires utilities to credit consumers for electricity put onto the grid by their rooftop solar systems. The California IOUs have fought consistently to artificially cap the amount of electricity generated under net metering as it was raised from an initial 2.5 percent, to 5 percent in 2010, then to 11.6 percent in 2012. PG&E now claims that the net metering law is unfair, because solar owners use transmission and distribution lines but pay less for those services (and contribute less into state assistance programs for lower-income residents, which are funded by grid power consumption) as they draw less power from the grid.

Given the fact that only about 1 percent of the state’s energy mix comes from solar, this seems a particularly weak argument to make. Sunrun vice president Bryan Miller characterized it appropriately: “Utilities pretending to be ratepayer advocates is like Cookie Monster pretending to be a slow food advocate.” That doesn’t mean it’s a non-issue, however.

“Net metering is a good thing,” Richard Caperton notes. “But the utility line on that — that it’s a cross-subsidization against poor people — is a real thing, and not just a talking point. I think we’re going to see regulators get real interested in those questions again.”

Many IOUs have also dragged their feet on upgrading their grids and incorporating new technologies that make it easier to accommodate more renewable power. Most recently, the Illinois utility Commonwealth Edison has been slapped with a class-action lawsuit for failing to comply with a state commerce commission order to upgrade its grid, install smart meters and improve system reliability, although it has been collecting customer payments for the new meters.

Adapt or die

Two weeks ago on Greentech Media, cleantech investor Rob Day opined that utilities should respond to the distributed generation threat by allowing their existing business to shrink until it’s essentially just a wire management business, while branching out into new business niches like efficiency upgrades and financing through unregulated subsidiaries. “They would need to embrace that the grid will be the source of kilowatt-hours of last resort in many cases, and stop trying to make their margin off of the kilowatt-hour thus sold,” Day wrote. “But it’s going to happen to them if they don’t get out in front of it. They need to eat their own lunch before someone else does.”

Day is right that such a transformation could be a “win for the shareholders of IOUs.” But that would take vision, and an appetite for risk and entrepreneurial leadership, which the change-averse utility industry does not have. As Washington, D.C.-based independent energy consultant Scott Thomasson told me, it would require “a cultural shift” that goes against the grain. The intrusion of private equity into the sector that Day wrote about — the “other financiers and other startups” who “take advantage of IOU inaction” — amount to “a hostile takeover,” Thomasson says, and “the IOUs have no interest in it.”

So what will the private utilities do? In an interview with Platts last month, Jim Rogers, the recently ousted CEO of Duke Energy, admitted that “we need a new model to be viable and to be able to deliver electricity in the future and to optimize the use of electricity among the different supply sources.” But the alternatives he mentioned — the “fixed variable” approach used by gas pipelines, or the “formula rate” approach — spoke of inside-the-box thinking, not the kind of transformative leadership that will be needed for IOUs to adapt. If that’s representative of the views of IOU CEOs, then Rogers will be right about their fate: “Some people won’t pay attention to [the disruptive changes] and then they’ll wake up five years from now and look back and say ‘Oh, my God!’”

If the IOUs sleep on energy transition, consumers will find other ways to connect rooftop solar and other distributed generation to the grid without restriction. There is no good justification for a cap on net metering other than slowing down energy transition, which is not in the public interest.

In short, the private utility business will either be made obsolete or transformed back into public utilities.

What happens next in Boulder, Colorado will be telling. If the division of Xcel Energy that currently serves the city capitulates to community demands for more renewable power, then other IOUs may follow its model. But if it does not, and Boulder proceeds with its effort to convert to a municipal utility, that may become the model for the rest of the country.

One way or another, the energy transition juggernaut will move ahead.

The utility industry is being disrupted on every side, prompting worries about its stability.

A new policy paper from the Edison Electric Institute (EEI), an association of shareholder-owned U.S. electric companies, details the “disruptive challenges” the sector faces. These private, for-profit companies, also known as investor-owned utilities or IOUs, serve about 70 percent of the U.S. population. They are usually subject to different regulations than publicly owned utilities or co-ops, and must turn enough profit to retain their investors in a difficult, constrained, low-margin business.

Here are just a few of the challenges the industry is facing.

Falling costs of distributed generation

The cost of power generation from solar PV, wind, geothermal, micro-hydro, and fuel cells running on natural gas has been dropping dramatically. Residential and commercial utility customers can now generate some or all of their own power economically instead of drawing it from the grid. The cost of such distributed generation is set to continue falling as more of it is deployed around the world, and “could directly threaten the centralized utility model,” the report acknowledges.

Distributed generation costs are falling too rapidly for lumbering utilities to adapt to these new business realities. In Germany, which has deployed renewable power aggressively for the past decade, solar power has nearly reached grid parity and may now be “unstoppable” even without subsidies, according to Macquarie Group, a global investment bank. Germany’s coal-fired and nuclear power generators are now struggling to remain profitable as their share of the market shrinks as higher-priced peak hours of the day are increasingly met by solar.

Worse, being able to generate your own power means that you might eventually decide you don’t need the grid at all. The EEI raises concerns that “the longer-term threat of fully exiting from the grid (or customers solely using the electric grid for backup purposes) raises the potential for irreparable damages to revenues and growth prospects,” observing that it may become difficult to recover investment costs over a 30-year period, as it has done in the past.

Increasing use of demand-side management technologies

The cheapest watt is the one you don’t have to generate, and the most expensive watt (for fossil-fueled power) is the one that has to be generated at peak times. As such, improving efficiency and shifting loads away from peak times has made good sense for consumers and the economy as a whole. But those aren’t necessarily good things from a utility’s standpoint.

Efficiency gains have gradually reduced the amount of power customers use, and “spending on energy-efficiency programs will increase by as much as 300 percent from 2010 to 2025, within a projected range of $6 billion to $16 billion per year,” according to research by Bloomberg New Energy Finance. This investment will have “a meaningful impact on utility load and, thus, will create significant additional lost revenue exposure,” the EEI report says.

And now there’s a new threat: demand-response technologies and “curtailment service providers” (CSPs), which offer voluntary programs that let customers reduce their power consumption at peak times in return for lower, off-peak electricity rates. Some demand response customers will cut their power consumption when the utility requests it — for emergency response, or to reduce peak load — while others will voluntarily reduce their demand when advised by CSPs, in order to reduce their overall power costs.

Demand response is one of the reasons why power consumption is decreasing overall, which means that utilities aren’t able to make as much money on peak power generation or increase their revenue by building new power generation capacity.

Government programs to incentivize selected technologies

The utility industry complains about “selected technologies,” but only when those technologies — like renewables, efficiency upgrades, and demand response — cut into their fossil-fueled power generation business. They don’t complain about the government picking winners when it sells coal mining leases on public lands at well below fair market value, or when it continues the 100-year-old percentage depletion allowance which lets oil and natural gas companies write off a certain percentage of the oil and gas they extract.

The declining price of natural gas

The boom in U.S. shale gas helped to drive the price of natural gas down from $13/MMBtu in 2008 to $2/MMBtu in 2012, which reduced the price of wholesale power and hurt utility profits. As the ratings agency Fitch noted in February, low gas prices put pressure on “power producers that depend heavily on the sale of excess power to subsidize their retail revenue,” and over the long term, it can accelerate the retirement of coal-fired power plants whose capital costs were recovered long ago. Ultimately, it may force utilities to spend significant capital on newer, cleaner natural-gas fired plants in an environment of relatively low wholesale power revenues.

Capital expenditures required to upgrade the grid

The U.S. power grid is old, creaky, largely analog, and suffering badly from decades of deferred maintenance. This is precisely what one would expect when it is operated by for-profit companies, which have every incentive to “maximize shareholder value” by squeezing every last month of service they can out of their capital assets. They are required by law to maintain certain service levels, but they have not been required to proactively upgrade their equipment in order to, for example, adapt to a more robust, security-oriented digital age. Indeed, as a recent New York Times article noted, “some utilities don’t know if a customer has lost power unless that person calls to complain,” adding that “[m]any utilities still rely on paper maps of their systems that become outdated quickly.”

Market forces, the spread of technologies like distributed generation and smart meters, and growing regulatory pressure are now forcing the utility sector to face its deferred maintenance backlog, and those costs are going up. This has the utilities worried, because it is coming at a time of industry contraction.

Slowing economic growth trends

Thanks to the recession, improved efficiency, and distributed generation, electricity demand in recent years has been “anemic.” The power industry is no longer growing as it did in the past, but now it must ” deploy capital investment at almost twice the rate of depreciation to enhance the grid and address various regulatory mandates.”

IOUs are now facing a “vicious cycle,” EEI says, wherein the industry’s decline will make it harder to pass on the costs of providing service, because customer rates are tied to usage. As usage declines, the costs of new investment must be passed on to a shrinking pool of customer demand, which in turn forces per-unit prices higher still. As those prices rise, investment in efficiency and renewables becomes even more cost-effective, which shrinks usage further. Ultimately, these dynamics could leave a small number of customers supporting the costs of a large chunk of grid infrastructure, leaving utilities with “stranded investments.”

It could become very difficult for IOUs to remain profitable, which will test the loyalty of investors. Brokerage firms are forecasting earnings per share of 4 percent to 7 percent for IOUs, EEI notes, but if the utilities aren’t able to meet these investor expectations, “a wholesale reevaluation of the sector is likely to occur.”

While all of these challenges to the traditional private-sector utility model are indeed disruptive, it’s instructive that they’re also for the good of the environment, and for our communities. Nobody ever said that energy transition was going to be easy, or that there wouldn’t be losers as well as winners. Some utilities will navigate the transformation successfully, while others will fight it tooth and nail until they die. It may make sense for some of them to convert to public entities. Stay tuned to this space for more on that.

***

Chris Nelder is an energy analyst and consultant who has written about energy and investing for more than a decade. He is the author of two books (Profit from the Peak and Investing in Renewable Energy) and hundreds of articles, and has been published by Scientific American, Slate, the Harvard Business Review blog, Financial Times Alphaville, Quartz, the Economist Intelligence Unit, and many other publications.