On December 22, 2013, as freezing rain stopped falling, Toronto Hydro’s CEO Anthony Haines promised hundreds of thousands of people blacked out across the city that everyone would have their power restored within 72 hours. He had the most advanced and costly urban Smart Grid system in Canada at his fingertips. The weather over the next several days proved far gentler than forecast on the 22nd. Despite these advantages, it turns out that Haines’ December 22 recovery promise was short by a factor of almost 4. Haines’ continuous stream of false assurances during the blackout casts an unfavourable shadow over those who have maintained Haines in his position of CEO of the country’s most important distribution utility.
No large urban utility in Canada has ever been better prepared for a major ice storm than Toronto Hydro was when the freezing rain fell on the city December 20-22.
The Eastern North America ice storm of 1998 showed once and for all that utilities in our part of the world must be prepared for damaging freezing rain. Whereas the ice accumulation in the worst parts of Toronto reached 30 mm of ice, in parts of Quebec the 1998 storm brought 120 mm.
For many years, Toronto Hydro has been ramping up spending massively, with a promise to strengthen reliability.
Capital spending, which in 2004 was $85 million and $113 in 2005, has been ramping up with the approval of the Ontario Energy Board. For the first three quarters of 2013, it has hit $297 million. Toronto Hydro’s capital spending explosion has happened as sales have declined. The sales pitch for the spending increase has been that it will improve reliability. The Ontario Energy Board is so enthusiastic for the plan that it exempted Toronto Hydro from the normal rules designed to control capital spending applied to the rest of the province’s regulated power distributors.
The core of Toronto Hydro’s spending extravaganza has carried the Smart Grid brand, an undefined jumble of hype drawing on all the currently fashionable buzzwords about power like Smart Meters, electric cars, green power, conservation, and customer empowerment.
Toronto Hydro initiated its Smart Meter deployment program in 2006. Smart Meters were the first major step toward Smart Grid. To get a sense of the vast scale of the overall Smart Grid project, consider that during the period 2008-2013, the utility spent $69.6 million on metering capital.
Smart Meters are not the only Smart Grid project where ratepayer money flows like water. For example, during the period 2008-2011, the utility spent $54.1 million on IT for distribution operations, IT infrastructure and Smart Grid IT. (ref: EB-2010-0142, Exhibit D1, Tab 8, Schedule 8-1, p. 3 of 4)
Where did this swollen tributary of cash flow? One example is that in 2011, Toronto Hydro replaced its Outage and Distribution Management System (OMS/DMS). The OMS/DMS is used to log and prioritize outage calls, diagnose problems, and dispatch work crews to fix them. The previous version of the Outage Management System, implemented in 2006 was not set up to incorporate Smart Meter data. The upgraded OMS/DMS was marketed by Toronto Hydro to its regulator as providing “functionality to support smart meter integration as well as add distribution voltage secondary data which had not been available in the previous version.” To emphasize the point, the utility promised that, “The upgraded version will improve Toronto Hydro-Electric System Limited’s ability to better isolate and respond to outages.” (ref: EB-2010-0142 Exhibit D1, Tab 8, Schedule 8-4, p. 2 of 5)
Haines, who has previous experience as a utility IT executive, has spent years repeating his talking points about the main purposes of the Smart Grid program from the start has been to provide high resolution, automated mapping of outages to speed restoration after outages and to provide customers with accurate information on expected restoration times.
As one example of his ongoing sale effort, on November 5, 2012, Haines delivered a keynote address to the Ontario Energy Network. In his speech, Haines explained how Toronto Hydro’s Smart Meters are producing “three million phone books of data per day” and this data is central to Smart Grid technologies that is delivering faster restoration times after outages. (check out the last video clips)
“So what we have is all these meters, firing information in. There’s a couple of things this allows us to do. No longer do we have just a bit of SCADA (Supervisory Control and Data Acquisition) points out there. You know (with SCADA), you are getting a vague idea of what is going on out there. Now we have this nervous system pulsating away, sending you back data. The control room is managing the load like they never have before. You are managing your asset base like you never have before.”
Capital projects were not the only big spending increases approved by the Ontario Energy Board on reliability grounds. Tree trimming has been another big ticket increase. Starting in 2008, Toronto Hydro adopted and implemented what it relabeled as a so-called “Reliability-Based” tree trimming program. Toronto Hydro also started an additional “proactive” program in 2008 to supplement the regular pruning work. The marketing pitch for the additional work since then has been “to prune beyond the regular pocket of tree branches around the power conductors and to identify branches outside of the pruning zone that carry a high risk of breaking and falling on power conductors during adverse weather conditions. By identifying the high risk areas for adverse weather and/or tree-contacts and removing potential hazards, THESL is working to improve the reliability of the system as well as remove the risk of injury to our workers and the general public.” (EB-2011-0144, Exhibit F1, Tab 2, Schedule 1, pp. 6-7 of 30)
Spending on preventative maintenance, where tree trimming is accounted for, surged. In 2008, the utility was spending $6.5 million per year. By 2010, that figure had risen to $11.4 million.
The bean counters at Toronto Hydro also decided that contracting out tree service provides better long term value. The utility’s forestry department has been largely replaced by contractors.
Efficiency gets a lot of air time in the utility’s marketing materials, but reality is a different matter. Even before the spending binge of recent years, Toronto Hydro charged by far the highest rates of any of Ontario’s urban power distributors. With its accelerated spending of the last several years, Toronto Hydro is now stretching out its lead in highest rates.
The Ontario Energy Board has enthusiastically endorsed Toronto Hydro’s high rate strategy, concentrating on ensuring “smoothing” the resulting increases.
Haines is so admired in the industry that he is by far the highest paid distribution utility executive in the country and chairs the Canadian Electrical Association. Another measure of confidence in Haines is that he has maintained the confidence of the City Council-appointed board of directors over the course of the last year as evidence piled up proving that Haines has been falsifying his credentials in sworn testimony over a period of more than 20 years, claiming that he has a university degree. The Ford brothers appear to be Haines’ strongest supporters on Council.
Electricity customers in Toronto should expect to have among the lowest delivery rates in the province. Toronto Hydro has huge inherit efficiency advantages arising from the city’s high density, Toronto Hydro’s huge scale, declining usage of power across the city, and a legacy of some fully depreciated, sometimes obsolete assets performing reasonable well. One example is the utility’s working stock of paper insulated lead cable (PILC).
“We’re Winning the War”
Throughout the storm Anthony Haines issued a stream of unreliable restoration promises.
CP24 tweeted at 9:12 pm, 21 Dec 13 “Toronto Hydro says number of outages expected to grow, restoration times are anticipated to be 12 to 16 hours.” (Note that all twitter time references are PST, whereas all other time references in this post are EST.) The utility issued this press release near midnight on the 21 and this press release from 3:21 am on the 22nd repeating the same recovery time.
CP24 tweeted at 8:26 AM, 22 Dec 13 “Toronto Hydro says could be 72 hours before power restored throughout the city. ‘Prepare for the worst,’ says CEO Anthony Haines.” The utility issued this press release at about 8 am repeating this message.
In this press release issued Sunday the 22nd at 4:08 pm Toronto Hydro stated that “the utility hopes to have everyone restored by Wednesday (December 25)”.
CP24 tweeted at 3:39, 23 Dec 13 “Haines said power outages continue to happen as branches continue to fall. Christmas is ‘a very aggressive schedule. Plan for the worst.’”
On December 23 at 1:50 pm, Toronto Hydro issued this press release stating that it expects restoration efforts to continue into the weekend (December 28/9). Later on the 23rd, Haines reassured the public that the utility was “winning the war”. At 7:11 pm on the 23rd, Toronto Hydro issued this press release stating “restoration efforts could continue for the next few days.”
The first time I can find that Toronto Hydro notified customers about the need to repair their own stand pipe/mast was in this press release issued at December 25, 2013 1:02 AM. As the recovery continued, damaged stand pipes/masts became a major factor keeping customers out of power.
CP24 tweeted at 5:06 am 25 Dec 13, “Customers should still be prepared for power restoration to take place over the next few days, Toronto Hydro tells CP24.”
On December 25 at 11:41 pm, Toronto Hydro issued the least informative press release of the emergency pointing out that the restoration effort “takes considerable time.”
At the morning press conference on the 26th, Haines told reporters, “We’re reaching the point that is the hand-to-hand combat of it.” http://www.680news.com/2013/12/26/toronto-ice-storm-5th-day-of-restoration-efforts-as-snow-wind-hit-gta/
CP24 tweeted at 8:18 am, 27, Dec 13 ‘”Plan for the worst and know that Toronto Hydro will not rest until all power comes back on,” said Haines’
In this press release from December 27 at 6:31 pm, the utility’s restoration schedule was “customers who are still without power should be prepared for restoration to continue over the next few days.” The next day at 1:12 pm and then again at 6:05 pm, the utility repeated the same phrase regarding the restoration schedule and here.
On Friday December 27, Haines refused to specify for the Ottawa Citizen when the reconnection work would be complete saying, “it would be ‘irresponsible’ to give residents false hope in such an unpredictable situation.” http://www.ottawacitizen.com/Authorities+warn+storm+related+power+outages+possible/9327889/story.html
CP24 tweeted at 8:07 am, 29 Dec 13 “Toronto Hydro CEO says less than 6,000 customers are now without power. ‘We are almost standing on finish line.’” From the same press conference, the Toronto Star reported Haines promising full power by the end of the 30th. Four days from the end of the crisis, Haines was off by 100% in his time estimate.
In this press release from the 29th at 10:30 pm, the utility informs us that the restoration work is proceeding “as expected”. The full statement is, “As expected, in this final stage, our equipment at many locations has experienced severe damage due to the ice storm, and therefore restoration is taking much longer than would normally be the case.” The whole “as expected” statement is repeated in the next press release from December 30, 2013 10:17 AM.
CP24 tweeted at 5:39 am 30 Dec 13 “Power will be restored to last remaining customers by end of day, mayor says.”
By December 30, 2013 10:42 PM, Toronto Hydro’s excuse making was at full power. “As expected, our equipment at many locations has experienced severe damage due to the ice storm and there is also a great deal of forestry work and clean up required. As a result, power restoration, is taking much longer than would normally be the case.”
Even late in the day on December 31, blacked out customers were reporting on Twitter and through the media that they could not get a statement from Toronto Hydro as to when their service would be restored.
January 1, 2014 at 8:25 pm Toronto time, Toronto Hydro issued this press release declaring an end to the emergency.
Inside Toronto Hydro’s Emergency Response
It appears that senior managers at Toronto Hydro panicked soon after the storm abated when they realized that their CEO was spouting nonsense about how fast the recovery would be. Automated systems designed to track the damage, upon which ratepayers have shelled out big bucks, failed to produce usable intelligence to guide the restoration. It appears that some of the skilled staff required to reactivate manual outage management systems had been downsized, particularly during the unplanned February/March 2012 downsizing that saw the company lose about 280 staff. (Meanwhile, the utility has continued a massive expansion of management staff.)
The company’s increasing reliance on contractors didn’t pay off for blacked out customers, as a low fraction of contractor staff showed up for emergency duty.
Just as blacked out customers were unable to reach Toronto Hydro’s phone operators, the same communication traffic jam was happening among the recovery crews. Line crews were unable to reach the utility’s control room, where Toronto Hydro’s distribution system is managed. Without clearance from the control room, crews were stuck waiting for clearance to fix problems they had discovered all across the city. In the confusion, some crews were sent off, apparently by senior management, doing useless tasks, like patrolling circuits that had already been repaired, inspected, and re-energized.
With the automated outage mapping not working, the utility was still relying on manual outage mapping. Throughout the blackout, only Toronto Hydro’s low resolution outage mapping was available to the public on the utility’s web site. Even this rudimentary information was only sporadically available due to outages on Toronto Hydro’s website. What information was available appeared to be often out of date, particularly early in the recovery effort.
The recovery performance of Toronto Hydro showed that power customers in Canada’s biggest city receive a rural standard of service recovery.
The ultimate expression of failure for Canada’s leading Smart Grid utility was the repeated message Toronto Hydro sent out over all available communication channels day after day: “Leave your lights on so we can see if you have power.” So much for “three million phone books of data per day.”
Post Script January 3, 9 pm: Edited and added to the time line of Toronto Hydro’s official statements regarding the sliding restoration time.