Ontario Electricity Regulation Crisis Report Part 112: Toronto Hydro’s Ice Storm Response Was Awesome (Or Not)

On June 18, Toronto Hydro released its 2013 ice storm report. The “independent review panel’s report” — commissioned, scoped, staffed, and funded by Toronto Hydro — found that Toronto Hydro did an awesome job fixing the blackout. “Toronto Hydro performed the restoration in a manner consistent with industry norms.” Toronto Hydro’s communication performance “met or exceeded industry best practices.” The CEO performed his communication role “well”. The root cause of any problems customers might have experienced rested mostly with our love of trees and customer ignorance. These were the key messages regurgitated by many news outlets, with particularly unthoughtful coverage from the Toronto Star’s Christopher Hume, CTV, and City TV.

The report is replete with contradictions.

The most troubling aspect of the report is the conclusion that “Toronto Hydro’s safety performance during the 2013 Ice Storm restoration was excellent.” (p. 53 of the print version) While it is happily true that only one relatively minor lost time injury was sustained by field staff during the event, it appears that the utility’s consultant, Davies Consulting, and the “independent” review panel did not consider close calls. Had they looked, they would have discovered least one near electrocution of a worker (or group of workers) on a circuit re-energized without notice, directly violating the utility’s most basic work protection rules. The report’s failure to address close calls contradicts its emphasis that “Preserving the safety of public and utility workers is the most important aspect of the restoration execution.” (p. 52) Any safety assessment that reaches a conclusion of excellence without considering close calls is incomplete.

Davies produced a graphic representation (Figure #2) intended to reassure readers that Toronto Hydro’s costs and recovery time for the event were reasonable in comparison to comparable events. The data and methodology behind the chart are undocumented and unverifiable. Specifically on the question of cost for the recovery, in the utility world there is generally great flexibility in cost allocation, often used by utilities to game local regulatory requirements. This makes cross-utility cost comparisons for subcomponents of service cost challenging to interpret without a careful review of the survey methodology. Since the Davies methodology, such as the capitalization of recovery costs, is undocumented, the significance of Figure 2 remains a uninterpretable.

While the report summary concludes that “Toronto Hydro performed the restoration in a manner consistent with industry norms”, these findings from the body of the report point in the opposite direction:

– There is an extensive discussion of lack of planning, staff training and role definition for emergency response (pp. 28-31).
– Planning to utilize mutual aid resources (crews from other utilities) was not included in emergency plan, requiring the utility to develop training and safety briefings, among other requirements, on the fly. (p. 40)
– “The Grid Disruption Plan was found to not include a comprehensive, detailed damage assessment process,” a gap that lead to different parts of the utility developing inconsistent approaches to damage assessment. (p. 47)
– The report found “inconsistent coordination of the work across the Local Incident Command Centres.”(p. 53)
– The consultant found a lack of training for staff in the call centre to handle outage-related calls. (p. 66)

One of the basic reasons that electricity distribution utilities are allowed to have sweeping monopoly powers and guaranteed revenues is so that they can provide effective emergency services. Toronto Hydro’s failures to plan and train for emergencies seems hardly within industry norms. Instead, such failures constitute a dereliction of the utility’s license requirements which, if enforced by the regulator, would require the utility to abide by the provisions of the Ontario Distribution System Code (DSC). The DSC contains the following provisions:

4.5.6 A distributor shall develop and maintain appropriate emergency plans in accordance
with the requirements of the Minister of Energy, Science and Technology and in the
Market Rules, regardless of whether the distributor is a wholesale market participant. A
distributor’s emergency plan shall include, at a minimum, mutual assistance plans with
neighbouring distributors or other measures to respond to a wide-spread emergency.
4.5.7 A distributor shall establish outage management policies that include the following:
“¢ Arrangements for on-call personnel in accordance with good utility practice.

The report concludes that Toronto Hydro’s communication performance “met or exceeded industry best practices.” Compare that assertion with some of the findings from Davies:

– The report acknowledges that customers could not obtain timely, accurate information from the utility about when service would be restored. “An ETR (Estimated Time of Restoration) usually is the most critical piece of information to a customer. The Company did not have in place a systematic and documented process for creating accurate and timely ETRs at multiple levels of specificity (e.g., circuit, customer).” (p.48)
– The outage map on the utility’s web site did not provide accurate information because it was not fully integrated with the Outage Management System. (p. 68) (Note that the report didn’t comment on the fact that the outage map was frequently completely unavailable on THESL’s web site during the recovery period, although the site homepage continued to function.)
– The number of customers receiving busy signals when they phoned in during the storm is unknown. (p. 65)
– The performance of Toronto Hydro’s phone system was not “in line with peer utilities”. (p. 72) 20% of calls in queues within the call centre were abandoned, whereas in other storm responses the consultant has studied, the maximum abandonment rate found was 12%. (p. 67)
– A large but unspecified number of callers to the utility reporting outages were incorrectly told that their power was actually on. (p.68)
– THESL’s Integrate Voice Response (IVR) handled only 4% of calls, whereas the consultant’s database on recent storm performances shows that other utilities achieve at least a 50% IVR rate. (p. 65)

Part of Section 4.5.7 of the DSC requires:

Establishment and operation of a call centre or equivalent telephone service to provide consumers with available information regarding an outage.

Toronto Hydro might argue that it complied with the “available information” clause of the DSC through much of the outage in that the utility had no information available about when the power would be back. At no time over the course of the storm recovery did Toronto Hydro issue an even approximately accurate statement of when all customer would be reconnected.

Note that the objectives of Toronto Hydro’s review (listed on p. 11) did not include assessing the accuracy of statements made to the public in the corporation’s communications. Notwithstanding this deficiency, the review did address one of the false signals prominently promoted by the utility early in the event. The review explained how internal reporting from the System Operations Centre early in the blackout period was inaccurately converted into a public assurance of quick recovery.

On December 22, 2013 (11 days before complete restoration), one of the key messages from the System Operations Centre was that the restoration would last at least 72 hours. This estimate was translated into a restoration time of 72 hours, essentially omitting the “˜at” least’ reference from the 72″hour restoration message. It was soon apparent that this ETR was not aligned with the key message of the day and it did not accurately reflect the level of damage or the effort that would be required to restore the damage. From that point on, Toronto Hydro wisely did not issue any other projections for restoration, because it did not have a strong process for developing and validating restoration estimates at the time of the event. As a result of this decision, customers were not provided any specific estimates of their outage durations. (p. 80)

The review does not comment on how it could conclude that false signals “met or exceeded industry best practices.”

After it became obvious that the promise of restoration within 72 hour was inaccurate, one of the talking points the CEO focused on was that it would be “irresponsible” to give precise recovery times, an obvious contradiction of his previous statements. See more at here.

Why was it that all the return to service information provided by Toronto Hydro during the ice storm blackout was wildly unreliable? Davies and the “independent” review panel provide no further explanation.

The report acknowledges that the utility’s massive Smart Grid investments did not deliver benefits for consumers. For example:

Outage status and updated damage assessment below the feeder level “were not readily available throughout the event.” (p. 47) )

The utility was an early adopter of the McGuinty government’s Smart Meter policy. Toronto Hydro was one of the first to undertake mass installation, starting in 2006. The utility uses technology today for both the meters and upstream systems receiving meter data that is now obsolete. For example, the utility lacks the automated ability to ping meters, where that information can be automatically updated to outage maps. Instead, Toronto Hydro relies on manual pinging. During a mass outage, this manual process requires excessive time to perform. (p. 86) Note that while some of Toronto Hydro’s newer meters have last gasp technology, the report doesn’t comment on whether those signals were utilized in an effective way.

The report justifies the observation that Smart Meters were not an effective restoration tool by claiming that Smart Meters were never intended to play a role in reliability. It is interesting to contrast this assertion with Toronto Hydro’s own assertions prior to the event about how Smart Meters are now delivering a wide range of benefits including greatly enhanced outage recovery.

As documented in Part 100 of this series, on November 5, 2012, Toronto Hydro’s CEO Anthony Haines delivered a keynote address to the Ontario Energy Network. In his speech, Haines explained how Tony Hydro’s Smart Meters are producing “three million phone books of data per day” and this data is central to Smart Grid technologies that are delivering faster restoration times after outages.

“So what we have is all these meters, firing information in. There’s a couple of things this allows us to do. No longer do we have just a bit of SCADA (Supervisory Control and Data Acquisition) points out there. You know (with SCADA), you are getting a vague idea of what is going on out there. Now we have this nervous system pulsating away, sending you back data. The control room is managing the load like they never have before. You are managing your asset base like you never have before.”

While Toronto Hydro’s Smart Meter design and implementation proved useless during the mass outage of the ice storm, the technology is a useful tool for the utility to solve little outage where manual pinging is practical. Toronto Hydro cares a lot about performing well in response to little routine outages, like when a traffic accident damages a pole. Notice how this capability aligns with the way Toronto Hydro calculates the reliability portion of executive bonuses, where big payouts are at stake. Outages occurring on what the utility calls Major Event Days (MEDs) are excluded from the bonus calculation. The Ice Storm of 2013 did not cut into the bonuses of the executives.

Davies reports that the utility’s Outage Management System bombed during the blackout. (p. 87) It is interesting to contrast this finding with the testimony of the utility’s CEO and his technical chief for the restoration to the City Council during a meeting on January 10th. Either Councillor Fletcher or Councillor Davis asked the two witnesses whether Toronto Hydro’s Outage Management System worked properly throughout the crisis. The gist of Toronto Hydro’s reply was that the Outage Management System performed just fine. Unfortunately, I don’t have a transcript of the passage. I document other similar comments in post 102 of this series.

The report claims 98% of staff were deployed — by which I think they mean that 98% of the staff showed up for work at least one day during the recovery. I notice that the report doesn’t include the work logs for different functions — which is something I specifically recommended to them in my oral and written submissions. If Davies had reported the work logs, I believe they would have found that early in the storm, key technical coordination and field positions were staffed only at a level that is typical of non-emergency conditions despite the fact that the storm was accurately forecast by Environment Canada and others long in advance and additional qualified staff were available and could have been called in. I discuss this further here.

At p. 26, the Davies report documents that part of the reason the utility performed poorly in many areas in restoring service was that the emergency management organization within the utility is under-resourced. The report directly blames this deficiency on the OEB decision in January 2012. (Toronto Hydro’s immediate response to that decision was the initiation point for this “Ontario Electricity Regulation Crisis Report” series.) In response to that decision, the emergency planning group within the utility was downsized from four people to one person. Hang on. That OEB decision related to capital programs. The utility’s decision to downsizing emergency planning — an operating program not a capital program — in response to that OEB decision was a self-inflicted wound.

Davies and the “independent” panel state that the CEO performed his communication role “well”. I understand this statement to mean that the CEO issued statements consistent with the utility’s press releases and was consistently the utility’s single point of communication contact. (p. 78) The fact is that Toronto Hydro’s CEO Anthony Haines came up with a new, completely unreliable story on the remaining restoration time every day of the blackout. See more here.

Davies and the “independent” panel provide extensive recommendations for improvement. Like a restaurant without prices on the menu, no costs or benefits are estimated except a brief mention of the unaffordability of complete undergrounding.

No utility storm report would be complete with including several junk science claims, supported only by anecdotal evidence, that the frequency and intensity of major storms are increasing. (p. 22, 96)

If the report had properly benchmarked Toronto Hydro’s storm responses against its peers, one of the issues that should have been addressed was Toronto Hydro’s prohibition against the public speaking to field crews during the emergency. How common is it for utilities to issue a blanket prohibition on communication between the public and workers? I am not aware of any other instance, but perhaps a better informed reader might fill us in.

The ice storm occurred as a strike deadline for Toronto Hydro’s main labour union approached. It seems to me likely that Toronto Hydro’s motivation in issuing a blanket prohibition against the public speaking to field crews during the emergency was related to labour negotiations, not blackout recovery.

Customer ignorance about details of limit of utility responsibility for damaged masts is identified by Davies and the “independent” review panel as a root cause of some of the customer disruption. Blaming consumers for this confusion seems a bit rich. The first time I can find that Toronto Hydro notified customers about the need to repair their own stand pipe/mast was in a press release issued at December 25, 2013. As the recovery continued, damaged stand pipes/masts became a major factor keeping customers out of power. – See more here. Why did Toronto Hydro’s emergency plan not include immediate customer communication about responsibility for damage masts? Why did Toronto Hydro wait until four days into the outage to start informing customers?

Not every element of Toronto Hydro’s restoration effort was substandard. For example, Davies documents the successful use of a new email capability allowing the utility to email out to the largest commercial customers throughout the storm. (p. 64) This is a promising development for the future.

The Ontario Energy Board has shown no interest in Toronto Hydro’s performance during the blackout. Contrast the OEB’s lack of interest in emergency response with the Newfoundland & Labrador’s Public Utility Board recently completed thorough, professional, and independent investigation of blackouts in that province last winter. If the authors of this report had thought for one minute that the OEB might do their job and follow up on the ice storm, the report would have been very different.

11 Comments

  1. I’m disappointed that the media has been indifferent to the lack of independence of the review. It was obviously a self-serving effort, ordered up and tailored by Haines purely as a public relations exercise. With four major newspapers and plenty of other media, somebody in Toronto should have smelled a rat.

    • Toronto Hydro is a pretty big media purchaser, including in the Toronto Star/Metroland. I don’t know if Toronto Hydro is still doing it, but for a while the utility bought a central page in the Metro every week.

  2. Isn’t bringing in a consulting firm to clean up a mess, provide excuses and divert attention away from any situation just standard operating procedure now-a-days? Or is this just my opinion?

  3. (This comment was authored by Tom Griffiths, who introduces himself at the end of the comment.)

    Background – The city of Toronto got literally hammered when an ice storm occurred just before Christmas, 2013. The main problem was trees, and/or parts thereof falling onto the overhead power lines causing outages. The urban forest in Toronto is a mish mash of aggressive non-indigenous trees and native ones. Toronto Hydro does not “Manage the urban forest growing in the vicinity of its facilities” It merely meets a city tree bylaw of 3 feet clearance from its lines. It defers utility forestry management to the city parks and forestry department. In fact Toronto Hydro’s Executive Vice President and Chief Operations and Procurement Officer Ben LaPianta stated last January that “Tree trimming is not designed … to prevent against the damage that arose from the ice storm.”

    What Should Take Place – Toronto Hydro is mandated to manage the urban forest in the vicinity of its overhead lines as a responsible owner and as directed by section 40 (4) of the Electricity Act:

    (4) A transmitter or distributor may enter any land for the purpose of cutting down or removing trees, branches or other obstructions if, in the opinion of the transmitter or distributor, it is necessary to do so to maintain the safe and reliable operation of its transmission or distribution system. 1998, c. 15, Sched. A, s. 40 (4).

    Toronto Hydro does not do this now and this is a major reason the damage was so severe last December. I suggest they acquire the expertise of a registered professional forester at a senior position with utility experience to develop proper tree clearing specifications based on applied science and work with all stakeholders (customers, public, and city partners) to manage Toronto’s urban forestry canopy near the electrical environment. I can provide evidence this can be done.

    In Conclusion – Davies Consulting (a US firm) carried out a forestry study for Toronto Hydro a few years ago and they were rehired to in effect comment on their previous work – subjective??

    I am a retired Canadian professional forester with extensive utility experience and I offered my assistance to Mr. Haines (letter included) and Toronto Hydro did not even provide the courtesy of a response.

  4. Trees and power lines don’t mix and those in charge at Toronto Hydro should know this but took the chance that a major ice storm would not occur in the GTA.

    Masts around here are just galvanized pipe and there is no way that people should have been charged thousands of dollars to replace them. Friend of mine got charged about $2,000 to replace one after summer storm damage here.

  5. Weather Radar
    http://www.shelf3d.com/i/doppler%20weather%20radar

    Scroll down about 1/3 of the page:

    Canada and other northern countries use the less costly kind of radar as the precipitation in such areas is usually less intense.

    The U.S. uses more costly Doppler weather radar system.

    What kind of Doppler weather radar does Toronto rely on? Does anyone know?

    There are private companies that supply weather radar to companies that require a lot of weather information.

  6. CTV News, Aug.11,2013

    “Electrical grid faces new threats, 10 years after blackout”

    “The U.S. electrical grid is better managed and more reliable a decade after its largest blackout but remains vulnerable to increasing extreme weather, cyber security threats, and stress caused by shifts in where and how power is produced.”

    http://www.ctvnews.ca/business/electrical-grid-faces-new-threats-10-years-after-record-blackout-1.1406675

    And no one noticed this kind of information in Toronto?

  7. Pingback: Toronto Hydro’s ratepayers pay dearly for electing GTA Liberals | Energy Probe

  8. Pingback: CityPlace blackouts will be self-investigated by Toronto Hydro | Tom Adams Energy - ideas for a smarter grid

Comments are closed.