FERC’s 21 Lessons Learned in CIP Audits

FERC report

FERC has released a 23-page Staff report entitled “Lessons Learned from Commission-Led CIP Version 5 Reliability Audits.”  A copy of the report can be found here.  This report is a must read for compliance and cybersecurity professionals that support utilities with NERC CIP compliance programs.

Two years ago, FERC announced that it was going to take the lead on various audits throughout the country, stepping in front of NERC and the Regional Entities.  In the past, I have been involved in audits where FERC has served as “passive observers” (well, sort of), to audits led by the Regional Entities.  FERC conducted its own audits in 2016 and 2017, and is now taking the opportunity to inform the industry on what it views as lessons learned from reviewing compliance with CIP Version 5 implementation, as well as the remnants of CIP Version 3 programs.  

I could try to summarize FERC’s 21 observations, but it is probably easiest to just repeat them here.  With each lesson learned, FERC includes a paragraph or two of elaboration.  While extremely high level, FERC’s comments are practical, and can be the basis for developing internal audit and compliance activity.  Kudos to FERC Staff for providing insight to the industry in this regard.

Happy reading and have a great weekend.


1. Conduct a thorough review of CIP Reliability Standards compliance documentation; identify areas of improvement to include but not be limited to instances where the documented instructional processes are inconsistent with actual processes employed or where inconsistencies exist between documents; and modify documentation accordingly.

2. Review communication protocols between business units related to CIP operations and compliance, and enhance these protocols where appropriate to ensure complete and consistent communication of information.

3. Consider all owned generation assets, regardless of BES-classification, when evaluating impact ratings to ensure proper classification of BES Cyber Systems.

4. Identify and categorize cyber systems used for supporting generation, in addition to the cyber systems used to directly control generation.

5. Ensure that all shared facility categorizations are coordinated between the owners of the shared facility through clearly defined and documented responsibilities for CIP Reliability Standards compliance.

6. Conduct a detailed review of contractor personnel risk assessment processes to ensure sufficiency and to address any gaps.

7. Conduct a detailed review of physical key management to ensure the same rigor in policies and testing procedures used for electronic access is applied to physical keys used to access the Physical Security Perimeter (PSP).

8. Enhance procedures, testing, and controls around manual transfer of access rights between personnel accessing tracking systems, Physical Access Control Systems (PACS), and Electronic Access Control Monitoring Systems (EACMS) or, alternatively, consider the use of automated access rights provisioning.

9. Ensure that access permissions within personnel access tracking systems are clearly mapped to the associated access rights within PACS and EACMS.

10. Ensure that policies and testing procedures for all electronic communications protocols are afforded the same rigor.

11. Perform regular physical inspections of BES Cyber Systems to ensure no unidentified Electronic Access Points (EAPs) exist.

12. Review all firewall rules and ensure access control lists follow the principle of “least privilege.”

13. For each remote cyber asset conducting Interactive Remote Access (IRA), disable all other network access outside of the connection to the BES Cyber System that is being remotely accessed, unless there is a documented business or operational need.

14. Enhance processes and controls around the use of manual logs, such as using highly visible instructions outlining all of the parts of the requirement with each manual log, to consistently capture all required information.

15. Enhance processes and procedures for documenting the determination for each cyber asset that has no provision for disabling or restricting ports, to ensure consistency and detail in the documentation.

16. Consider employing host-based malicious code prevention for all cyber assets within a BES Cyber System, in addition to network level prevention, for non-Windows based cyber assets as well as Windows-based cyber assets.

17. Implement procedures and controls to monitor or limit the number of simultaneously successful logins to multiple different systems.

18. Implement procedures to detect and investigate unauthorized changes to baseline configurations.

19. Ensure that all commercially available enterprise software tools are included in BES Cyber System Information (BSCI) storage evaluation procedures.

20. Enhance documented processes and procedures for identifying BCSI to consider the NERC Critical Infrastructure Protection Committee (CIPC) guidance document, “Security Guideline for the Electricity Sector: Protecting Sensitive Information.”

21. Document all procedures for the proper handling of BCSI.

FERC Nominees Breeze Through Senate Committee Hearing


The Senate Energy and Natural Resources Committee held a two-hour hearing today to review the nominations of Democrat seat appointee Richard Glick and Republican seat appointee and prospective Chairman Kevin McIntyre to be members of the FERC.  The Committee also reviewed the nominations of Joseph Balash to be Assistant Secretary of the Interior for Land and Minerals Management and Ryan Nelson to be Solicitor of the Department of the Interior.  If you have the time and the inclination, the hearing is archived here.

Each nominee began with a prepared opening statement.  Mr. Glick noted his prior experience on the Hill, working for Senator Cantwell and previously for Senator Bumpers in the 1990s.  He also highlighted is broad experience in the industry, including representation ranging from public power entities and state regulators while in private practice to serving as in-house counsel to utilities such as PacifiCorp and Avangrid.  Mr. McIntyre highlighted his 30-year career which included substantial practice before FERC on behalf of a wide swath of energy industry participants and his “rule of law” approach to reasoned decision making that would be applied if confirmed as a Commissioner.

The questions from both Republicans and Democrats were handled very well by both nominees, with each expressing expert understanding of the primary jurisdiction of FERC, pending rulemakings and matters before the agency, significant policy considerations and challenges to grid reliability, resource integration, and state law issues like renewable portfolio standards.  A few recurring themes appeared that I would note:

  • The recently released DOE Staff Report surfaced in questions by several Senators, including questions about the ability for FERC to express a fuel source preference.  Mr. McIntyre asserted that FERC does not pick fuels, that it should be open to the science, and that reliability and economics also play a key role in FERC’s role in market design and operation. Mr. Glick provided complimentary responses, noting that the Staff Report indicated that the significant loss of baseload generating resources hasn’t impaired reliability to date, but is something that FERC needs to keep its eye on.
  • Both nominees noted the ongoing price formation and transparency and energy storage initiatives that are already underway.  While not expressing opinions on how they would vote on the matters currently before FERC, both indicated that these policy initiatives were important and (in my view) signaled they would be continued under the new Commission.
  • Both nominees acknowledged state’s rights in the context of the right to establish renewable portfolio standards, dealing with state specific issues like nuclear generation, and in infrastructure siting determinations.  Mr. Glick cited to the recent Hughes Supreme Court decision, and Mr. McIntyre provided a complementary follow up response, consistent with his “rule of law” comment provided in his opening remarks.

All in all, there were no gotcha moments and no zingers from either the dais or the nominees.   The two nominees expressed great poise and knowledge to every question thrown their way, and neither expressed what I would consider to be extreme positions in relation to criticisms lodged against other administration appointees to the Department of Energy, Environmental Protection Agency, or other agencies that impact the energy business.

Chairwoman Murkowski indicated at the end of the hearing that it was her desire to move to a vote quickly, and told Messrs. Glick and McIntyre “we can’t get you there fast enough.”  Let’s hope that is true and it takes less than the time it took for the confirmation of Messrs. Chatterjee and Powelson. 


DOE Staff Report Released


Last night, the Department of Energy released its highly anticipated Report to Secretary of Energy Rick Perry on Electric Security Markets and Reliability.  A copy of the DOE Staff Report, along with Secretary Perry’s cover letter, can be found here:

Four months ago, Secretary Perry issued a memorandum directing a study to explore three main topics: (1) the evolution of wholesale electricity markets, including the extent to which Federal policy interventions and the changing nature of the electricity fuel mix are challenging the original policy assumptions that shaped the creation of those markets; (2) whether wholesale energy and capacity markets are adequately compensating attributes such as on-site fuel supply and other factors that strengthen grid resilience and, if not, the extent to which this could affect grid reliability and resilience in the future; and (3) the extent to which continued regulatory burdens, as well as mandates and tax and subsidy policies, are responsible for forcing the premature retirement of baseload power plants.  As most are aware, the Trump Administration has a desire to preserve the coal industry (with an emphasis on the jobs it creates), and the recent spate of baseload coal generating plant retirements and litigation involving state policies has put new pressures on wholesale markets.

The results of the DOE Staff Report are perhaps a little different than what were originally expected and more importantly, telling of how FERC may advance its future policies toward market design reforms, grid resilience, resource preference, integration of variable energy resources, fossil fuel delivery and availability, changing and expanding federal and state policy objectives, and least cost vs. reliability concepts.   At the same time, some of these initiatives are already underway in FERC Rulemakings issued over the last three years, all under the general auspices of the price formation and transparency improvements to organized markets.  The DOE Staff Report recommends policies be created to acknowledge the importance of fossil fuel baseload and responsive peaking generation as critical to the reliability of the electric grid.  It will be interesting to see whether and how the recommendations are advanced, and how the DOE Staff Report shapes FERC’s agenda over the coming months.

So, get a large cup of coffee this morning, and start reading….

FERC Quorum to be Restored Monday August 14

March Counsel

Commissioner Chatterjee was sworn in yesterday.  According to news reports, Commissioner Powelson has indicated that he will come to D.C. on Monday and along with his colleagues, get started on resolving the large backlog of proceedings right away.  It is still unclear when FERC will commence its regular monthly Open Meetings, but typically, there is not a meeting in August even when there was a quorum.   FERC can and often does use notational voting, however, meaning we could start seeing substantive orders soon.

There have been no formal announcements on any advisory staff positions yet for either new Commissioner or whether President Trump will seek to replace Chairman LaFleur with one of the two new Republicans as we wait for Kevin McIntyre to be confirmed and seated as the new Chairman.  There are also no indications that Chairman LaFleur would leave FERC if removed from the Chair position, as was the case with Commissioner Bay.

Interestingly, Commissioner Powelson has indicated that he will commute from his Pennsylvania home (west of Philadelphia and northwest of Wilmington) to D.C.   He commuted to Harrisburg as a Commissioner on the PUC as well.

FERC Commissioners Confirmed!

March Counsel


At 6 PM today, I posted an insight explaining that while Majority Leader McConnell successfully pushed through approximately 65 of the President’s appointees pending before the Senate, the Chatterjee and Powelson nominations were NOT among them.  Several other key positions, including FCC and CFTC Commissioners were cleared. These nominations were made “on block,” meaning there was no individual vote or floor debate as to each.  It also meant that the Senate Democrats consented to those nominations.   I sent the post, jumped in my car and headed home, thinking that we’d be without a quorum for several more weeks until the Senate reconvened.

Shortly before 7 PM, the Senate did push Chatterjee and Powelson through!  The Washington Examiner and Politico separately reported that Senate Democrats were holding up the FERC nominations.   The reports also indicated Democrats were first waiting for the official nomination of Richard Glick to materialize, which it did.   Apparently, content with promises that the Glick nomination would be brought to vote, the Chatterjee and Powelson nominations were brought to the floor of a largely empty Senate chamber.

So ends the drama of the quorum-less FERC and my one hour of being #FAKENEWS.

Senate Confirmation Process Update – Will the FERC Quorum be Restored Today? (Spoiler: Only Time Will Tell…)


At 11:45 A.M. today, the Senate will begin considering Dan Brouillette to be Deputy Secretary of Energy.  Mr. Brouillette participated in the same committee confirmation hearing as prospective Commissioners Ronald Powelson and Neil Chatterjee in late May.  Having now cleared more high profile nominations such as Director of the FBI, a position on the NLRB, federal judicial posts and many military posts, it may finally be the time that we see the FERC’s quorum restored.   At least one official Senate website is posting that “additional votes are possible during Thursday’s session.” 

No one is expecting controversy on these two candidates.  I, for one, am surprised it has taken this long, considering (i) that Mr. Chatterjee was sponsored by the Majority Leader, Senator McConnell, and (ii) the economic development that is being held up by a failure of FERC to move swiftly on pending infrastructure applications.

The Congressional Record today also officially acknowledges the receipt of the nominations of Richard Glick and Kenneth McIntyre from the White House.  As expected, Mr. McIntyre will be considered for two terms, one expiring June 2018, and the other in June 2023.  The Energy and Natural Resources Committee has scheduled (link here) a committee meeting to consider these nominations on September 7, 2017.   



SPP Votes to Terminate Regional Entity Responsibility; NERC Compliance to Transition by End of 2018


On July 25, the SPP issued a press release publicly announcing that the SPP and NERC have agreed to terminate their Electric Reliability Organization delegation agreement.  Action had been taken by SPP’s Board of Directors on Sunday, July 23rd.  This move will dissolve the SPP Regional Entity, one of NERC’s eight delegated Regional Entities responsible for compliance and enforcement for the Reliability Standards promulgated under Section 215 of the Federal Power Act.   This termination and transition to a new Regional Entity (or Entities) is contemplated to be complete by the end of 2018.

 A copy of SPP’s press release can be found here.

Nick Brown, CEO of SPP stated that the goal is to focus on strategic objectives, including Western expansion, wholesale market operations and transmission planning of the SPP RTO.   SPP was the only remaining organization to operate as both an RTO and as a Regional Entity.  As such, it could both recommend fines and pay fines, raising concerns about independence.

The SPP Regional Entity utilizes 24 of SPP’s employees and has 120 entities under its compliance monitoring and enforcement program.  It is unclear at present which of the other Regional Entities these companies will be transferred.  It is quite possible that the answer will vary within the SPP footprint.


The Fifth Golden Ticket Has Been Claimed: Final FERC Commissioner is Nominated



(Couldn’t resist; rest in peace Gene Wilder, you were a brilliant actor and this was one of your finest)

Thursday evening, President Trump announced his intent to make his final appointment to the Federal Energy Regulatory Commission: Kevin McIntyre, a well-respected Jones Day energy attorney.  Mr. McIntryre’s name has been out there for several months; now it looks like it’s going to happen.  Once confirmed, McIntyre is expected to be tapped as the Republican-appointed majority Chairman.

 According to The Hill, Trump is asking the Senate to confirm McIntyre to two terms, through 2023.  This sounds more bombastic than it really is; Commissioner terms are staggered, 5-year terms.  Ron Powelson is getting a seat that expires in June 2020, Neil Chatterjee is getting a seat that expires June 2021, Richard Glick is getting a seat that expires in June 2022, and current sitting Chairwoman LaFleur’s seat expires in June 2019.  That leaves the seat that expires June 2018.  The law says that “any Commissioner appointed to fill a vacancy occurring prior to the expiration of the term for which his predecessor was appointed shall be appointed only for the remainder of such term.”  And it would be very awkward for the Trump Administration to tap an established energy practitioner to lead the Commission, only to have to work for his reappointment after a short period of time on the job.  Perhaps not that appealing to Mr. McIntyre either.

For you legal eagles out there that read my insights, the details of FERC appointments can be found in 42 U.S.C. § 7171(b).

As we wait for the Glick and McIntyre confirmation hearings, I watch the Senate Executive Calendar daily (ok, this may be a bit of a stretch, I do have a life), wondering if we will see action on Chatterjee and Powelson before Congress takes the month of August off.  We can only hope.  FERC’s backlog is growing. We need a fully functioning Commission as soon as possible.

And then there was one….

Colette Honorable is leaving the Federal Energy Regulatory Commission tomorrow, at the end of her term.  While some Commissioners stay on beyond their five-year term until his or her position is filled, that is not the case here. Based on my scan of her Twitter feed (and she’s quite the Tweeter!) Commissioner Honorable hasn’t announced plans for the future.  That leaves the FERC with one sitting Commissioner and a significant backlog of cases to handle. Yesterday, Senator Lisa Murkowski, chairwoman of the Committee on Energy and Natural Resources, issued a statement, which in relevant part said:

“Commissioner Honorable has served with distinction. While I did not agree with her on every issue, she has been an outstanding regulator, and I wish her and her family well. Although I understand and respect Commissioner Honorable’s decision to step down this week, her departure again underscores the urgent need to re-establish a quorum at FERC.

FERC’s work is increasingly important. Getting the agency back to the normal course of business remains a top priority for me. I will continue to push for a confirmation vote for Neil Chatterjee and Robert Powelson, who were favorably reported by the Energy and Natural Resources Committee earlier this month on a strong bipartisan basis. Even with Commissioner Honorable’s departure, FERC will be able to get back to work on the day these qualified nominees are confirmed. I hope my colleagues among the Senate minority will join us in enabling a quick vote for Mr. Chatterjee and Mr. Powelson.”

So, hopefully, help is on the way soon.  Powelson and Chatterjee have been sitting on the Senate’s Executive Calendar for approving nominees (along with 26 other nominees dating back to mid-May).  Senate confirmation of these two Republicans is not expected to be contentious.   

After Commissioner Honorable announced her departure, Senate Democrats began to float the name of Richard Glick as her replacement.  Mr. Glick, serves as Democratic General Counsel of the Senate Energy and Natural Resource Committee.  So, like Chatterjee, he is a known commodity on the Hill.  Beyond his work on the Hill, Mr. Glick’s resume includes a long stint with Iberdrola Resources, as a policy adviser to Energy Secretary Bill Richardson under the Clinton administration, and as a legislative director and chief counsel to Senator Dale Bumpers.   Given these bona fides, Mr. Glick should be able to navigate a position at FERC quite well.

Reports on AP News this morning suggest that President Trump is going to appoint Mr. Glick in the near future. No official news yet, and there would certainly be a further waiting period until hearings are scheduled or confirmation (recognizing that the Senate has very few work days in session between now and Labor Day). 

Assuming all the above happens, that still leaves the fifth seat to be filled, which would be more important now given that a Commission of LaFleur, Glick, Powelson and Chatterjee would be a 2 Democrat/2 Republican Commission.  The final appointee could be the one that is going to be the new permanent Chairman. I personally would not be surprised if Mr. Glick was paired up with the final appointee for confirmation hearings. 



ExPetr/Petya and the NERC CIP Reliability Standards


This week brought another global ransomware scare, dubbed “ExPetr” and/or “Petya”, similar to the “WannaCry” attack a few weeks ago.  By most accounts, the U.S. energy industry was largely unaffected by this round of worm-based ransomware intrusions.   There are important lessons to be learned that tie directly back to the NERC CIP Reliability Standards though, so I thought I’d provide a quick insight on it for those that do not regularly participate in NERC compliance activities.

So, in a nutshell, what’s going on?  ExPetr/Petya used a hacking tool called EternalBlue, allegedly developed by the National Security Agency, to exploit Microsoft Windows and lock up computer systems until a ransom of $300 in untraceable Bitcoin is paid.  Of course, paying the ransom is no guarantee that your system will be unlocked.  Based on the reports so far, ExPetr/Petya was allegedly introduced through a Ukrainian company called MeDoc, a legitimate financial tech company which sent out an update on June 22 to its tax preparation software to its customers with the malware embedded in it.  Once it was released, it spread through Ukraine and Russia, and then globally, including to U.S. companies like Merck and the law firm of DLA Piper. 

Unlike less sophisticated clickbait malware, which usually requires a user to enable the worm through accessing a malicious website link, this is a patch/configuration based hack that will happen when the update is added.  Microsoft released a patch for this vulnerability months ago that would close the Windows entry point, but if your IT department is not updating Windows religiously, or allows users with their own updated equipment to access computer networks, the worm can gain access and once there, spread like a wildfire.  And when you deal with hundreds or thousands of connected devices, it the odds are against you that one will slip through the cracks and not be updated. 

That brings me back to the CIP Reliability Standards.  First and foremost, CIP Reliability Standards only apply to a subset of computer systems in the electric utility industry: those systems designated as Bulk Electric System (“BES”) Cyber Systems.  Among other responsibilities, the suite of CIP Reliability Standards are designed to require entities to configure such BES Cyber Systems in a manner that (1) eliminates access through ports or other entry points to critical systems to operate the grid, (2) reduces the number of software applications on such critical systems, (3) reduces the number of users with physical or electronic access to the systems, and perhaps most relevant in this context, (4) requires regular updating of systems through detailed patch management processes, including a process to evaluate patches to ensure they come from a reputable source, that the patch works as intended in a test environment, and that the patch is installed correctly and promptly. 

Would application of the CIP Reliability Standards to all computers and associated systems (not only BES Cyber Systems) have stopped this worm?  The answer is probably, and it did seem to serve its designed purpose as to the BES Cyber Systems.  First, regular patching of Windows would have closed the vulnerability exploited by ExPetr/Petya had it been done promptly.  The patch was released by Microsoft in mid-March.  A typical CIP-based patching protocol would have permitted the entity to have approximately 30 days to locate and find a patch, another 30 days to test it in a safe non-production environment, and then another 30 days to verify and install the patch.  The full 90 days envisioned here would get you to about mid-June, so it might have been completed just in the nick of time. 

Second, a typical CIP-based patching process would also require testing, as noted above, so test installation of the MeDoc accounting patch in a safe environment may have resulted in a lock-up situation and alerted the IT user to take further action before installing it in a production environment that could affect all interconnected systems.  CIP standards require that users validate a source for patching; here it was the software company itself that released the patch; that would typically be a reputable and acceptable source.  That’s a bit scary.

Third, the CIP Reliability Standards would have likely declared an application such as an accounting/tax software as not necessary to be on a system that could affect the reliability of the BES.  By limiting and isolating the BES Cyber Systems, it protects against malicious attacks such as worms entering important systems.  Conversely, it does nothing to protect against attacks on the rest of a company’s IT, which can bring normal business operations to a grinding halt, as it did to several companies this week.

The CIP Reliability Standards are all about a “defense in depth” approach to protect our critical energy infrastructure.   There are important lessons to be learned here that validate the NERC’s program and the importance CIP Reliability Standards.   

Finally, I note that FERC had a technical conference on June 22, touching on further revisions to the reliability standards enforcement program as it nears its 10th anniversary of mandatory applicability.   The archived stream of the conference can be viewed here for the next 3 months.