Throughout my thirty-year career in IT and cybersecurity, I carried with me many lessons learned from my fourteen years in the Naval Nuclear Program (the “Program”).
Hyman G. Rickover, father of the U.S. Nuclear Navy, was a brilliant, demanding, and eccentric leader. Rickover was one of only four people to have been awarded two Congressional Gold Medals. His combination of rigorous demands for reactor safety and technical achievement was unparalleled, and his accomplishments included the U.S. Navy's record of zero reactor accidents. I am not sure if this is still the case, but at the time, U.S. nuclear warships were welcomed in any “friendly” port in the world.
One of the major tenets of doing business in the Program was complete and exacting documentation. There was no room for mistakes.
I witnessed a young, promising engineer lose his job when his technical letter had a misplaced decimal point for a resistor’s value. The litmus test was the scenario where a nuclear submarine experienced equipment problems while under the polar ice cap. If something as small as resistor was needed to fix the problem and it was the wrong value, it could result in the loss of the sub and its crew. To track the voluminous documents that passed between my organization and Naval Sea Systems Command (NAVSEA) my programmers developed a tracking program. The database program was a running “log” that followed the serialized letters from NAVSEA and my company’s serialized response.
Starting with my CISO role at the American Red Cross that spanned September 11th through Hurricane Katrina, I began my own letter log database. It was a simple Excel spreadsheet with a running log of my correspondence. Each memo or letter was serialized in the format of “CISO.Year-Sequential Number” or CISO.2019-.001, followed by the date of issue and the document’s title. The serialized number was used as the document’s file name and each was stored in a protected file folder. This scheme allowed for speedy retrieval in the event I needed to recover a document (although now, of course, there are more efficient software platforms available).
I also color-coded the document descriptions in the spreadsheet, which included using a red text/font coloring for documents involving investigations. The sequential numbering scheme were incremented and carried over from year-to-year and only changing the “year”. At the end of the year, or sooner if requested, I could easily report on my body of work by the various categories of correspondence.
One of the more delicate situations we face as CISOs is conducting investigations involving internal matters. I always stressed to my team that when we conducted an investigation, we needed air-tight and thorough documentation. We owed it to the “perpetrator” as well as management to provide as complete an investigation as possible. After all, people could lose their jobs!
During my two years as CISO at the largest healthcare provider in the Washington D.C metro area, my team and I conducted over 200 cyber-related investigations! Most involved inappropriate use of the Internet and some were HIPAA violations. I had to testify in court six times, so it was paramount that these cases be investigated fully and thoroughly documented.
I developed an investigative template showing a logical relationship that started with the relationship of the offending IP address with computer name and then associating the computer name with the user ID. The user ID was then linked to a specific individual and then we showed the ID being logged on during the time period of the inappropriate usage. From then on, we could use other internal security systems to augment the baseline user-to-computer and time stamp data already established. For example, we used “8e6” for our web content filtering that would show the inappropriate websites visited in a date and time stamped manner. We also had firewall data and evidentiary output from our Data Loss Prevention system. The key was developing near-foolproof documentation that best illustrates to management and potentially legal systems the factual nature of the investigation.
Nothing validates the effectiveness of a well-run cyber security program more than having robust documentation. External and internal auditors appreciate high levels of documentation. In my last CISO role before retiring, we set up a program of ad hoc as well as monthly, quarterly, and annual audits and reviews we performed on our networks and systems. Each of these reviews was documented using the letter log and serialization scheme identified above. When the auditors showed up, it was trivial to “open our books” for inspection and retrieve the reports.
Many of these network and system reviews were the result of recommendations by auditors. For example, we conducted a monthly review of the “firefighter ID” in SAP. The firefighter ID is a powerful, temporary user ID that grants the user exception-based, yet regulated, access. The firefighter ID is created by a system administrator and assigned to users who need to perform tasks in emergency or extraordinary situations. It was essential that the use of this ID be reviewed on a routine basis to ensure it was not abused in some way.
The Bottom Line: Documentation Covers Over Other Shortcomings
In most of my CISO roles, the cyber security programs were underfunded and lacked resources. This is not an exaggeration. At the American Red Cross, there were four of us to cover 95,000 employees and volunteers in diverse lines of business - corporate, disaster services, biomedical services, chapter services and armed forces emergency services. At the healthcare system comprising over 25,000 employees, 8 hospitals, and 56 professional offices, seven members of my team were dedicated solely to managing access controls to our networks and systems. This left me and one other security professional to handle all other cyber security duties.
Despite these shortcomings, I was able to build reasonably effective and respected security programs which were successful in large part to a robust and rigorous documentation strategy.
Share This Article