Metrics that Matter

Security Professionals are swimming in data.  One of the most often discussed topics among CISO’s involves developing meaningful metrics. 

1.      What Metrics are needed to monitor the security program from the CISO position?

2.      What metrics are necessary to demonstrate diligence to Leadership and the Board of Directors?

3.      How does an organization demonstrate diligence to the public?

In this post we will examine items 1 & 2.  Number 3 is probably best demonstrated by being transparent with the data you collect and maintaining compliance (PCI, SOC2, GDPR, HIPAA, etc.)  as validated by an independent auditing firm. Also note the data an incident responder values is probably not relevant to Senior Leaders, so you will not be covering details like anomaly detection or network data flow.

One common theme I’ve heard is “Measure Outcomes”, but what does that mean?  For some that can be a simple as “Am I in the news?” For other’s it may be demonstrating a reduction in data loss or a reduction in laptops being re-imaged.   Every organization is going to have different answers to the question.  The following are some ideas that we have found useful without needing copious explanation.  Senior Leaders generally like quick and efficient views into performance, too much information or detail may result in the dreaded TLDNR.  Adding a “brief” description for each may be appropriate but do your best to keep it simple.  Ideally you will have an opportunity to walk through the metrics periodically and answer questions.

1.      Depending on your industry, you may fall under some compliance requirements. PCI if you are in retail, HIPAA for Healthcare, and GDPR or CCPA depending on the markets in which you do business.  I would encourage you to present these at a high level, indicating the areas where you may be deficient.  This is both a scorecard and a tool that can be leveraged to gain support for new initiatives. Be sure to identify the problem and solutions that you are exploring.  For example, if you are a retailer struggling to implement strong encryption at store locations be sure to have the supporting information for P2P capable payment terminals, or firewalls with improved VPN capabilities.  PCI in particular has a lot of overlap so you may find that a single investment to upgrade firewalls helps solve issues in requirements 1, 4 and 10.  A similar approach can be taken with any of the requirements/standards you follow.

Compliance Metric

2.      With more and more organizations choosing to leverage SaaS software, it is increasingly important to have insight into how your partners are protecting your data.  Remember, though they are responsible for the technical implementation “you” are ultimately responsible.  For most organizations that will mean administrative controls in the form of Contractual Provisions and ongoing compliance with Industry Certifications.  The #1 red flag for me is a provider that cannot produce relevant evidence of a diligent security program through something like a SOC report.  Being able to document and demonstrate to leadership the providers who do not meet these minimum requirements will help you the next time your HR department wants to send employee data across the globe to a startup company.

Third Party Risk Metric

3.      Data Security is no longer just an IT problem.  All of an organization’s members have a role to play.  Employees need to be made aware of their responsibilities as it pertains to data. They also need to be trained on how to spot potential digital dangers.  Be sure to include a section that highlights what is being done in this regard.  I believe it should include both the creation, distribution and education of policies as well as traditional Cyber training like what not to click. 

Governance Metrics

4.      On the technical front Vulnerability Management should be highlighted. In addition to patching, I like to include Penetration Testing/Code Review in this section. Trending over time is always a good way to demonstrate improvements as well as the changing landscape.  Be prepared to discuss any ups or downs and use it as an opportunity to demonstrate emerging risks.

Vulnerability Management Metrics

5.      Significant time and money are spent implementing security solutions, but in order for them to be effective they must be maintained over time.  There have been numerous times in my career when I’ve discovered that a solution which I was expecting to provide some protection, had failed (another reason why Defense in Depth is critical).  Whether it is writing an App to drop an EICAR on a system to test AV, or automating connection attempts to validate FW segmentation I encourage you to look at your critical controls and develop a mechanism to alert you when they are failing.  It’s always better to know this before you have an incident or when you’re sitting with an auditor.

Security Tool Coverage Metrics

6.      Identity.  At a minimum you should be able to report on your use of MFA for remote and sensitive usage as well as data accessible via SaaS/cloud.  Other areas to focus on include the proper provisioning and deprovisioning during the user account lifecycle.

Identity Management Metric

7.      Data Protection and Resiliency.  Each of these can be broken down into many sub categories, but I think what most leaders are interested in is a) “is my organizations data protected?” and b) “how quickly can we recover from failure?”.  For this, reporting on the number of critical systems using encryption, the number of systems being backed up, and demonstrating that your back ups are working is critical.

Data Security Metrics

8.      Lastly, it’s important to have some information around incidents.  The metrics captured here are more or less a scorecard of how you handle the more “normal” threat events and incidents.  Trending this data over time can also help highlight deficiencies or tell the story of investments that drove down the cost of incidents.

Incident Metrics

As mentioned earlier, there is no one size fits all solution for reporting on the state of your organizations Information Security practice, but hopefully some of this information provides ideas to help you identify and communicate some of the challenges we face each day. With a little luck you can also demonstrate the value Information Security Teams bring to your organization.