Metrics Revolutions.
(Can I get Keanu Reeves over here to frown broodingly at a screen and say, “That’s odd ...“?)
Following on the previous post, Andrew Jaquith brings his alliterative and quantitative skills to the flat screen. How thrilling is this?
He makes a lot of sense with his post, but I’d like to pick apart some of his examples and try them on for size in MY world.
Likewise, for learning and growth, we want to spread responsibility for security, equip employees with the right security knowledge and skills, and promote adaptability in the face of changing threats. These themes imply the following typical objectives:
- Delegating responsibility for authoring user activities to business units
- Increasing collaboration between IT security and business units
- Ensuring effective levels of security certifications for security staff
- Promoting security awareness throughout the organization
- Integrating secure behaviors into employee’s everyday activities
- Ensuring that security features are easily understood and adopted
- Heightening awareness of emerging security threats
- Exploring discretionary security frontiers
- Giving employees the skills needed to properly handle security incidents
I’ve purposefully listed the objectives for both perspectives (internal process and learning/growth) as objectives, not metrics, because they illustrate the behaviors that organizations should be encouraging. Every one of them can be readily mapped to process metrics — key indicators — that show whether an organization is achieving those objectives.
Let’s try some process metrics for those then, shall we?
Delegating responsibility for authoring user activities to business units - I’m not completely sure I understand this, but okay, assuming you can measure their activity somehow, you can argue about how much they were doing as a result of having it delegated to them.
Increasing collaboration between IT security and business units - How do you measure this? By the number of business unit keggers that IT security gets invited to? By the decreasing number of flames being sent back and forth with a Cc: to senior management?
Ensuring effective levels of security certifications for security staff - Bwahahaha. Define “effective” and then we’ll talk.
Promoting security awareness throughout the organization - Okay, I do this now through the number of newsletters issued, the number of training classes provided and the number of attendees, and the results of an annual OCTAVE survey.
Integrating secure behaviors into employee’s everyday activities - What kind of secure behaviors are we talking about? Not going to MySpace? Not opening spam attachments? I suppose you could count the number of times screens were left unlocked, or if you were really ambitious and nosy, you could count the number of outgoing emails that should have been encrypted but weren’t. Other than that, though, I’m metric-less.
Heightening awareness of emerging security threats - How is this different from the awareness activity just above? Do we count the number of CERT advisories we mail around? Do we do another survey?
Exploring discretionary security frontiers - I think this means researching new security technologies, but I’m not sure. Okay, so we measure spending dollars and dedicated man-hours, or number of pilots resulting in new procurements, or something. Help me, Jean-Luc!
Giving employees the skills needed to properly handle security incidents - You mean, other than giving them a phone number to call? What are some metrics for this one?
Don’t get me wrong: I’m not arguing that these aren’t great objectives, or that we shouldn’t be focusing on security behaviors. Far from it. I’m just having trouble making the leap from objective to mapped metrics. Then again, Jaquith is doubtless much smarter than I am. I am Only An Egg.
on 11/19 at 03:07 PM:
)
There are definitely some very different worlds when it comes to “security metrics.“ It really feels like people come from an auditing, managerial, operations, training, or “trenches” perspective, and as such their actual metrics and view of metrics are vastly different. I don’t think someone from auditing will ever be able to explain their metrics properly to anyone outside of auditing, likewise training function metrics will never be useful to people in the trenches, and vice-versa.
Of course, then there’s the academic/theoretical approaches versus those of us stuck in reality. Honestly, I’ve yet to work in a company or have close friends in a company that measures something as easy as unscheduled downtime or uptime of systems. And I’ve worked in companies whose bread and butter are the systems they offer. That stuff is simple compared to “security metrics,“ but even those are not utilized properly, and sometimes not at all. (Then again, maybe that’s coupled with my frustration in my recent jobs, hehe.)
In further contemplating this subject, I do like the approach of defining catagories, then objectives, then determining metrics to measure for those objectives.
Switching gears, some of those learning & growth objectives (metrics) listed above make me quiver in a not-so-good-feeling way. Giving employees the skills needed to properly handle security incidents? I hope that means security employees or technical employees; or perhaps that means regular employees able to challenge strangers or handle mysterious emails themselves. I’d hate to try to empower my regular users with enough knowledge to troubleshoot their own spyware infections and/or delete rogue-looking files in &#xsy;stem%/system32/. And I’ll echo skepticism about security certifications. I’m sorry, but some of the most talented people in our area don’t have certs, and some of the moronic ones do.
I dunno. Sometimes I feel that metrics only serve the purpose of making people feel better, whether they have any substance or not. Here’s a trend analysis of our security awareness programs which shows a huge upward movement which reinforces these objectives listed above. Two days later, a database is hacked into and execs wonder why this occurred while our metrics looked so good.
Anyway, to close off my rambling, I’d like to just point out that some of the companies who have the best-looking metrics are the ones who have the least measuring. “We have had zero security incidents in 5 years!“ “Do you have an IDS, look at firewall logs, deal with spyware, or audit user permissions?“ “Well, no, but we don’t have to because we don’t have any incidents!“ Effective security measuring needs to highlight and bury that problem.