Metwics.
(Imagine Peter Cook from The Princess Bride saying the title.)
Metwics is what bwings us togethaah today. That dweam wivvin a dweam ...
There’s just no satisfying way to measure how secure you are, or how well you’re doing your job as a secuwity pwofessional.
An executive asks me: “So, if you let me have this wireless access point, how insecure will we be?” (WAPs are today’s modems.)
And I have to say: “I dunno. Five? Red? What scale do you want? Should I draw it in the form of a graph?”
Yes, I know there are extremely sharp people who are valiantly trying to tackle this issue, and I wish I could be a fly on the wall at Metricon.
The way I see it, there are five (three, sir) main questions that I need metrics to be able to answer:
1) How well are we tackling the threats we know about? How much are we preventing?
2) Due diligence: How well am I covering all the various bases that a security officer is supposed to address? Am I expending the right amount of effort and attention on application security reviews, awareness, training, system configuration standards, business-enabling engineering, legal compliance, and monitoring and detection? Am I tackling the newest security issues at the right time—not too late, but not so early that they don’t matter yet to the organization?
3) How much value am I delivering for the money the organization spends on security? How can I justify the budget I’m requesting?
Number one is a pain in the butt to try to answer. You can talk about the attacks you actually see, and you can point to statistics from other similar organizations on what THEY saw, and you can even wave around a bunch of C-level magazine reports on the latest threat activities. But that doesn’t tell you anything about what didn’t happen just because nobody felt like trying it. The most popular n00b meTriC, antivirus software statistics, is completely useless. That’s only a reflection of how well your particular vendor is updating the signatures, how quickly you’re getting them (and if it’s automated updating, that isn’t even a reflection on you at all), and how many adolescent pukes are spending time trying to impress their buddies or Make Money Fa$t.
n00b meTriC number two is time to patch. Depending on your particular organization, that might just indicate how persuasive you are at getting the sysadmins to put in the patches, or even how homogeneous your systems are (because if some patches break some machines, your deployment is going to be spotty at best). Sure, it’s a key indicator for auditors, who want to see that you’re keeping up. But it’s not an indicator of how you, personally, as a security professional are doing your job. Shoveling patches these days is like shoveling coal.
n00b meTriC number three is bad passwords. Yes, it’s a way to measure your vulnerability, but let’s face it: you only need one bad password. It’s an unreasonable measure of protection. It’s more a measure of how your awareness program is working and whether you were able to configure your systems to enforce password quality (good luck on that mainframe). And the number of bad password attempts is just the number of bad typists.
The problem is, unless you’re a bank, you don’t really know for sure how big a target you are. And you don’t know how the landscape will change tomorrow if you become a bigger target. So you don’t actually know what you’re preventing, except for the bits you see stuck in your filters.
Number two is comparatively easier to tackle. You can point at the legal requirements, you can make lists of what you’re complying with, and you can collect data on what similar organizations are spending their time on, in what proportions. Assuming that you’re aiming for conformity, you can demonstrably get there, and prove to your bosses that you’re keeping up with the Joneses. Listing the things you’ve done is straightforward.
Number three ... well, you can try to fudge it and compare your spending with that of your peers. But it’s hard to make comparisons when your choices are between an incredibly expensive appliance and a bunch of open source software with three frazzled security analysts trying to put it all together. If you’re extremely lucky, you can actually show your executives that spending what you did saved your collective butts, as opposed to the poor schmo across the pond. Usually, though, you can’t tie spending to effectiveness, because you can’t prove what you prevented. In the best case, you can point to the losses you incurred and use a nice formula to show that they need to give you more money to prevent them next time. Let me know if your CFO buys that one.
When you’re trying to show that you’re doing a good job, I think the best you can do is to show that you know what you’re doing. You understand the business, you understand the network, you’ve thought about and written down what you need to do, and you can make a decent swipe at a risk analysis upon demand. You’re aware of what’s going on, and you understand your user base and what they’re likely to do. You know how to handle an investigation without making things worse. You’ve got information, and you know how to use it.
Good luck putting numbers or colors on it, though.
