Metrics reloaded.
Just as tornado season is starting up again, so is the topic of metrics swirling around the sec-blogosphere. rmogull has nothing but reasonableness to offer, but the Best Security Analogy of the Week Award goes to Amrit Williams:
Apparently measuring security is like anal probing. Aliens, with their advanced technology, have cracked the space/time continuum but apparently the mysteries of the human rectum still elude them – security metrics are like the ass of IT, with all our advances it still eludes us.
Maybe we should work on our advances? 
I’d been meaning to write about an excellent metrics talk I heard by William H. Murray, who is as refreshingly pompous, crusty, colorful and practical as they come. He’s one the first people I’ve heard speak who actually made me feel as if there were simple, concrete metrics I could take home and try.
Many of them get echoed time and time again, and appear to be straight IT metrics:
- State
- Populations of systems
- Progress
- Compliance
- Spending
- Attacks
- Losses
- Program
- Service
- Customer satisfaction
- Risk acceptances and risk accepted
Source: Cybertrust slide
Measuring state is a given. Enumerating your network segments, address ranges, system configurations, and the currency of your security-related software. Funny how just knowing what you’ve GOT is a security issue.
Progress. Spending. Service. Customer satisfaction. Wow, how many people have even THOUGHT of trying to measure customer satisfaction as a security metric??
Now, compliance I’m not so sure about, because I personally always find that when I try to use any given tool to measure compliance with policy, I’m actually relying on someone else’s technical/legal yardstick, and I’m having to explain all the cases where I think we’re compliant but the tool doesn’t. It’s a starting point for discussion with management, to be sure, but I think it’s just a wee bit inaccurate.
Measuring attacks and losses now brings us into real security and risk management territory. Murray brought up things like measuring the mean time before failure and the mean time to recovery. Both of those require serious incident reporting skills; there’s no point in talking about mean time before failure if the failure isn’t provably security-related.
Murray also recommended budgeting for losses, to which I said, “WHA-ha-ha-ha??“ It’s a sophisticated form of risk management, but I can’t see that working in an organization with a restricted budget or extra accountability requirements. As I said before, you can budget for first-order costs of incident response, but my management would confiscate my keys and passwords if I tried to suggest that I get a gambling pot and keep what we don’t lose over the fiscal year.
What I don’t think we measure enough is behavior. By that I mean application behavior, user behavior, and system behavior, especially as it translates into network traffic. Never mind what the IDS says: who’s talking to our server, and how often, and why? What does it tell us that web usage spikes at the magic hours of 8 am, noon, and 5 pm? What are we allowing, and what does that say about our business rules and policies?
I’m going to leave a lot of these IT inventory and configuration metrics, like time to patch, to the rest of IT Operations. What I think I should be more concerned about are in the areas of controls, intelligence, and response. I also like to measure awareness: I’ve been using the same OCTAVE survey for two years now, and can finally show some trending (okay, two dots’ worth, but that counts).
And there’s nothing like measurement to highlight the warts in business processes. When you’re trying to define authentication and authorization rules, all sorts of inconsistencies show up that you have to bring to the business to figure out. When they want to delegate, what’s their definition of a suitable candidate, as opposed to an unsuitable one? Whom are they authorizing for unofficial reasons? When you do a user audit and can’t find anyone who can explain why half of the accounts were created to begin with, that sends you right back up to layer 8.
So I’m getting closer to choosing metrics, but I start by getting some data and asking myself: Are we better off now that we know this information? Is it telling us something useful that we didn’t know before? I know how far behind we are on our patching; that’s not a useful metric. But discovering network traffic from an application that none of the developers claimed to know about: THAT’S some useful security intelligence.
Onward thru the fog ...
Posted by shrdlu on Thursday, November 16, 2006
(4) Comments • Permalink •
