Security satori.
I have found the Metrics Prophet for our times, and his name is Andrew Jaquith.
I stumbled home yesterday from work, sleep-deprived, jittery, and feverish from an oncoming cold. I tucked myself into bed, hoping to sleep—but I could not sleep until I had read Security Metrics cover to cover. It was That Good.
Now, either that makes me the biggest saddo anorak west of the Pond, or it means Jaquith is an extraordinary writer about what would otherwise be an extremely dull subject. I would of course prefer to think it’s the latter, and I’m sure he would too.
First off, his writing is chock full of playfulness and amusing literacy, from the literary nods ("Call me Analyst.") to the rimshots ("… the top and bottom 50% are divided by—wait for it—the median!").
Secondly, his metrics are for the most part accessible, meaning that as soon as I see them, I think, “Yeah, I could get those!” And a whole lot of them are ones I’d already thought of, but there are a few gems in there that were like little Altoids in my mouth, that made me sit up and go, “Whoa.”
For example:
- Tracking the number of change control exemptions, including by business unit—to identify the “cowboys” in the organization.
- Correlation of password strength with training latency
- Cycle time to deprovision users, by system type
- “Toxicity rate” (I love this concept) of customer data and ratio to all other data
His section on analysis techniques is also well-written, but I think he misses another reason to avoid some instances of taking the average or median of data instead of focusing on the outliers. The reason to avoid averages is this: humans don’t think that way. And by “think,” I mean using that funny combination of flawed psychology and intuition to make risk decisions. When an executive is making a decision on security, she doesn’t give an auditor’s ass what the average number of incidents is across the country, or the industry. She only cares about numbers that can indicate what she might expect for her organization. Give someone an average and he will immediately try to apply it to his own situation, either by going into denial ("I’m sure we’re below the average") or overestimating the risk ("That means it’s sure to happen to us.").
So I think it’s important to know when to avoid averages and medians in analysis when there is a good reason to believe that your decision-maker will try to apply those numbers to calculate probability for risk assessment. And if your purpose in showing a median is to lead in to discussing the outliers, for heaven’s sake, just go straight to the outliers; don’t make him do the math in his head while he’s looking at a number on a page.
In his chapter on visualization techniques, Jaquith really does me a huge favor by giving loads of hints on how to bend Excel to my will. When I’m doing my annual security risk management report, by far the largest time-sink is just plain trying to get the numbers and charts to say the right things. His “rules” on making things easier on the eyes are great—although I would tweak him about not following his own rule about previewing the charts in black and white first; his Figure 6-14 is well-nigh unreadable on the printed page.
One thing I do know for sure after having read this chapter:
I’ll never get a Treemap Chart; I never hope to see one.
And I can tell you this right now: I’d rather die than make one.
Another side note: I’m sorry, but Figure 7-2, the “Logical Model of IT Security Controls (Level 2),” makes me think immediately of the Hamster Maze of Pain.
However, Figure 7-4, “Metrics Sources in the Data Center,” should be printed out as a wallet card and laminated. Boy howdy, is that one useful.
The final chapter on designing security scorecards is good, but I will probably never make it as far as a Balanced Security Scorecard. I will probably spend the next several years just trying to automate the collection of my key metrics.
And that final chapter just stopped short and dumped me off the end of the book, without so much as a fare-thee-well Final Overall Summary. It just stopped, and without another word, put on its clothes and went home, and I was left staring at the Index. But I’ll get over the disappointment.
One other thing, though: I was terribly gratified and relieved to read that I’m not the only one who doesn’t think “asset value” can be practically calculated. All the risk-assessments-in-a-box I have seen have started off with an inventory and asset value, and hell, how am I supposed to compute the asset value of a firewall? In terms of its hardware cost? In terms of business loss if it’s a single point of failure? In terms of the criticality of data it passes or blocks?
In short, I still have some burning questions, but they’re the good kind: they’re the kind you’re left with after a stimulating barstool conversation with a very smart partner. My overall enlightenment level has been raised, and that is a very fine thing indeed.
Posted by shrdlu on Saturday, April 14, 2007
(4) Comments • Permalink •
