For the want of a nail …
Security isn’t pretty.
You hardly ever get the chance to start with a clean slate; you don’t get the advantage of the calm, leisurely 1000-foot overview when you get into an environment. People say blithely that security should be built in from the beginning. And so it should. But in an existing organization, chances are, your biggest job is to solve the problems that are already there.
Network security can be messy in and of itself. I’ve worked in places so big and so dispersed that they had no idea how many external connections they had, much less whether they were firewalled in any way. And just try counting modems; I’ve known users to disconnect them and hide them in their desk drawers when you walk by. (I’ll bet you they’re doing the same thing now with WAPs.) So you sit down and start gingerly pulling threads: why are we permitting this in from the Internet? What’s that IP resolve to? If we cut it off, what else is going to break?
Oh no, they say, you can’t get rid of that. The third-party application will only work with all those ports open. No, they don’t do encryption. So you back off and say, well, okay, what “best practice” CAN we do? The answer is usually depressing.
Application security is even worse. The problems you find there are either the result of ignorance, laziness, crippling legacy dependencies, and/or trying to accommodate the lowest common denominator of user. Some problems come from trying to make the application easier to administer or update. (Thanks so much, Matasano. I was in denial until you ruined my weekend.) You try to ask questions like: can we partition this off? Do we really have to use this data? Does the developer really need to do all this himself? What will happen with the users if we tighten these parameters?
Even assuming you have all the people around you who can give you the answers (and that’s not always the case; legacy apps live on long after their creators have left, and if they’ve been running fine, nobody’s bothered to learn about them since), you’ll find that the applications and infrastructure are all depending on each other in complicated ways, AND there is no central architect who can speak for or manage the whole shebang.
The situation is ripe for a cascading security failure. Because you had a shortage of help desk people, you had to let users choose their own passwords. Because they chose bad passwords, one or more of them were cracked. Because you had to make it easy to get to the apps without additional authentication, the hacker got to one of them. Because the other apps trusted the one app, they got popped too. Because you had no money to buy additional disks, you had no disk space for logs. Because you had no logs, you had nobody monitoring them. Because you had nobody monitoring them, nobody noticed the security breach. Because all your servers had to talk to each other, the hacker was able to spread out. Because you were used to plenty of workstations ignoring your updates, you didn’t notice anything different when they were taken over. And so it goes.
Pick any combination you like. You can’t get rid of SSNs because you don’t have any other enterprise-wide unique identifier. You can’t enforce one security policy because you have to develop a whole new infrastructure to make it possible for people to comply with it. You can’t clean up this database without going through six months of cajoling in committee meetings. You can’t secure this site because the local admin resents your very existence and is protecting his domain. You can’t spend time and money to secure this server because “it’s going away real soon, we promise.”
It’s amazing that we can make any progress at all. The trick is to find a reasonably short thread to follow, and then another one, and then another one, all in the same region. By shoring up the weft, you may eventually be able to address the warp. Except this weave is eight layers deep, and you pretty much have to work on all of them at the same time. (Think I’m wrong? There are plenty of times where puzzles at a higher layer can only be finally solved by tracing the cables.)
My dreams are full of missing horseshoe nails. But there’s only so much I can do.

