When someone claims that one system is “more” or “less” secure than another system, what does that mean?
Advanced sports statistics enthusiasts distinguish between what they call “rate stats” and “counting stats”. Counting statistics are just that – counting the cumulative number of something. Rate statistics reflect the frequency of occurrence of something per something else. So, one way to describe how secure a system is focuses on the cumulative number of attacks and frequency of attacks over time. This creates a good baseline dimension of whether or not there are attacks taking place and if they are how popular a target the system is, but ignores that being attacked and falling to an attack is different.
This leads us towards a refinement – not number of attacks, but number of successful attacks; not frequency of attacks per second, but frequency of success per attack. A lower rate of success per attack is one way to describe the security of a system; it says something about the defensive strength of the system when it is under attack. The definition of success matters here, and that might vary based on the kind of attack; resource depletion attacks are successful in different ways than privilege escalation attacks which in turn successful in different ways than man-in-the-middle attacks. How good a system is at defense is attack specific, but it is also time specific.
Asking how long a defense is effective leads us to the next dimension that we might look at when measuring aspects of the system’s qualitative strength against attack. In the world of physical security this is typically measured in time to failure. The best vault in the world, for example, will be rated “TL-30”, which indicates that a group of professional safe crackers with a full set of specialized tools and accurate design schematics and blueprints of the vault will need 30 minutes to get inside. This is a design requirement that guides how they are built. The electronic information industry lacks a comparable rating scale.
So, a technical measurement of a system’s security might reasonably be expected to consider how popular a target it is, and the success-rate-of, and time-to-defeat for particular attacks. But that still wouldn’t tell you if it is more or less secure than another system, because security has three more axis of vulnerability and this only looks at the logic of the software running on the computers and network devices that make up the network and application servers. If I want to attack the it over the network, that might be enough, but if I want to actually care enough to attack the system, I probably want to explore all options, even if I’m not that committed to exploiting vulnerabilities on all four axis.
As the old saying goes, the only secure computer is one that is turned off, unplugged, and locked in a vault guarded by very highly paid guards who religiously believe in their work. And once you defeat the guards, break into the vault, plug the computer into the power and the network, and turn it on, how long does it for someone to notice? If I have physical access to a part of your system and time, I win. So being able to notice that the system’s integrity is under attack and do something about it becomes a factor in our evaluation.
When serious professionals want something that is being withheld from them in the modern world, they call a lawyer. Is your super-technologically secure system vulnerable to legal attacks? Do you even know what that means? Can you protect your bits and bytes from a subpoena-wielding enemy? What about an enemy with the power to legislate? How do I measure legal attack security? Is there something short of dead-man’s switches and Cryptonomicron inspired EMF coils in the doorways that can be reasonably done and accounted for in both quantitative and qualitative count and rate metrics? I will happily argue that those without answers in this domain can’t be considered security experts.
Human beings are – and will continue to be for a long time – the weakest part of any system. I don’t need to break the crypto if I can beat or extort or otherwise coerce the passphrase out of a person who knows it. I don’t need to avoid detection if the person that the alert is directed to is dead or incapacitated or compromised. Does the system employ multi-person security procedures or other techniques that raise the bar against those who will attack it by attacking the human beings that are supposed to use it? Are you collecting statistics about how effective those procedures are working? For truly secure systems, compromising a person who is already authorized access is cheaper, faster, and easier than any technical or legal means and no amount of vendor kit is going to change that.
My point is that when someone says “this is more secure than that” they are probably full of crap unless they can also provide a qualitative and quantitative accounting of the system’s security against various logical attacks, as well as physical, legal, and human attacks for each of the two systems. If they can’t… well that is what I mean when I say that “IT security is exploiting fear to sell snake oil”.