POSTED: January 10, 2020
Warfighters have used offensive engagements to improve their planning and execution for centuries. Military planners “red team” battle plans with dedicated teams designed to emulate the adversary’s capabilities and behaviors. After World War II, Admiral Chester Nimitz famously stated, “The war with Japan had been enacted in the game rooms at the War College by so many people and in so many different ways that nothing that happened during the war was a surprise—absolutely nothing except the kamikaze tactics toward the end of the war. We had not visualized these.”
In our infosec universe, penetration testing or “pen testing” has been around in various forms for decades. When I was at Microsoft, I started the first internal offensive testing program that targeted Windows NT 5 (which became Windows 2000). Microsoft, like most product companies, then focused on “functional testing”—verifying that features work as specified. It did not formally consider overtly hostile attacks until after the Melissa and Love Bug mass mailers of the early 2000s. I called my program “malicious testing” to differentiate it from benign “functional testing” and to highlight the critical difference in mindset.
What I saw at Microsoft still largely holds true today: defenders don’t understand attackers. The “defenders” I interacted with in 1996 were young developers trying to get their cool new features to work and checked into the nightly builds. They weren’t thinking about security, and they weren’t specifically trained in attacker techniques. Similarly, today, the average enterprise IT department is consumed with daily firefighting and doesn’t understand how attackers abuse its systems.
Today, when a classic “pen test” is performed, it almost always proves that similar, difficult-to-detect attacker techniques continue to work time after time. Without timely detection, attackers have unlimited time to breach defenses, and they will always be successful. Tightly scoped, short-term penetration testing engagements don’t present a realistic test of defensive people, processes, and technologies, and often don’t result in improvement, as evidenced by the fact that the same attacker techniques tend to work over and over.
Today’s classic “pen tests” do not result in an improved defense. They are activities without progress. Admiral Nimitz learned from the valuable red team exercises at the War College that activities made them better and directly contributed to the outcome of the conflict. However, classic “pen tests” aren’t making us better as infosec defenders.
Offensive Exercise Maturity Model
There are several stages enterprises go through with respect to offensive testing; most enterprises are stuck in Type I and Type II engagements that provide limited value. The maximum security value comes from Type III and Type IV engagements.
- Vulnerability Scan: Staff (or possibly an outside firm) run a tool like Nessus, Qualys, or Rapid7. This exercise primarily tests the efficacy of the vulnerability management program. The result is a lengthy report which is often noisy, irrelevant, and not acted on for a variety of reasons.
- Classic Penetration (“Pen”) Test: A short, time-bound, and tightly scoped engagement in which an external party attempts to gain an unauthorized level of access. Depending on the scope, the attackers may originate from the Internet or from within the enterprise (bypassing NAC and similar controls may be in-scope in the latter case). Defenders/staff may or may not be aware of the engagement. The result is a lengthy report. The engagement primarily tests client vulnerability management, basic access controls (authentication/authorization, network segmentation), and detection capabilities. The defenders do not typically interact with the attackers, and the attacker techniques used may or may not be disclosed in detail.
- Purple Team Engagement: An engagement model that combines the offensive activities of a penetration test with specific defensive awareness, coaching, and improvement. A purple team engagement is organized around effective interaction between attackers and defenders to transfer general knowledge about attacker techniques and specific knowledge about how defender systems behave when faced with these techniques. The objective of the engagement is to create real improvement in the client’s detection and response capabilities during the engagement, something that won’t just culminate in a report.
- Long-Term, Lightly Scoped Adversarial Emulation: This engagement model is designed to emulate the activities of an attacker closely and to test the client’s defensive capabilities when it is faced with real-world threats that are not tightly scoped and do not materialize on a set schedule. As all offensive engagements before this point have contained a number of artificialities, the objective of this engagement is to evaluate the performance of defenders, processes, and technologies together when presented with realistic threats. The engagement primarily tests client detection and response capabilities, and results in a report describing prevention, detection, and response efficacy against specific threat tactics. Time to detection is a critical factor that will be measured, understood, and improved. As with fire alarms, the tighter the time to detection, the lesser the damage.
I want to encourage all defenders to increase the utility of offensive engagements in 2020. Think about how to create progress and improvement from these activities. Don’t settle for classic “pen tests” that generate the same result over and over. Don’t accept a “pen testers always win” defeatist attitude—gain confidence in and tighten your detection timelines. Like Admiral Nimitz, use offensive engagements to understand the playing field before you find yourself in a firefight.