POSTED: April 6, 2021
Warfighters have used offensive engagements to improve their planning and execution for centuries. Military planners “red team” battle plans with dedicated teams designed to emulate the adversary’s capabilities and behaviors. Famously, Admiral Chester Nimitz stated after World War II, “The war with Japan had been enacted in the game rooms at the War College by so many people and in so many different ways that nothing that happened during the war was a surprise—absolutely nothing except the kamikaze tactics toward the end of the war. We had not visualized these.”
In our infosec universe, penetration testing or “pen testing” has been around in various forms for decades. When I was at Microsoft, I started the first internal offensive testing program that targeted Windows NT 5 (which became Windows 2000). Microsoft, like most product companies, then focused on “functional testing”—verifying that the features work as specified; they did not formally consider overtly hostile attacks until after the Melissa and Love Bug mass mailers of the early 2000s. I called my program “malicious testing” to differentiate it from benign “functional testing” and to highlight the critical difference in mindset.
What I saw at Microsoft still largely holds true today: defenders don’t understand attackers. The “defenders” I interacted with in 1996 were young developers trying to get their cool new features to work and checked into the nightly builds. They weren’t thinking about security, and they weren’t specifically trained about attacker techniques. Similarly, today, the average enterprise IT department is consumed with daily firefighting and doesn’t understand how attackers abuse their systems.
Today, when a classic pen test is performed, it almost always proves that similar, difficult-to-detect attacker techniques continue to work without detection time after time. Lacking timely detection, attackers have unlimited time to breach defenses, and will always be successful. Tightly scoped, short-term pen testing engagements don’t present a realistic test of defensive people, processes, and technologies, and often don’t result in improvement—evidenced by the fact that the same attacker techniques tend to work over and over.
Today’s classic pen tests aren’t resulting in improved defense. They are activity without progress. Admiral Nimitz learned from the valuable red team exercises at the War College—these activities made them better and directly contributed to the outcome of the conflict. But classic pen tests aren’t making us better as infosec defenders.
Offensive Exercise Maturity Model
There are several stages enterprises go through with respect to offensive testing; most enterprises are stuck in Type I and Type II engagements that provide limited value. Maximum security value comes from Type III and Type IV engagements.
- Vulnerability Scan: Staff (or possibly an outside firm) run a tool like Nessus, Qualys, or Rapid7. The exercise primarily tests efficacy of the vulnerability management program. The result is a lengthy report that is often noisy, not relevant, and not actioned for a variety of reasons.
- Classic Penetration (“Pen”) Test: A short, time-bound, and tightly scoped engagement where an external party will attempt to gain an unauthorized level of access. Per scope, the attackers may originate from the Internet or from within the enterprise (bypassing NAC and similar controls may be in-scope in the latter case). Defenders/Staff may or may not be aware of the engagement. The result is a lengthy report. The engagement primarily tests client vulnerability management, basic access controls (authentication/authorization, network segmentation), and detection capabilities. Typically, the defenders do not interact with the attackers and the attacker techniques used may or may not be disclosed in detail.
- Purple Team Engagement: An engagement model that combines the offensive activities of a pen test with specific defensive awareness, coaching, and improvement. A Purple Team engagement is organized around effective interaction between attackers and defenders to transfer knowledge: general knowledge of attacker techniques and specific knowledge about how defender systems behave when faced with these attacker techniques. The objective of the engagement is to create real improvement in the client’s detection and response capabilities during the engagement, not just culminate in a report.
- Long-Term, Lightly Scoped Adversarial Emulation: This engagement model is designed to closely emulate the activities of an attacker and test the client’s defensive capabilities when faced with real-world threats that are not tightly scoped and do not materialize on a set schedule. As all offensive engagements to this point contain a number of artificialities, the objective of this engagement is to evaluate the performance of defender people, processes, and technologies together when presented realistic threats. The engagement primarily tests client detection and response capabilities and results in a report describing prevention, detection, and response efficacy against specific threat tactics. Time to detection is a critical factor that will be measured, understood, and improved. Like fire alarms, the tighter the time to detection the lesser the damage.
I want to encourage all defenders to increase the utility of offensive engagements in 2021. Think about how to create progress and improvement from these activities. Don’t settle for classic pen tests that generate the same result over and over. Don’t accept a “pen testers always win” defeatist attitude—gain confidence in and tighten your detection timelines. Like Admiral Nimitz, use offensive engagements to understand the playing field before you find yourself in a firefight.
Protect your business against an evolving threat landscape by learning more about offensive security.