Challenges with the Cybersecurity Status Quo

Today, most mature organizations build their information security program around the “Red/ Blue” paradigm. Blue Teams self-assess to identify risk, implement continuous vulnerability management programs to mitigate risk, and (hopefully) detect and respond to incidents as they occur. Red Teams are often the “hired guns,” brought in to occasionally test the defenses and identify previously-unknown gaps. Larger organizations may have permanent in-house Red Teams, but they typically are spread thin and operate similarly to consultant services with regards to frequency of engagement with any given business unit.

This is certainly an incredible improvement over the state of affairs at the turn of the 21st century, when dedicated defensive teams were a rarity and offensive security was in its infancy. This progress has been primarily driven by a recognition of the need to incorporate offensive security as a pillar of any information security program. Many of those curious folk with skills who were once disdained as unprofessional at best and criminal at worst have joined the mainstream security community as recognition of their value has gained acceptance.

While this progress is laudable, continuous improvement demands that we seek new ways to address the challenges we all face. Our adoption of offensive security is incomplete; we have opened the door to our hacking brethren, but tribalism persists which degrades our ability to maximize the value that offensive security principles can bring to our organizations.

Our previous blog post titled “The Cybersecurity Status Quo: Red vs. Blue Teams”, discussed how the current makeup of the cybersecurity field is a fragmented one, with a linear path to remediation without much efficiency or collaboration to speak of. The status quo is logical, but logic doesn’t always equate to efficiency. The existing paradigm suffers from elongated remediation times and frictions which hinder collaboration and may even foster adversarial relationships. This buildup of adversarial frictions is what has led to the “Red vs. Blue” status quo. There are many problems with the status quo, and in this blog post we’re going to dive deep into those problems and learn how we can get better to overcome those problems.

Lack of Common Goals

If you ask a Blue Team member how they measure success, you are likely to hear a variation on one of the following themes:

Absence of data breaches that impact the bottom line
Number of malicious attacks thwarted at the perimeter
Reduction in detection of incidents which trigger response actions

If you ask a Red Team member how they measure success, you may very well hear a different set of metrics:

Level of access achieved on the target domain
Evasion of detection when executing key functions
Total number of vulnerabilities discovered

For both red and blue, these are commendable tactical objectives. But they are not necessarily strategic objectives. While “absence of data breaches impacting the bottom line” may seem like a strategic objective, it is actually a strategic outcome when another objective is met: Detection and response before the attacker achieves their objective. There is a common phrase oft repeated within the industry: “Prevention is ideal, but detection is a must”.

Blue teams certainly want to stop the adversaries at the gates. Red teams want to uncover as many vulnerabilities as possible. However, these are supporting objectives of what should be our primary concern in a world where the perimeter is rapidly dissolving: Detection and response.

All teams must share common strategic objectives to be successful, even if tactical objectives differ. In today’s Red vs. Blue paradigm, it is not clear that the teams share common strategic objectives. We posit that the primary strategic objective should be detection and response, and our Purple Teams should be organized, trained and equipped to further this cause from a proactive perspective.

Information Security Skills Gap

We have all seen the numbers, and no matter which set you believe, the bottom line is that we don’t have enough trained information security professionals and the problem is only going to get worse. Many organizations cannot find (or afford) in-house information security expertise. As a result, remediation of discovered vulnerabilities is often performed by IT staff. For simple remediation efforts like patching, this is perfectly fine. However, remediation of more complex vulnerabilities can be a challenge for personnel who lack an understanding of offensive security concepts. Furthermore, without a clear understanding of the root causes, those same personnel may be prone to repeat the processes that resulted in the vulnerability.

Continuing to silo our professionals into purely Red/Blue tribes will only exacerbate the existing skills gap due to missed opportunities to pollinate wider audiences with offensive security principles. Red Teams exist to test, but ultimately the test is subordinate to a greater goal: to teach.

The current Red vs. Blue paradigm also ignores one of the foundational principles of teaching, one you have surely heard countless times before: Crawl, then walk, then run.

You can almost hear the sound of a starting gun at the commencement of an engagement. The Red Team begins stringing together all of their tactics, techniques and procedures to achieve their objectives. Unless the Blue Team is fortunate enough to detect each attack path in real time, they are missing key learning opportunities. They will never glean as much knowledge from a report as they can gain by observing and understanding as an activity is occurring.

We know that a building block approach composed of partial-task training (PTT) events is a much more effective method for knowledge transfer. Nobody reads a book on JavaScript and then sits down to code a new web app. Everyone starts with a simple “Hello World” program, then progresses through a series of increasingly more complex exercises.

The information security skills gap is a daunting issue that few of us have the power to meaningfully impact at the macro level. However, remember the words of pioneering tennis legend Arthur Ashe: “Start where you are. Use what you have. Do what you can.” You are in a position to change your organization. You have intelligent, savvy and eager employees on the Blue Team. You can effectively train them in offensive security principles and we will discuss how purple-teaming is the perfect medium for achieving this objective.

Uninformed Threat Modeling

Which of these two types of test will provide the greatest value to your defensive efforts?

A penetration test where the testers use their best tricks to find any openings to whatever data they can compromise.
A penetration test where the Red Team analyzes historical attacks, understands what data is coveted by adversaries and uses this information to model realistic attack profiles.

In order to perform the latter, the blue team needs to share locally-generated threat intelligence with the Red Team during the planning phase. Unfortunately this rarely occurs. For example, when was the last time that a Red Team asked you to review incident response reports during the planning stage of an engagement?

Additionally, Red Teams should be informed about existing defenses prior to planning an engagement, or to take it further, the defenses you are truly focused on testing. Red Teams that choose attack vectors which are closely monitored are wasting resources. Red Team activities should guide the Blue Team to examine attack paths that are not well fortified to provide the greatest ROI. The Red Team should also be closely connected with the defenses the Blue Team wants to specifically test and augment the test plan accordingly. It is vital that the Blue Team knows the areas of focus have been addressed in addition to the new tactics and techniques exercised by the Red Team.

Lack of Post Assessment Collaboration

In an ideal world, the process of knowledge transfer would be an ongoing process with collaboration occurring between teams independent of a defined test or assessment. An interim step towards that ideal world is simply post-assessment collaboration. No matter how detailed a finding or how many artifacts are included with the report, questions will inevitably arise.

Unfortunately most engagements with external assessment teams do not include post-assessment collaboration in the Statement of Work, and the blame is shared. Penetration testers enjoy hacking, and there is a cultural bias against becoming involved in the actual remediation work, so few even pitch the concept. Managers may not understand the value (but they do understand the high cost of billable hours), so do not ask for (or better yet, insist on) the consultant “sticking around” after the report is delivered.

This is a lose-lose paradigm and one that we must overcome as an industry. As the talent gap continues to grow, there will be an even greater reliance on external information security services. If we continue in a transactional mindset, where communication and collaboration starts and ends with a document-based report, we will never realize the efficiencies that are possible with outsourcing. Organizations will pay consultants to identify risk, but will lack the in-house expertise to address all the risks uncovered. Already today, we lament the number of reported findings that are never addressed; the image of the pen-test report “collecting dust in a desk drawer” has become an archetype in the industry. Granted, there are a variety of factors which contribute to unaddressed findings, to include budget, competing priorities and the rapid pace of technological change. But it is also true that remediation of many findings are beyond the skill set of in-house IT staff.

In this transaction-based approach, consultancies are also leaving money on the table. Billable hours are billable hours, but this isn’t about milking the clients. If a block of additional hours enables more effective outcomes, then the value proposition for the customer is real. Aside from the financial benefits, consultancies can also become more effective advisers to their clients when they have regular visibility on the true burdens of remediation. Some mature consultancies today provide a “burden of implementation” score with each recommendation, and some go so far as to provide an evaluation of the cost/benefit of a given remediation solution. This is extremely valuable information for the client but cannot be provided if the red team does not have relevant experience in actually implementing the solutions they prescribe.

In an enterprise environment with a dedicated internal Red Team, enlightened management can facilitate regular post-assessment collaboration between the Red and Blue teams.
Unfortunately, this is not standard practice in most organizations. The information security workforce is a revolving door between internal testing teams and consultancies, and those same cultural biases against involvement in remediation work exist in internal teams. Blue Teams have their own biases as well – some rooted in real or perceived arrogance of Red-Teamers. Management is faced with two tribes that don’t particularly want to work together, and lack an understanding of the value proposition in forcing collaboration.

Inadequate Tools for Effective Collaboration

Today the primary tool for communication between the red team and the blue team is the “final report,” usually delivered in a PDF. This traditional format suffers from several inherent weaknesses.

First, the usability of the report for the consumer is inversely proportional to the length of the report. The more artifacts the testers include, the harder it is for blue teams to separate the signal from the noise. Testing teams collect reams of useful information, but make judgement calls about what to include in the report so as to avoid overwhelming the consumer.

Second, the process of transferring data from the report to the customer’s workflow system is manual and laborious. Copy, paste, repeat. Inevitably, some data and artifacts never survive this process and the Blue Team is provided with a subset of the available information – which is itself a subset of the information that the testing team collected. Additionally, the current workflow system for tracking the findings from a report is predominantly a spreadsheet that doesn’t trace back to previous reports or could get lost in the shuffle of personnel changes.

Finally, document-based delivery prevents the use of one of the most effective methods of communication: video. If you want to learn how to fix your washing machine, do you read the service manual? No, you go to Youtube. If a picture is worth a thousand words, a 30 second video clip can be worth 50 screenshots and a lot of unnecessary narrative.

With all of these challenges in the current paradigm, how do we continue to make progress in achieving the goal of avoiding compromise or detecting compromise as soon as possible within the lifecycle? The answer lies with a shift in the paradigm towards true purple teaming and effective collaboration.

Overcoming These Challenges with Purple Teaming

There’s one way to combat all of these challenges: Effective Purple Teaming. Purple teaming is the collaborative function performed by Red Teams and Blue Teams to mitigate all of the pains discussed thus far. It’s a new approach to collaborative testing and remediation that seeks to break down cultural barriers, improve communication and “level up” everyone’s skills. It is also aimed at reducing the mean time to remediation for reported risks and vulnerabilities. Note that purple teaming is a role but not a job; there are no dedicated Purple Team members. A team member’s function is either Red or Blue, but everyone’s role is strictly purple with a common mission of detecting compromise as early as possible within the attack lifecycle. So what do this role look like? There is no canonical definition of purple teaming, but common tasks and objectives include:

Design realistic tests based on shared priorities, informed by locally-derived threat intelli- gence and tailored to test the defenses’ critical assets.
Speed up the process of remediation through established channels for collaboration
Prevent related future occurrences of issues through knowledge transfer of root causes
Help foster an offensive security mindset across all members of the cybersecurity team