The Future of Manual Security Testing

Will video kill the radio star?

Episode #2 of Friends Friday — a cast bringing you informative conversations with innovative thinkers, creators, practitioners, influencers, and leaders in the industry — sought to answer the existential question: Will video kill the radio star? Or in other words, will automation make manual security testing obsolete? 

Nicholas Popovich, founder and “hacker on staff” of Rotas Security, joined the cast to bring his perspective on the future of manual pentesting. Nick’s career in hacking has focused on adversarial threat simulation, offensive and defensive security, and advanced technical security assessments. He is also a U.S. Army Signal Corps veteran and has worked in the public and private sectors performing advanced cyber security assessments. 

Jordan Treasure, PlexTrac senior professional service manager, hosted the session. Jordan assists security teams in improving outcomes during security assessments. Jordan is a GIAC-certified forensic analyst with seven years of experience performing risk assessments, vulnerability assessments, and threat-hunting missions on national security assets. Many of his experiences stem from his 14-year United States Air Force career. 

Watch the full episode or read on for the highlights of their conversation.

The automated vs. manual testing spectrum

The conversation began by acknowledging the rapid growth rate in automation and the spectrum of perspectives regarding it, from complete trust in automated tools to significant fear of them. 

Nick said, “There’s a lot of talk about a lot of different automation tools. There always has been. And whenever there’s a new paradigm or an introduction of a new sort of tooling or system, there’s naysayers and there’s folks who are early adopters. And I sit right in the middle.”

He continued, “I certainly say that automation is not the devil. It helps when you’re trying to scale in an organization, and if leveraged by folks with the requisite expertise to understand it, to tune it, to leverage it to its fullest potential, automation can be a huge asset. Where I think there’s trouble — especially maybe in unregulated industries — is where folks can’t go beyond the automation, if they’re beholden to only leveraging systems and tooling without being able to understand the results in an expert fashion or be able to innovate and adjust and manually take advantage of technology stacks.” 

Automation for scaling 

To express the value add of automation to scaling security testing, Nick shared an example: “In web application testing there are a lot of phenomenal tools, and we leverage them. When we do app testing, we’re going to leverage tools that spider the environment, that look through pages, that make some of the requests and really give you a sitemap, and start looking for some of the maybe the tiki tack stuff or just some of the low hanging fruit.” 

Jordan agreed that automation tools are essential for scaling security programs but explained that a balance is required between tooling and human oversight. “The automation piece is great because you can go in, you can set everything up, and you can just start moving at scale relatively quickly. But the issue that most organizations have is that every little corner of your network or your applications, your tech stack, is going to have its subtleties that maybe don’t equate to exactly how the automation is configured. And that human piece gets you that last mile, I think. Automation is not the devil necessarily, but it has its limitations. People do. And the complement of the two is really where it’s at.”

Humans for validating and maximizing value 

“Tools are only as good as the folks that wield them and the expertise behind them. The thing that always blows my mind is just the implicit trust of a tool out the box,” said Nick in describing the continued necessity of humans in the security testing program. 

Their discussion highlighted two key ways in which manual oversight of automation will continue to be essential. The first way is in validating the results of automation by ensuring the tools are working as expected and finding gaps that are missed by algorithms or configuration issues. 

Jordan used an illustration of a dam: “If we created some really cool tool to just make sure that the dam is always safe and there’s no cracks in the dam or anything like that, you’d probably still want an engineer to inspect the dam from time to time, make sure the tools are working as expected. That there isn’t something, some new environmental factor that, you know, you want to be aware of that the automation wasn’t originally configured for.”

The second human aspect that they anticipate remaining significant even as automation continues to advance is being able to maximize the value of automation investments. Despite their ability to manage huge data and repetitive task quickly, automation tools also have limitations if not weilded by skilled practitioners who understand how the tools work and the results that they should be producing. 

Nick explained, “If you’re using different tools that are going to provide you value and you don’t know how to interpret the results or ensure the results that you’re getting are accurate and appropriate, you’re really not leveraging that tool to the best of its ability.”

Jordan continued, “For CISOs, when you look at the cost of some of these tools, and if you’re going to spend a significant amount of money — especially if you have an extensive toolkit, you’re going to spend a good amount of money. And while the idea for a lot of people in that C-suite, that executive leadership level, is that automation helps with cost scaling, if you’re not investing in the setup configuration and then the care and feeding over time of that automation, that toolset, you’re really just throwing your money out the window in some cases.” 

Then there’s physical pentesting 

Next they covered the subject of physical pentesting specifically and how automation is, or is not, changing the game. 

Nick has extensive experience in physical pentesting and actively conducts physical testing with his team. He shared, “So from a proactive planning and site survey perspective, there are a lot of neat things coming out leveraging some of the tooling and automation. Really, being able to start correlating alerts and being able to do some of the stuff on the back end is neat.”

That said, he also noted, “When it comes to being able to execute physical security assessment and testing activity. Still, there’s not too much that we’re able to leverage in the automation world other than intelligence operations and gathering information.”

Despite the fact that true automation in the physical testing space is far in the future from Nick’s perspective, he does see some near term benefits in automating detection and response activities related to physical security. 

He explained, “And I think as we continue to mature as organizations, [using automation to lower] the barrier to being able to have cohesive communication in your tech stack and your alerting and those types of things will be super valuable. And that’s where I see a lot of value, not just from assessment and being able to perform automated assessment activity, but the real value is going to be derived from being able to take a lot of the legwork out of connecting the dots. For example, you say, okay, this was this IP address, which is this system, and it goes to nine different dashboards. And then you’re like, that’s the binary that was downloaded here. That’s where you really get a cohesive kind of telemetry view.”

The need for technology whisperers 

Nick summed up the cast with his prediction on one of the ways he sees skilled humans as necessary as automation and AI continue to improve and evolve. “I think that the ability to be able to communicate and speak the language of the tech stack to the tech stack in natural language processing and the ability to be an interface [will be important]. There’s going to be a lot of liaisoning.” 

Nick concluded, “I do still think there’s always going to be a necessity for a deep understanding of technology under the hood. There’s going to be tinkerers and they will be a necessity, and I think, they’ll actually be more important.”