The Continuous Pentesting Gold Standard: Taking what’s great and making it better

So, Tom, maybe let’s start talking a little bit about what have you seen over the past several years around the myths around penetration testing and how do we kind of help dispel some of those?

Yeah, there’s lots of myths over the years and decades that I’ve been involved in penetration testing. I remember the good old days when it was very manual. That’s where manual penetration testing evolved from. And frankly, there weren’t a lot of vendors in the space.
It was just a bunch of us hackers cobbling scripts together and then writing open-source tooling. And then some. Some of us got together and, you know, I go back to, like, HD more. Right? And Metasploit, like a perfect example. Started out as, like, a pet project of HD more and then involved, like, an entire community, you know, developing exploits and all kinds of tooling for that.

And so nowadays, it’s become filled with different vendors, different players. And I think it’s interesting because it’s just the way that the cybersecurity industry in itself has evolved from a very oddity type of function when it was information security evolving into something so much bigger now. And that has been because of the different threats and the different security landscape that has really just taken place over the last several decades. So penetration testing has evolved along the way, too.

And there’s some things that I’d say that not necessarily for the better either. Like, I think we confuse, you know, everything is a red team these days, right? And the term red team gets, you know, it’s not the correct term for every single type of penetration test. It’s kind of like back in the day when we would have arguments over vulnerability scanning versus pen testing, two different things. And then we have to explain, we have to, like, give examples. And now it’s like red teaming is not the same as penetration testing.

And so I think we have, it’s become very popular. I still see a lot of kind of new people coming into the industry that the first thing they say is, I want to be a hacker, I want to be a pen tester.
And sometimes I have to have some difficult conversations with them. It’s like, maybe that’s not the best place for you to start because a lot of us, as we kind of grew up through our careers, we didn’t start as pen testers. I mean, it wasn’t even a career option at that time. You know, we all grew up either as sometimes developers. I mean, my background was in it. I was a break fix help desk guy and eventually worked my way up into like, an actual vulnerability management function at a company. And then I discovered this thing called pen testing, and I was like, wow, well, this is, this is really cool.

So I think the evolution of pen testing is just, it’s been a lot of, just growing up, I think, for a lot of us old timers, and I hate saying that, like, I’m an old timer now. What the heck? Like, this is weird. Like, I shouldn’t be saying I’m old in the industry, but we really kind of are. I’ve been doing this for veterans. Yeah, we’re veterans. I like that. I like that we’ll call ourselves veterans because like 18 years doing this stuff is like, yeah, what the heck happened? I don’t know.

Yeah, well, yeah, and I think, I think, I think it’s right because, like, you know, one thing that I kind of like, you know, have always observed is, like, one, people’s backgrounds really do actually play a big factor into, like, how, how good of a pen tester they are. And so, like, I, you know, some of the best pen testers I’ve ever worked with were former software engineers and sysadmins. Right. I mean, you know how things work, right. And you have a deeper understanding, and that’s really important. And that’s like kind of the hacker mindset. And then what you also alluded to is like that, that hackers, you know, naturally are lazy. It’s like if I have to do something twice, I can try to automate it. Right, totally. So, and that’s, I think that’s what we’ve seen, how the industry has evolved. So where it’s not, I think the low-hanging fruit really has become a lot more automated. Right. Being able to find, you know, the low-hanging fruit. So it does free up the more, you know, being able to spend more of the mental capital on the really complex exploitation to a degree.

Right, yeah. And that’s been a good thing. Right. Like I’ve always been a proponent of, you know, from a manual pen testing perspective, is we should spend more of our time looking for the things that automation does not find or does not find very well. Business logic flaws, eye doors. I mean, all these things that really, you need the expertise and you need the talent to really discover those things. Nowadays, there’s even more automation that can start to discover and start to assist the pen tester is how I look at it.

Right. We’re not replacing the pen tester, but we’re giving them tools and better tooling to make their job more efficient. So there’s a lot of positives, I think, that I’m seeing, at least in the last couple of years. Oh, yeah, for sure. And I think that kind of leads into the topic. Right. In terms of, I think one of the myths that, you know, and we still hear it, and I’m sure you do too, is like, well, we only do pen tests once a year because expensive. And, you know, like, they require a lot of resources and stuff. But, you know, with, with the dawn of automation for a lot of that, a lot of that stuff, you can really, you start to see a lot of other consulting firms doing more managed service offerings from a continuous basis, and then we obviously could really encourage people to do internal testing continuously. Right. And, you know, maybe speak to some of your experience there and it’s valuable, right?

Yeah, it’s extremely valuable. It’s interesting, I was just reading the Verizon data breach report, and again, I think this is maybe the second year where they talked about time to patch vulnerabilities is still not where it needs to be. And I think organizations are struggling right now with the rate of exploitation and how quickly things are discovered, exploited. And the next thing you know, you’ve got ransomware you’ve got to deal with.

And so pen testers, we have to evolve as well. One time a year is usually not enough for an organization from a testing perspective. And so I think we’ve all adopted more of the mindset of like doing pen testing on a continuous basis. I mean this is nothing new, right? I mean we’ve been talking about this kind of thing for years, but I only think in the last couple of years, maybe five years or so, that we’ve really seen the way we do continuous penetration testing really start to take hold. There’s a lot more vendors in the space, obviously, but I think pen testers and pen test teams are starting to change their methodology and change the way that they do things.

And I don’t want to always say it’s like just run a scan and then validate vulnerabilities because that’s not what I mean by continuous pen testing. Yeah, that is just traditional vulnerability management. You’re running a scan, oh, somebody’s validating the findings and maybe doing a little bit of exploitation. Right. I see it more now of like you’re actually doing like breach simulation. You’re taking things to the next level and you’re doing these things on a more continuous basis and you’re really focusing the testing on the organization’s risk or the riskiest assets. And so when I think of continuous pen testing, I think of asset management understanding where your assets are, understanding the vulnerabilities of those assets and then really focused and then doing targeted testing on that continuous basis. So it’s, you know, you’re, you may be doing one test at one time, but there may be other testing that’s overlapping that existing test. You’re doing something continuously.

Yeah, yeah. And it’s, it’s like, I view it as like kind of, you know, adjusting the scope. Right. You know, when you’re, when you’re thinking about a full-fledged pen test or maybe like a full-fledged red team engagement for like kind of your annual thing. Yeah. That’s, that’s broad and wide and could go deep, but when you have a, when you have a continuous paradigm, you can really segment like, hey, we’re going to test these specific things, you know, and, and truly, you know, get more coverage is really what it’s about and, and it’s helping you evolve with the threat actors of the present time, right? One year ago, things were a lot different. Right? Yeah, yeah, yeah.

It’s interesting too, because, you know, with all the different vendors in this space, and there’s, there’s a lot of different scanning technologies that will say you’re getting a continuous pen test. And, you know, we all know that you’re just getting a scan. And I am a big proponent of using automation automated scanning tools. I mean, we both came from Veracode, so very well aware of the value that these tools provide. But what I’m seeing now too is there’s so many vendors and there’s a lot of marketing hype around some of this, things that say that, yeah, we’re offering you a continuous pen test of your environment and it’s all powered by AI and ML. And I mean, I’m really scared, you know, Dan, I’m going to RSA next week, so I’m going to walk through the expo floor and I’m just going to count like how many vendors actually say AI pen testing in their booths, right. And I’m going to cringe a lot and maybe cry.

I don’t know. But we’ve reached the next gen. I remember everything was next gen, right? Yeah. And you’re like, oh, come on, guys, seriously. I mean, it’s a good topic. You know, we had it, we had some folks on, you know, in one of these casts a few weeks ago around AI. And, and I think the notion of like allowing AI to help the testers actually get their work done faster, which is more automation and things, but it doesn’t replace the tester.

I think that’s another myth to dispel is like, as we continue to enter the age of automation, it’s truly meant to be an augment to help the tester get deeper into the organization or deeper into the web app or whatever you’re testing. Right. It’s never gonna replace, it’s never gonna replace a tester. Yeah, not anytime soon at least. There are way too many issues. I mean, we are in the massive hype cycle, the extreme of it right now with AI. And I think everyone needs to like, okay, take a step back, you know, like, let’s not like put all of our eggs in the AI basket.

But we have to like think about how it is going to help us. Not 100% rely on the results of AI. I mean, the hallucinations, the, I mean, I just using it myself. I mean, I’ve got to check the work that is outputted from a GPT. You just have to, for a pen tester. If you’re putting potential vulnerabilities, sensitive client data, vulnerability information exploit details into an LLM, we get into some very serious issues that I think we all as an industry need to start addressing of. We keep feeding this machine and how much of that machine are we feeding it with sensitive data that shouldn’t be there. So I’m all for it, but I’m all for using it responsibly and making sure that as a pen tester, using it well, responsible yourself, but also using it in the right ways to help you do your job better.

Yeah, yeah. Being diligent is super important. I mean, you know, cause like we recently announced our AI model to help write reports, right? That’s a very natural thing for us. But we implemented secure by design and this is not a huge plug. I was just kind of like, we implemented secure by design and private models. But I’ve been surprised at how many people have admitted to us that, oh, yeah, we throw our stuff into chat GPT to help write our reports.
I’m like, yeah, that’s. Why are you doing that? Yeah, but, but I, but it’s, it’s, you know, it could be ignorance, it could be, you know, naivety, you know, but I think it is important for us to, as an industry to recognize.

I was joking with some of the other, it reminds me of the conversations we were having about the cloud like ten years ago, right, where you get fired if anything was put into a cloud system, even if it was like Google Docs. Right? Yeah, totally, you know, but I remember, you know, as it’s, as it’s evolved, it’s like, yeah, actually, you know, cloud companies have to be, they actually have to be much more secure as just part of their job. And so now it’s, it’s actually, it’s actually kind of a help and assistance for an organization in many cases. Yeah. I think one thing we all need to understand in our industry, because we, we often get involved like we’re all like in an echo chamber. I mean, we really are when you think about it.

But I mean, been in consulting as long as I have and all the customers that I have talked to over the years, everybody’s on a different stage of that journey. Right? Like you mentioned, the cloud. I mean, I still have customers that are still trying to figure out how to move from on prem to cloud native. Like they’re not even starting their journey or they’re thinking about how are they going to do it. It’s the same thing with AI. You’ll be ten years from now and someone’s be like, I haven’t touched AI yet in my environment and I don’t plan on it. So we all have to, like, have a little bit of understanding of, like, while it’s all being talked about in the industry is like the next big thing and everybody’s doing it. Like, that’s not necessarily true. Right. I mean, it’s just like how there’s a very small percentage of the entire security community on Twitter. Surprise. Like, literally a very small, like, so when we all talk about this stuff, like, you’re really talking, we’re only talking amongst ourselves, so it’s important for us to just think about that. Yeah, take a step back. Make sure you’re garnering all the perspective.

Yeah. And it’s so true because everybody’s going to be on their own journey, especially in this, you know, but like, kind of coming back to like continuous penetration testing, I think that what has excited me is like, what you alluded to over the last five years. It does feel like the message is like one everybody. I think, again, maybe I’m sitting in an echo chamber here, too, but I think that most people recognize the value of penetration testing. Right. And that it’s highlighting some of the key, the most sensitive and high, highly critical risks in your environment, you know, because they can be exploited, but, but also the fact that, like, yeah, that’s really valuable, but it can be expensive if you don’t, like, plan for it the right way. And there are some economical and resource-friendly ways to do that, not purely through automation, but like an augmented mixing manual testing, whether that’s internal or contracted service with automation.

Right. I think that’s what excites me is that I think that that that paradigm is slowly shifting. It really is. I mean, you know, that gets into kind of like the evolution of pen testing. I see it is like into something more like attack surface management or ASM, where, you know, I’m not like trying to promote Bishop Fox, but like, I mean, that’s what we literally do, which is, you know, we have a large number of traditional pen testing, you know, that we’re doing for clients, but a lot of our clients are moving to a continuous fashion. So we have Cosmos and we, we move them in that direction to give them not just the continuous pen testing, but we’re really looking at emerging threats and how quickly that an organization can address the next move, it or the next whatever that comes out. Because, like, we just saw in the Verizon report, like, it’s that time window from identification to exploitation. We have to limit that as much as possible. And pen testing and continuous pen testing is a huge part of that now.

Yeah, yeah. And I would even say being able to collect better data beforehand, how valuable could it be if you can tell your executive staff or stakeholders that, like, board or whomever that like, hey, our mean time to remediation from true, like attack start, you know, is, is XYZ days as opposed to. Because like, you know, the only way to really track that, you know, before if you’re not doing like continuous testing and actually saying like, hey, I hit the button to exploit on at this point in time, you don’t really know when a true attacker hit the button to exploit. You only know when you have some kind of forensic evidence. But they could have, they could have been attacking for a lot longer than that. So, so being able to truly say like, hey, it’s this, the instant an attack truly starts, we have, you know, 38 hours or 36 hours or something like that. That’s pretty powerful, right?

I mean, that very, very. Yeah, yeah. It’s interesting to me just to see how attacker, attacker tactics have changed and, you know, we still deal with ransomware obviously being the biggest thing still and extortion. And I just, you know, it goes back to the things that we’ve always really talked about in this industry of, you know, the security basics and still we as pen testers are still doing the same. Like we haven’t changed the way that we attack a system. Like, I mean, maybe the tools or the, you know, the technologies has changed a bit, but the end of the day, it’s still the same to the same ways that we’re exploiting systems.

Yeah, yeah. And I mean, it’s like, you know, in some respects, it highlights. Like, man, this is hard, right? You’re on a, you’re on a, you’re on a journey. You know, it’s not a sprint, but, but at the same rate, um, you know, sometimes you kind of say, like, why, why is this so hard? You know, I mean, it’s, and it’s just, you know, that’s, I think that’s just kind of the nature of where we’re at. It is, it is. I, I mean, I remember when I was a pen tester, I would get so frustrated when, like, I couldn’t break in to an app or a system. And then I thought about it, I’m like, well, that, that’s actually a good thing for the company. I can’t break in. But as a pen tester, you’re like, I got to make findings. I got to have a finding. I got a finding. And I always try to, like, when I talk to my teams and such and other consultants and, you know, how do you get around that feeling of, like, I can’t do it? Like, well, you got to change the conversation with the client. Like, you have to highlight the good things that they’re doing.

Exactly. Highlight their. Know, if they got strong security controls. Awesome. Like, let’s put that in the report. Let’s talk about the good things. It’s not always about the bad stuff, right? Yeah. And, I mean, yeah.

I mean, I felt the same way when I was testing. I mean, you get this imposter syndrome. Like, Thursday night rolls around, the test ends Friday, and you’ve. You’ve not found, like, hardly anything demonstrable, right? And you’re like, I’m a terrible tester. But then it’s like, oh, no, actually, here’s all the things that we tested. Here’s all the things that I tried, and they didn’t work. And so, like, kudos to you.

Right? So, yeah, that’s a. It’s a really good point. I think just for any pen tester to kind of. Obviously, you’re always trying to get yourself better and learning new stuff because we’re in that industry, that type of space where you just really have to stay on top of it. But. But at the same rate, like, you could give credit, you know, tip your cap, so to speak, to the people that are defending themselves. Right.

So, yeah, you have to. You have to give credit to the blue team, you know, like, as much as I love being on the red team and breaking things, but, you know, we all say this, right? The blue team has the hardest job in the industry, and we have to give them some credit. We have to do more of that. For sure. For sure. And just continue to collaborate with them, you know? So, like, you know, kind of, like, you know, what would be some of your tips in terms of, like, you know, establishing a continuous testing. Continuous pen testing paradigm?

Yeah, it’s. From what I see, it’s. It’s different, obviously, for every organization. You know, it doesn’t matter the size of the. But, um, there’s. There’s usually different requirements for it, but I would say start small is my first recommendation. Um, we have better success, like, with our clients at Bishop Fox. It’s like, starting with, like, give me, like, three of your key high risk applications.

Like, we start there, and then we put that into a continuous testing program, um, and build it out from there. Um, if you’re trying to bite off more than you can chew, you’re usually not going to be successful because oftentimes we have to customize things for the client or. One of the biggest challenges we still have is credentials. Like chasing credentials from the app teams and the developers. And like, this is a, this is a massive problem in our industry. Still is. We haven’t found a good way to like work with.

It’s a human problem, right? Like as you have a person on the client side or the customer that needs to somehow get those credentials for you. And so there’s this waiting game and this back and forth and, you know, a portal and all these things can help, but at the end of the day still, someone still has to give you the creds or the app. So, like, if you’re talking a thousand apps, you know, it becomes a scaling problem. And then if we’re testing these things continuously, how often are they touched? I’m a big fan of doing your one large initial assessment of an application. Then you do Delta testing throughout a set period of time where you’re capturing changes. You’re building that along development lifecycle when new releases come out and you’re doing very specific targeted testing that seems to work a lot better than just doing one full pen test every time on the same stuff. And there’s now some technology that can help with that of doing better kind of delta testing, which we’ve seen.

So that’s where I start. Start small and grow from there. Yeah, yeah. And I think like, you know, I’ve given advice in the past of things that I, when I was at a security director for a company, you know, we, we did something very similar on the kind of the network side too, where we were like, hey, we, you know, I’m a huge fan of MITRE, ATT&CK and like, it speaks the language around, you know, potential breaches. And so we’re like, hey, we feel like we’ve got maybe some gaps in like lateral movement detection. So like maybe we should just test some things out of there every couple weeks. Right? And that, that’s, that that worked really well. And that, you know, whether that’s true continuous pen testing, I don’t know, but like, it’s at least trying to simulate an attacker’s true behavior on your network, which, you know, like, I think it’s the point, right?

So, yeah, yeah. The other point just to make too, because I mentioned the developers, especially in application security testing. You really should really. I mean, I advocate talking to your development teams and understanding their development lifecycle. You really have to have, the security team needs to be aligned with your development team and have a good relationship where you’re talking about, hey, we’re going to implement a continuous pen testing program and. And viewing them as a partner in that versus, hey, we’re the security team. We’re doing continuous pen testing. Here you go. You just have to comply. That’s not a good way to do it. So I see a lot more success when the security teams are partnered and have great relationships with their development teams that will go a long way.

Yeah, yeah, no, that’s great. I mean, and I agree. I think. I think those of us that have been around the security industry for a long time probably experienced a lot of the, you know, the tension that, especially early on when we were trying to convince people to, like, invest in security and, you know, that was, you know, I think we’ve all learned that, like, hey, you know, coming at it from a partnership perspective, like, here, we’re all help. We’re all here on the same mission.

Exactly. Yeah. Yeah. You know, nobody wants to be on the news. No, definitely not. So. Yeah.

Well, this is great, Tom. Thanks so much. I mean, I love chatting with you. Thanks for joining the cast. Like, why don’t I see? I think we’ll share. Like, how can. How can people, you know, learn more about your podcast and follow you? Yeah.

So if you’re looking for a weekly security podcast, you can check us out at sharedsecurity.net. You can find us wherever you like to listen to podcasts. We’re also on YouTube, so we’ve been almost 15 years now. We’ve been podcasting. So one of the, I think one of the oldest running podcasts in the industry, I dare to say.

Yeah, great. Well, awesome. Well, thanks so much for joining us. Really appreciate your time, and really, I know you’re a busy guy, but really appreciate it and look forward to hearing good stories from RSA.

Yeah, definitely. Thanks, Dan. Appreciate it.

Yeah, thank you.