The Evolution of Product Cybersecurity: Securing products, embedded devices, and the IoT

You know, jumped headfirst into that whole world, and really, really enjoyed using my electrical engineering background, development background to apply that to all things related to embedded cybersecurity. And then from that, did the world of consulting, seen multiple different industries and types of devices and different vulnerabilities throughout. And now at Solasec, and that’s my primary focus: how do we help organizations that are developing products, make them more securely, deal with them remotely, and make sure that we have a safer infrastructure supporting us with all these IoT devices.

Yeah, no, that’s great. I think your journey sounds very classic hacker oriented. You get started in tech in one area, and then you just in the classic hacker mentality of figuring out how things work, and then it kind of evolves into security. So that’s cool. Like, my experience was very similar, and, like, you know, we’re talking about embedded device security, product security IoT.
My background in penetration testing was being in appsec. And so when working with Veracode and Mayo clinic and a couple of these other groups, we got some fun projects. And because I had two degrees in computer science, and I actually had some experience with reverse engineering, I loved kind of binary analysis and then, like, picking up embedded systems code and trying to figure out the, you know, the gaps. I think one of my, I joke that, like, my non claim to fame was like, I actually got to work on a project. This was kind of before, you know, the car hacking stuff had actually become very public. But I had gotten to work on a reverse engineering project where we were testing some head units in some vehicles. I can’t name who it was, but, like, you know, some of the things that we found, they were code based. It was hypothetical, right. Eventually did come out publicly, like, not, you know, it was completely independent, like, of the work we were doing, you know, because obviously we were hired to go find this stuff, but these guys actually found it, you know, on their own. So it is kind of fun to kind of see some of the stuff that we had been exposed to and like. And actually hypothetical. Cause we didn’t have any dynamic testing. It was just static testing. Right. But that kind of stuff where it’s like, hey, this software is getting more and more embedded in our daily lives. As we all know, Internet of Things is as a true thing now. It’s not just kind of some hypothetical.

And these embedded systems are being used in very critical infrastructure. So it’s been interesting, now that I’ve kind of been out of testing, I don’t have as much exposure, which is why I kind of want to, you know, pick your brain on how things are evolving. But, you know, back when I was doing testing, it kind of was. It was variant, right? It was, you know, you would do some testing on some things that were very locked down and had some good security protocols in place, and then others where it was like, it was easier than a lab device in the OSCP, right, where it’s like, oh, it’s running a vulnerable version of Tomcat. You’ve got an easy exploit you can just go throw on it, and now you’ve broken into like an insulin pump or something like that, right? So it sounded really sexy and really cool. But at the end of the day, from a hacker’s perspective, it was not that, it was very trivial.

I’m sure things have evolved since then, at least I hope so. So I’d love to kind of get your take on, like, you know, how you’ve seen, how you’ve seen it evolve, you know, coming at it from that, you know, kind of that, you know, low level aspect.

Yeah, definitely. And I’d say, you know, the unfortunate reality is that it’s evolved in the security capabilities and the, you know, the recognition of some of these types of attacks. But it’s also evolved in just the threat landscape of these devices. I mean, that’s kind of the nature of IoT, is that, you know, everything, everything in your house is connected now. And, you know, you can make a strong argument as to why you need a connected refrigerator or something like that. So, you know, the surface is expanding. So I still see the easy OSCP level one type boxes when it comes to embedded product. And a lot of that is related to the drivers for some of these things are just first to market or the cheapest or whatever other externalities that result in thinking about security as an afterthought. Right.

You know, that really materializes into some pretty nasty IoT exploits in terms of the network. And then even more, you’ve got all these devices that have these components that are inherently vulnerable to more of these advanced attack vectors that we’ll talk about today. But that’s kind of the nature of how things have changed in terms of the landscape. But I don’t want to be all doom and gloom because I think that there have been certain industries, especially industries that are heavily driven by regulation, like medical, for example, or automotive or some of the industrial control systems to some extent are pushing more and more for security just because they understand the risk. Right. So, you know, the, the types of attacks where you’re running a, you know, a Windows XP system and you go download some, you know, Metasploit type module, right. And apply it to a system, you know, I think those times are hopefully, you know, few and far between now, depending on who you’re working with. So we’re getting much lower in the stack and understanding sort of the foundation of security that we’re building things on top of which is really exciting to see. Just as an example, seeing medical device manufacturers that are using commercial resources or SBC’s or something like that and then validating the security of that platform that they’re building on top of rather than just assuming it at face value is secure. And that’s the manufacturer’s problem. So as everything. It’s a spectrum of different maturity levels throughout.

Yeah. What have you seen, how have you seen just in terms of as IoT has become much more prevalent, like at the home as well as critical infrastructure, what have you seen as some of the common themes around some of the attacks in the threat landscape as that’s continued to evolve?

Yeah, I mean, I think that one of the most common things that I would say, is most of the time, like I talked about, the refrigerators and innocuous device. I think we’ve seen several instances where people will just, for whatever reason, not really understand the risk associated with some of these devices. And, you know, if we talk about like a DDoS type attack where multiple devices are compromised or you use one of those devices and it compromises your internal network, or those are used to attack a different target, that’s a big one.

But then I think similarly, some of the, you mentioned the industrial controls and manufacturing facilities and things along those lines, those networks that are historically not emphasized because of obscurity to some extent, or, you know, it’s difficult to, to maintain and operate things like a manufacturing line. You know, more and more those are not only under more scrutiny. And we’ve seen some, some notable incidents related to some of those targets, but I think also, you know, we’re, we’re understanding that, you know, those, those infrastructure networks, those, those facilities have a major impact on downstream things. Right.

So, like, imagine a manufacturing facility where IoT devices are being produced and then they’re being, you know, the firmware is being updated before they get sent out to the field. It never touches a back-end server again. You know, the supply chain risk associated with one of these manufacturing facilities getting compromised and then malicious code just being flashed at a production facility is an incredibly scary thing to think through.

Yeah. Yeah, no, it’s interesting. I mean, I think, like, you know, one would hope, or at least like if you are in those industries, you know, kind of, you know, making sure that you’re keeping these things on them, on your radar in terms of how you’re evaluating your threat models for where, where attacks could insert. I mean, where attackers could be inserted or where backdoors could be inserted, I think, you know, supply chain is a great example both in, like, the software and the, and the firmware. Right. I mean, our software and the physical. Right. And I mean, I think anybody that’s probably been in and around the government at any point knows the term, like supply chain interdiction. Right. So these are not new techniques or new things, but becoming much easier to conduct, especially from an attacker’s perspective. So it’s interesting that just how much more pervasive that kind of an attack can be. Right. And I think that’s the interesting slash, you know, nerve wracking, but,

Yeah, yeah, absolutely. And I think furthermore, you know, I think the emphasis now, like you mentioned, this has been around for a long time. I think with the emergence of IoT and just how many devices are out there? You know, the threat surface is just expanding so quickly as a result of that. And that’s why I think that understanding those facilities and the controls on these types of devices is paramount now because everyone knows that when something gets identified on devices that have potentially millions of systems out in the field, it’s too far gone to some extent at that point.

I mentioned previously the various attack vectors. Something that’s interesting to me is the prevalence of these low level embedded devices or even some more of the complicated devices that don’t have strong controls against things like side channel attacks, whether it’s power analysis or fault injection or anything like that, where those attack, and I’m happy to nerd out on some of that stuff, those things are, I think, becoming more and more out of academia and more actual exploitable things. Right? Like there’s, there’s a ton of hardware that can help facilitate this attack, these attacks, and then can also, you know, the bar has been significantly lowered in the last ten years or so. And then, you know, all these, you know, I don’t want to call out a specific vendor, but you have these vulnerable microchips that are in these IoT devices that are known vulnerable and you’re not gonna be able to do anything about them. So they’re effectively compromised as soon as attackers realize that, you know, how easy it is to do some of these types of attacks.

Yeah, no, that’s fascinating. Yeah, because like my background coming from the DoD was like, we had to do a lot of testing and accreditation, certification around things like tempest requirements and side channel attack, you know, but that was a very government classified type of orientation. Right? In terms of like, what did they call at the time? Like EAL, like whatever, they evaluate assurance levels, right? So, like, systems that were going to be put in like sensitive areas, like you had to get all this kind of certification around it. And now that like, hey, those kinds of attacks that were fairly theoretical or took like nation states to actually conduct, like, you know, how often is somebody going to have the capability to do inference based attacks on like power out, you know, like, you know, power emissions, know, and so, you know, it’s like, yeah, very, very plausible for somebody that had, you know, unlimited time and resources, but like, you know, your average attacker, you know, wasn’t going to be able to do that.

And now I think you’re, you’re kind of highlighting like, hey, those things are becoming more, more real. And if you think about like, hey, I’ve got a coffee pot attached to you know, you know, a network that is, if that’s the initial vector in and then you can start, you know, being able to, to use that as a pivot point. These are all things that I think, you know, the IoT and the, you know, the low-level attacks, you know, are becoming much more important to, you know, to be aware of. I’d love to, I’d love to nerd out a little bit like, you know, what, what are some, what are some examples of some things that you’ve seen or like, you know, tested against the, I mean, you know, broadly speaking.

Yeah, yeah, yeah. So probably my favorite attack that hopefully people know about. And I’m by no means the premier expert on this type of attack, but correlation power analysis or correlated power analysis is by far the most exciting to me. So basically, it’s this notion that you can look at the power consumption of a device and then you can correlate that the power consumption directly to memory effectively. Right? So it’s this idea that, you know, the number of bits that you said is directly correlated to the power consumption. Right, which, which makes sense at a fundamental electrical engineering level. Right.

But what you can do. So just a simplistic example. So if I’ve got something where, you know, it’s easy for me to see ciphertext going in, and then I know that ciphertext is being decrypted, I’m guessing it’s, you know, AES 128 CtR ECB or whatever, whatever it is, I can collect power samples and then I can take that ciphertext that I observed in addition to a basically brute forcing an AES key. And then I can just go through and calculate what the output and what the profile or data set should look like with those AE’s key guesses in terms of the bits that are set and what the power profile should look like. Instead of just brute forcing the AE’s key, which is obviously going to take inordinate amount of time, I can correlate those two data sets and say, this is what I’m seeing from an AES perspective. This is what the power should look like. And then I’ve got the actual power itself. Then when I correlate these things, it’s very easy to go and reverse engineer the exact AES key that’s happening and then compute the main AES key and then you can compromise, you know, that cryptography to some extent.

So that’s one I just love because that’s the perfect convergence of my electrical engineering background directly to cybersecurity. And it’s something that, you know, I won’t say that, you know, basic IoT devices should, should introduce security controls to prevent against that.
I think we should just know as an, as an industry, these things are on the horizon. Right. And you should, you should understand that that is a risk in the future for sure. Right?

Yeah. Yeah. Like, like what used to take a team of people that had resources within nation state or high-level resources behind it, took a team. Now it’s in the realm of, hey, boutique consulting firms like yourselves that have backgrounds in this and have knowledge but don’t have the resources of a government behind them can now conduct those attacks. And then that’s the natural evolution of, hey, as we keep moving forward, there’s going to be more tools and techniques to the quote unquote script kiddies that then you don’t have to have as in depth knowledge or understanding of how to conduct it to still be able to do it.

So I think that’s important and I think it’s important for vendors and product security teams of these vendors to really keep that on the horizon because you just don’t know if they’re hiring folks with that level of background. It’s more general security folks and whatnot, which is fine, but just these are the kinds of things that people have to keep learning.

And not to just mention it for the algorithm or anything, but I mean, AI is a non-zero factor in all of this as well. Right? So when we start talking about correlation power analysis or we start talking about how to do some of these things and perform robust analysis on some of this difficult to manage data at some point, I mean, I think it’s only a matter of time before we start to see some of those capabilities emerge from that perspective.

Yeah. Yeah, well, I mean, like, I know from a hacker perspective, I loved getting to work on some of those things because it just, they were definitely more unique engagements and, you know, things like that. And so I’m sure, I’m sure that others of our ilk in terms of like the hackers and pentesters would love to kind of have some tips and tricks. How would you like, you know, how would you tell somebody to get, you know, where would you point them to kind of get started in product, you know, embedded and stuff like that?

That’s a good question. It’s funny, I was thinking about it when you were talking about just your introduction into hacking and kind of mine as well. When I was an embedded software developer, I used to get so upset with my manager about, he would make us actually bit shift into registers on embedded devices. And it’s like, hey, there’s libraries for this, there’s hardware abstraction layers. Why are we doing this? You know, I think I still feel strongly about that argument, but I don’t know if he meant to do this or not. But, you know, that notion of understanding exactly the registers that I was setting, understanding how to read the data sheet effectively and, you know, why I should enable a peripheral clock, for example.

Then I got to the point where I started doing reverse engineering on bare metal microcontrollers and I saw all of this, you know, these addresses with these offsets, and it was bit shifting certain pieces and it looked the same to me effectively. So I thought that that was kind of funny that, you know, going through that trial led to me being a better reverse engineer. But, you know, to answer your question, I think that that’s exactly it. You know, to properly secure something and provide recommendations on how to, how to secure or how to hack it even. I think you need to understand how to build it.

So, you know, one of my biggest recommendations, especially in the embedded world, is get your hands on, you know, a little discovery board, write some embedded code on it. There’s all kinds of wonderful, wonderful resources nowadays that really lower the bar, even from when I started not that long ago, that allow you to start writing that code on the embedded side of things and then understand, you know, write an embedded vulnerable application and understand how you would secure it. You know, that to me that’s the best way that I’ve learned in my time.

Yeah, yeah, no, I think that that’s fascinating. I mean, like, I loved reverse engineering, too. And I mean, this may or may not be as relevant of a skill in the embedded software realm, but, like, being able to understand like how much, how much space you might have in a very, like, you know, because embedded devices, they’re very, they’re obviously, you know, hard, limited on a lot of things, right. And like, out in the field, it’s out in the field, like, you can’t add more ram to it or, you know, so if you did find a Metasploit module that might work, you had to really make sure it fit within the registers and all those things, or at least the payload. So that, that was what I loved about it, too, because I really enjoyed reverse engineering, you know, but it took, it was very time consuming. Right.

So, like you. But yeah, yeah, I’d say, I mean, arm general, you know, most of my experience has been with arm. Arm core microcontrollers. And I think that there’s other architectures that are coming that it will be in a similar boat. But so much of what I’ve seen just in the industry on reverse engineering is heavily dependent on the things that are out, like X86, for example, for obvious reasons. So there’s really a ton of work that we can do as an industry in reverse engineering some of these low level things. I think it’s exciting. It’s exciting to know where we’re at just because there’s so much, you know, you rip apart one device, you can have multiple different operating systems and architectures just in a single device. Right. So it’s always fascinating. There’s a lot of work to do.

Yeah, yeah, no, that’s cool, that’s cool. Okay, so we talked about the hackers and like, kind of tips and tricks for them. What would be some advice for like the blue teamers or the product security teams that are, you know, we alluded to it a little bit, but I’d be curious, you know, what your advice would be for them.

Yeah, I mean, for, obviously my, my focus is on embedded, so I’ll answer it from that context. But I think the same principles apply. Introduce security as early into the product development lifecycle as you can, and you’ll have better days if you find something. So things like threat modeling and understanding realistic attack vectors, not just at the network and application layers, but also at the low level hardware layers. So that’s a big one.

I think another big sticking point for me that people will say I talk about too often is just the idea of secure code scanning and doing effective secure code scanning. That’s a very difficult thing on an embedded system that’s running an RTO’s or something like that. Having something that’s actually meaningful and identifying those different tools that are helpful to plug in, but don’t just provide a ton of false positives because they’re built for a different architecture or a different ecosystem. I think that’s a really important thing for us as embedded developers to keep a close eye on, for sure.

Yeah, no, I think that’s good. Yeah, like the classic principles around like software development lifecycle and all those things. But also we mentioned, you mentioned it before, the supply chain. So like your s bomb and stuff like that is still really, really important, even if you don’t have the skills on the reverse engineering side, but still understanding, like, hey, what libraries are you packing into an embedded system and what are you making available to the attacker once they’re in, because with a larger operating system, they may have additional APIs and libraries available to them for their exploits. But on an embedded device, you might be living off different lands, so to speak.

So I think it is important to emphasize, like, hey, think like the attacker and try to, try to learn their techniques, you know, get, get some exposure to reverse engineering because like you said, like, I had a lot of background in X86, not only on like, assembly language coding, but also reverse engineering in an X86 architecture. But when I started doing some more embedded device, that’s when I got introduced to arm. Right. And so, like, and it was just, it was like a mind meld for me. I had to. It took me some time to understand how the registers were working differently and all that, so. But, you know, understanding, I think like any, like any true hacker, even if you’re a blue team or like, understanding how things are working under the hood only gives you more advantage to understand how it could be broken. Right?

Yeah, yeah, the last thing I’ll say, too, and you kind of touched on it already, but, you know, make. When you have these conversations early on into a product development lifecycle. Right. Early on into the initial prototyping, you know, give yourself room to be able to incorporate some of these things. Right.

Like, you know, give yourself the potential for a secure element or a TPM or a trusted execution environment if you can. Or, I mean, you know, don’t, don’t lock yourself into 96% CPU utilization going into it, and then you don’t have any ability to add any security at any point in time. Right. So just make you use the information about security to give yourself the opportunity to do security. Right. That’s the key point there.

I think that’s really good. Yeah, that’s a really good point. So. Well, Caleb, I mean, it’s crazy how fast these sessions go, but that’s why I love them. It’s like, just get the nerd out and enjoy talking to really sharp people like yourself. So, yeah, I mean, one, thanks so much for, for your time. I hope. I hope people found some good takeaways, but how can people find you and what you’re working on? Let us. Let us know.

Yeah. So, you know, we’re, we are full speed with, with Solasec. So you can, you can find us on Solasec.IO. There’s multiple fields to get in contact with us and then we’re pretty active on LinkedIn as well. So, you know, we, we try to publish blog posts and thought leadership type materials like that. So keep on the lookout for that. And then, you know, feel free to reach out if you see me in the community. DEFCON, you know, all those types of things. We try to stay as involved as we can.

Awesome. Awesome. Well, yeah, and thanks. I mean, obviously, we’ve, we’ve worked with you in different capacities for many, many years now, so it’s been fun and. But, yeah, thanks so much for coming on. I really appreciate your time and expertise. I think this is a really fun topic, especially for me because, like, it brings back some of my day, you know, of what little I did. I really enjoyed it. So.

Yeah, yeah, no, thanks a bunch.

And everybody, if you have questions or comments, feel free to, like, leave them in the chat. We’ll try to get back to them as soon as we can. But other than that, we really appreciate, Caleb, your time. Thanks so much and thanks, everybody, for joining us on a Friday. Hope you guys have a great rest of your day and a great weekend. We will catch you next time. Awesome.

Yeah. Thanks for having me.