Episode 3: Crowdstrike, North Korean Spies, and CISO Scapegoats
Welcome to the Distilled Security podcast episode 3. My name's Justin. I'm here with Rick and Joe, and we're glad to have you here. Don't forget to like, comment, and subscribe. It helps, with our viewership and, distributing this around here.
Justin:I think we have a pretty good episode lined up here. We got a number of really good topics. A lot of things has happened over the last month, a couple of things. And I think we're gonna get into pretty good conversations here.
Rick:I think so.
Justin:Yeah. So to start off with I mean, it was a little thing that happened a couple weeks ago, but crowd strike, with this. So, if the audience isn't aware, there was a security vendor, you know, monitoring detection and response. They pushed something out, and it was bad. Brought down mainly Windows systems, all Windows systems into that, and a lot of damage.
Justin:I think you had some stats onto that. Like, how how many 1,000,000,000 of dollars was estimated to be lost
Joe:into that? It was huge. And, the first thing I think we needed to do is correct it. Crowdstruck is what I understand.
Rick:Oh, that's tense as the
Joe:at this point.
Justin:Yeah. Yeah. I always have to correct my kids on their tents. So
Joe:No. Some some big stats. You know, just 5,400,000,000 in direct financial loss is one of the quotes. Another one is Delta, Just today, I believe, was announcing that they were going for damages from both CrowdStrike and Microsoft in the area of $500,000,000.
Justin:Why Microsoft?
Joe:They looped them into it because it was a Microsoft, they were running.
Rick:It's like the deployment mechanism. They're
Joe:pulling it together for both. Interesting. Yeah.
Justin:I don't know how they would have fault into that, but I guess they throw it in the courts and be like, yeah, whatever sticks, you know? Absolutely.
Joe:Yeah. They brought in, somebody supposedly a heavy hitter for the kind of attorney that goes after this. And so you look at all the deep pockets. That's my guess.
Rick:Easier to settle and
Justin:And I wonder if it could eventually turn into a class action, you know, with something along this line.
Joe:I don't know, like, that's interesting.
Rick:I was wondering if I haven't seen anything on this, but maybe you guys have. If anyone I don't know if anyone that, like, has tried to invoke their business interruption insurance or anything like that either because I mean that's a big number, you know, 5,600,000,000 or whatever it was. I don't know. If if a company was truly down for days or whatever, if you're retail or something like that, like, I could see trying to be, like, well, it's not exactly an act of God. Right?
Justin:And maybe you know more of it. Like, I've seen that floating around. I don't know
Rick:too many companies that hold that. This interruption? Yeah. It's usually part my my in my experience, it's usually part of a package. So, like, if it's sort of like, oh, I need these three things, and you well, if you buy it's like the Comcast.
Rick:If you buy the bundle Yeah. Right, you get, like, this discount. Premium bundle. Yeah.
Justin:You know, this is also
Joe:thrown into I mean, that's part of my company's company's insurance plan for a degree. You have interruption. You can't fulfill certain services because of something that happened that's a a claim. Right.
Justin:It has to be like and I've seen some of the policy, like, it has to be over an hour and some other stuff. Like
Joe:Yeah. There's mine's even much longer. Yeah.
Justin:Okay. Gotcha. But I
Rick:would be interesting interested to see if there were, like, if underwriters or actuaries or whatever had, like, data on the claims that were filed based on some of that stuff. That'd be because to a large extent, like This
Justin:would be a claim. I I don't see how you can like, it was a vendor, you know. If you if you choose to
Rick:invoke it. Right? Because there's a whole bunch of pain in terms of getting paid out potentially.
Joe:There's gonna be lots of case studies on this. We should revisit this in 2 months, see what's unfolding from that. Plus, if there's a law lawsuits that are already happening. Yeah. Now the one of the things about the lawsuit is that it's the first one where somebody officially came out and estimated the $500,000,000 in damage.
Joe:So that was, kinda new for today. And, like, Delta's got some good things to complain about. 7,000 flights canceled. Yeah. They're gonna have to do a 176,000, refunds and reimbursement requests.
Joe:They got the process. So, yeah, that's not not little.
Rick:Yeah. But I I rarely see software vendors no matter who they are say, oh, yes. We're we're taking responsibility for, like, how you implement it or the systems you implement it on. Right? So, okay.
Rick:Well, not our fault. You put it on these business critical systems. Right? So, like, I don't know. I I don't know if
Justin:I mean, but that's our goal. That's our whole stick is put it on the the business critical systems, you know, type of thing.
Rick:It is. But, you know, but I I I can see I can definitely see arguments on both sides of the case where it's like, okay. Well, you didn't have appropriate business continuity. Right? This was clearly a single point of failure for you.
Rick:Yeah. We we had an error in our software, but it caused you a bunch of heartache. Well, you know, that's some of that's on you. You have you have these business processes. Like, I'm not saying right, wrong, or indifferent.
Rick:I could see that argument. I mean,
Justin:they can't make that, message publicly. Like, you're an idiot for running our software. Come on. Wow.
Rick:No. But not you're an idiot for running the software. I know. I know.
Justin:But What
Joe:do you
Rick:what do you have to set up in the background? And I think a lot of times, you know, there's there's the court of public opinion, and then there's the courts of getting sued. Yeah. And I think sometimes different arguments are made in different I mean, the
Justin:crux of it is it's gonna happen. It's happened with a whole bunch of breach stuff as well. Like, it's gonna last for years. Yeah. And then they're gonna arrive at a settlement.
Justin:That's gonna be a fraction of what they entered into.
Joe:I heard that today as, somebody else made a comment that what Delta's doing will never make its way all the way through. It will end up in a settlement. That's how they expected.
Justin:And the only people that win are the lawyers at the end
Rick:of the day.
Justin:We we should have a whole
Rick:other segment at some point about cyber insurance and stuff like that because I have some strong opinions. Yeah. Yeah.
Joe:Well, something else that, jumped to mind was, one of the articles I read shows the a factor of I I can't remember if it was 23,000,000,000 in amount of value that CrowdStrike dropped Oh. Based on this because their stock's way down.
Justin:It's not sustained. I haven't looked at
Rick:it for a while.
Justin:They were lost. Still weighed up.
Joe:Yeah. Yeah. Yeah. As of the luxury renewal.
Justin:Cover, especially within a year, you know, in most companies. I saw the day that happened. People are like, discount on CrowdStrike stock,
Joe:you
Justin:know, type of thing. Like, a lot of people look for that now.
Rick:Interesting. Yeah.
Joe:And if you wanna see some, interesting back and forth, the SentinelOne CEO, Tomer Weingarten, had been interviewed. And I don't have all the, you know, facts memorized, but a lot of calling CrowdStrike out for just doing things incorrectly, And then the integrity of the article, had a link to a rebuttal by CrowdStrike. So, the gist of it was you should never be doing this kind of work in the kernel. You should be doing it in user mode. You should be, you know, not really getting to that level the way his architect did, and it's really not a force majeure or an act of God kind of problem.
Joe:It was a bad architecture. He is claiming that just CrowdStrike had a bad architecture. Now, of course, he has a competing product. And the whole reason he was doing it is because he wants to, you know, bring up some positives on to what he's doing. Right.
Joe:Well, it was good. It was very compelling, but then I had to read the rebuttal. And the rebuttal was also very compelling. And
Justin:I would think you'd have to be in the kernel to prevent some of the malware that's out there and everything. No. But
Joe:They're saying that SentinelOne doesn't do that, and they can still prevent it. However, in the rebuttal, they, CrowdStrike says that they're able to do 11 times faster
Rick:Right.
Joe:Because of where they're operating, and they're following the Microsoft That's
Rick:what I gotta say too. They also said, and we did actually follow the process for doing like, the defined process for doing the thing that we're
Joe:Yeah. 701 said they weren't.
Rick:Right.
Joe:But then they responded and said we're following each one of those items. So, you know,
Justin:we're gonna believe it. Drama there. Yeah. It'll get flushed out. But an interesting thing why I wanted to kinda bring this up is, do we plan for alternate vendors in this type of thing?
Justin:Like do we have backups, or do we have like continuity to say, like, we have 2 MDR vendors, you know, and one goes on one, one goes on another? Like, what how far do you take this?
Joe:Well, that's what the SentinelOne, CEO was actually saying. He's saying, hey. If you were on the fence, feel free to introduce, our product in for a segment. That way you're not running everything in the same way as the gist of where he was going.
Justin:Feel like we're back in the, like, late nineties because wasn't that a thing, like, you need to use 2 separate, like, firewall vendors in case one had, like, a Well, sure. Those were
Joe:those were more in serial, and they're saying 1 half of this network, 1 half on this tool, and that that seems like a maintenance nightmare. Mhmm.
Rick:It it does it does seem like, you know, I mean, in in a way, I'm gonna bring up PCI again. Give you but but in a way, like, I mean, there's all the all the process. Like, hey, you have to have, like, the the the old knuckle busters and stuff like that. Yeah. Right?
Rick:Because if the tech goes down, you still need a secure way to do the things. Yeah. So it it actually puts me in the mind of that kind of stuff where it's like, look, no matter what technology you use, there are gonna be, like, unknown failure points.
Justin:Mhmm.
Rick:And a big part of business continuity is like, hey. If this piece of technology or this tool that we use isn't available anymore, can we still do the hypercritical stuff?
Joe:So what's the difference from, of this issue and being ransomware ed but recovering quickly? Same outage scenario, you still are unable to use your tools, and you still, from a continuity perspective, need to have manual processes, which is I really like where you're going
Rick:with that. True. I I think I think some of the thought process might say though that if you're ransomware and you have, like, good immutable backups or whatever Yeah. Not that everything's just like a push button restore Mhmm. You can do that.
Rick:Part of the real pain with this one was you're in this blue screen loop, and if you have, like, BitLocker on an older OS, you have to literally manually touch the machine to to solve the issue, which is a huge pain.
Justin:I had a client that got hit with this, and they're a convenience store type thing. So locations run out, you know, in United States, and they had to drive to every single convenience store location. Yeah. I know. It was are they back?
Justin:Yeah. They got back. It basically took them just about a week. A week?
Rick:Yeah. I I have a buddy.
Justin:But it was all hands on deck.
Rick:Uh-huh. That they like. He runs infrastructure at a at a big organization, 15,000 endpoints affected, but a lot of them were remote workers. Right? So then it's it just introduces this whole thing, like, well, if someone lives, you know, alright.
Rick:30 miles, okay. 50 miles, you know, 200 miles from
Justin:the office. So what they do, just ship?
Rick:Yeah. They did ship ship your thing back. We'll ship you the thing. And in in their case, they had. That's days.
Rick:They had exactly. Yeah. It's meant like backup
Justin:pocket. Yeah.
Rick:So everyone local did the local thing, like, you know, and everyone that wasn't local, they shipped the thing. But to your point, yeah, it took them about a week. And and again, same thing, like all hands on deck and pause everything else and all that stuff. So, yeah. But to to your point, Joe, I think some people would make the argument or could make the arguments, like, well, with a ransomware thing, there are certain patterns that don't require the same level of manual distributed intervention to solve.
Rick:But it is I think conceptually, there are similar different solves, which is, like, look, what's the business process? What's what are the critical elements of the business process? And can you do it without tools, or is it so baked into the tools that you have to use? Like, Delta flights, I would suspect there's not a lot of that level of logistics you can do without machines to do it. So in that case, I don't know if it's, like, one for every 30 gates at an airport or whatever, but you maybe you do have a separate thing, and all of a sudden you can at least, like, get some things through
Justin:Mhmm.
Rick:Or or whatever.
Justin:Yeah. I
Joe:just don't see how that would be practical in the kind of place like an airline unless you're Southwest and running, what what's the claim? There's a lot of rumors on that.
Rick:Yeah. There are a lot of good memes on that. Yeah.
Joe:I I saw something funny on that. But, bifurcating the 2 ideas, the one, that you're talking about is, you know, the all hands on deck and recovery and those kinds of things. And then the other thing that we're talking about is, well, what do you do to keep your customers moving with the knuckle busters or things like that?
Rick:For backup technology, it's not that different than, like, a warm site from a Doctor perspective. It's just endpoint world if you're talking about some of these systems. But it's not like a bunch of servers weren't impacted too.
Joe:Right.
Rick:You know?
Joe:And if I was running a backup site, I'd probably wanna have the same EDR protection on it as I'd everything else, and they would have just been
Rick:hit too. Well and then it gets infinitely complex. Right? Because how many elements of security software or any kind of software are on these endpoints? And so how do you ice do do you have a separate endpoint that runs literally an entirely different stack?
Rick:In case any one of the ones goes down, you can back up to, like, you can like, it just doesn't make sense.
Justin:Yeah. And and that there's some truth to that, but I think that this is a unique case where you don't have too many security software embedded into the kernel, you know, type of thing. Like, if you have log monitoring, that's not at a kernel level. You know, that's some agents shifting off logs to another, point. If it crashes, it just crashes.
Rick:You know? It's fair. I think that's a good point. But but this is also one of the situations where it's like, well, no one expected this type of blue screen loop to happen even though it's embedded in the kernel. And therefore, I wonder what else it's that unknown unknowns.
Justin:Well, and it gets in that, like, so MDR, we're talking about MDR with, CrowdStrike, but, like, what about, like, some type of password management, like, Dicotic or, or they call it Delinea now, or CyberArk. Absolutely. Everybody's, like, embedding those, you know, password management into that, and that's a point of failure. Even though they have, like, guidelines, like, here's how to back up and here's how to maintain this and everything, they push out a bad update that it could be hours, days, you know, to recover and get access.
Rick:And And it also makes me think, like, hospital stuff. Right? Because, like, again, they have to have alternate manual procedures for quite a bit of things. Right? When you act when you get into the nitty gritty of it, and they have separate continuity plans and all that stuff.
Rick:But it's a huge pain, and it's notorious for having things segmented in ways that security people don't like. Right? Because you can't upload all the patches. You can't do all, like, the things that are typically wrote good security. Right?
Rick:Or, like, oh, yeah. Don't let anyone know the the the password for this service account and rotate it automatically. Alright. Well, that's if that, you know, you snap your fingers and that's gone now, you oh, wait. What do we do?
Joe:Right. Yeah. Right. Yeah. Unlike LastPass, for example, how many people just be able to go to a LastPass or another one of the browser integrated, systems?
Joe:Was one of the ones you mentioned one of the browser integrated systems that your end users might use?
Justin:Well, I mean, they have those type of where you can retrieve the password portals and everything. So the Linea used to be psychotic, you know, that type of thing. It's really that or CyberArk is more expensive. Right. I want it every day.
Joe:And those are more like the system administrator level. Yeah. But they have where
Justin:you can go in and get passwords out of that Yeah. Type of thing. Yeah. But then you got the traditional last pass or one password or something like that where it's just a storage of passwords.
Joe:And you've had that blended into all of your your teams
Rick:Right.
Joe:And all of your employees now are getting that level of protection, but there's an outage that makes it, impossible to get to that. It's just another supply chain Right. Single point of failure. Yeah.
Justin:Yeah. I'll tell you now, like, just talking about this, like, I have a high reliance on 1 password. If they go down, I don't know any of my passwords anymore. You know, they're all auto generated, and
Rick:you
Justin:need to every single site. Like, I would be screwed if their service went to.
Rick:And to that point, like, you're talking about, oh, well, what do you do? Run alternate technologies? Well, completely infeasible for, like, a password or a secrets vault.
Justin:Like, how do you dump the entire database to a local, like, storage every now and then, I guess, would be a feasible thing, maybe.
Rick:Well, I it depends on what the issue is. Is it inaccessible or is it, like, corrupt? Yeah. So
Joe:Well, yeah. In in either case, you don't have access to it. So it doesn't matter if it's corrupt or inaccessible. You just have your own disruption. So exactly from, what you're saying.
Joe:So then what happens? You get people who, you could, but you get people who will then wanna export it. And where do they export it to? And they paste it into a OneNote sitting on a unsecured part of their laptop Right. That's only protected by whatever hardware encryption they're using Right.
Joe:If at all. Right.
Justin:Yeah. Well
Rick:and and I think the other consideration, which I'd be interested if there is ever any analysis on this or ever will be, but, like, you talk about, like, the direct images and all that stuff from this. I wonder how much shadow IT occurred in response to these things not working. Right?
Justin:A a bunch
Rick:of so so a bunch of people just wanna do their jobs. Their work machine is no longer able to do their job. So, okay, well, I still gotta do my job. I guess I'll just do it over my phone that I've been told, or get my home PC, like, all these things. Like, I know I'm not supposed to do this, but I'm also completely dead right now.
Rick:Right. And you use
Joe:this gaming PC that has, who knows what's it infected with.
Rick:So I do I mean, we're getting into, like, 3rd or 4th order conjecture, but I do wonder That's
Justin:a real dedication from an employee. Wow. Yeah.
Rick:I think it happens, but it depends on how much stress there are and like what it happens. But like, this
Justin:is a mandatory vacation. Alright. I'm taking it. Yeah.
Rick:But I do wonder how many like insecure modes of operation were implemented in response to the formal channels being out, and if that'll have second or thor 3rd order repercussions because now certain data might be available on someone's home PC or certain things might be there might be paths inward that were not there before.
Joe:Mhmm. And how many GRC teams were actually invoked in order to go through and create an actual exception process to the policy that was violated when that error
Justin:Oh, wow.
Joe:So that Yeah.
Rick:No. They're all they're all
Justin:going to, like,
Joe:us know.
Rick:Yeah. Right.
Joe:How many people who are watching this, actually knows of any enterprise who has a record of doing an exception process to their their policy for using alternate things before it happened. Right. So Yeah. How many black swan events do we need before they're no longer black swan events?
Justin:It's just the just the just gray swans. You know? Yeah.
Rick:Just birds. They're all just birds.
Joe:So well, SolarWinds jumps to mind too. Right? So that is another, issue that was different. Mhmm. Didn't cause in that kind of outage, but it allowed everybody to get, a lot of the bad guys to get access to any place that got pushed the update.
Rick:Right.
Joe:Just like what going on 2 years now?
Justin:Yeah. Yeah. Yeah. Yeah.
Rick:Yeah. Now now it puts me in the mind of, like, the CIA triad. Right? So it's like, okay. Well, we had, like, the confidentiality.
Rick:1 was Zola. 1's. We had, like, I I guess I would consider it I mean, it is availability. Like, okay, well, when's when's the integrity one gonna hit? What's that
Justin:gonna hold on?
Joe:What's the integrity one? What's that look like to you?
Rick:It's the some crazy widespread database. It's it's corruption of all the password databases. It's widespread. Some encryption key that you can't do.
Justin:That's blended with availability?
Rick:I mean It is. I mean, it's it's kind of a piecemeal part.
Joe:But here's the one I'm worried about is the one that goes through and slightly modifies
Rick:Right.
Joe:All of the doctor's orders. So all the medication for all the people in the hospital is slightly different than what, it said, and nobody knows the check and nobody validates it.
Rick:Or I transpose 2 digits in in every field across this whole database or whatever.
Joe:And every one of those is in your bank account in your bank account. Yeah.
Justin:When you get into finance, yeah, integrity has a whole different meaning. Right.
Rick:You start pulling that apart.
Joe:I mean, you talk about 0 defects. I had a CIO who was huge on 0 0 defect, Chris Kowalski, and we talked about that a lot. Yeah. And his point was, do you ever wanna pull up your bank statement and find it to be wrong? Right.
Justin:As if it's wrong up. That's monopoly. Bank, carry
Joe:me a favor.
Justin:That's never been my $1,000,000. I like
Rick:it. Yeah.
Joe:I've never had that one, Robert. Yeah.
Rick:But then you don't say anything. You do not pass go, and you go direct to
Justin:the real.
Rick:Yeah. Yeah. Yeah. But yeah. The but, like anyway, I I think it's interesting because you're absolutely right.
Rick:Major events, vendor driven, confidentiality, integrity or confidentiality availability. Are we thinking what's the integrity one? What could it look like? Where where are the weak points? Where are the single points of failure?
Joe:Yeah. So here's why that may not be something that somebody goes after. Because how do you, as a bad guy, monetize the integrity one? That's gonna be a little bit harder because you've changed the data, but you don't really tell anybody.
Rick:Just keep thinking of office space.
Joe:Right. I
Rick:mean, you take a penny from any transaction.
Justin:Fraud would probably be into that. I don't know if that's kind of concealment of actual source, you know, into there.
Rick:Well, and errors and attacks are different. Right? So, like like, the the I think the major vendor issues we're talking about were more I mean, it could you could envision something that's predicated by an attack somehow and causes one of these things, but Mhmm.
Justin:And, honestly, the CIA trade has been broken for years since this introduced it. You know? So it's it's not full encompassing.
Rick:No. Right. Of course.
Justin:Of course. But it but
Rick:it does make me think, like, yeah, what what would you know, in terms of, like, the what could go wrong, it's like, okay. Yeah. Well, if if I not not I can't see it. I can see it, but just not right. Mhmm.
Rick:And how do you and and, honestly, that's one of the scariest things from a data perspective because then, like, how do you how do you untangle that? Do you have the backups? Do you have the
Joe:Oh, yeah. Availability and, confidentiality are both side effects, or the results of, like, ransomware. Yeah. And then this one was availability. Hopefully, there was no data corruption or data leakage through CrowdStrike's item, but
Justin:you're right.
Joe:I haven't seen the big integrity one hit yet, though.
Rick:No. No. No. And I do wonder what, like, the second or third or impacts will be like in terms of, like, oh, you did this thing on your phone? Really shouldn't have done this thing on your phone.
Justin:Yeah. They usually something wrong, you know. Yeah.
Joe:Yeah.
Justin:Yeah. I had something, I won't say which company I was working with, but we had something right before we're doing a rock. We implemented a new encryption solution months past for our core database and everything. And we were having this issue, didn't think it was anything related, but we're having data integrity issues. Like, random things would just get corrupted and they'd have to do a full, you know, restore and everything.
Justin:Finally, after troubleshooting, found out it was encryption, solution that we had over the, it was MySQL, you know, database and everything. And, of course, we're, like, weeks away from getting our rock, and I made the hard choice. It was actually came down to me, like, is compliance more important or data integrity? I'm like, data integrity. Alright.
Justin:Let's remove it, you know, type of thing. And we we were noncompliant, and we had to scramble to find a new solution
Joe:Right.
Justin:You know, before we could, certify that. And then actually, we've worked together. It was like a month and a half, you know, delay from that. So about a month, you know, from our PCI, that were delayed. But, you know, if when you come down to it, it's like, integrity is more important.
Rick:Security supports the business. That's how that works. You know, type
Justin:of thing. And we even considered leaving it on until, like, after the QSA, but it was causing so much issues, you know, with our production. It was like, yeah, it's not worth it. Yeah. It's not worth it.
Joe:No. I love that. And I always say compliance is a side effect of just doing good security. And what you did there was good security. You manage the, organization effectively working for the purpose of the organization.
Joe:And I don't know if you had a compensating control of any way or
Justin:I mean, I already had any layers, you know, into this and everything. Yeah.
Joe:So lots of compensating controls, acceptable levels of risk. Yeah. Did you properly go get an exception signed off by leadership that you're gonna skip that control for a time period? No. Shame on you.
Rick:Yeah. He it was earlier in his career. Yeah. Yeah. Yeah.
Rick:Yeah.
Joe:So kinda tying this all up is, well, what what happened? We just had another oh, yeah. Yeah. And we'll go with that. Now tying it all up.
Joe:Yeah. CrowdStrike, just another one of your supply chain risk management items you have to worry about, and it introduced a new black swan event type thing that you haven't thought about, but is there. And so, you know, what do we do?
Justin:Yeah.
Rick:You you know how we have, like, 100 year events? Like, oh, it's a 100 year flood or a
Justin:100 year fire? I I wonder every, like, 2 or 3 years.
Rick:Well, it's true. But I wonder if we're gonna get to a point where you're like, oh, yeah. This is, like, this is the 50 year outage. Like, this happens roughly once every 50 years, like, across a major vendor.
Joe:So We should come up. Is there a Moore's Law that goes with this gets shorter time frame? So a 100
Rick:years back now. Complex and faster. I'm I'm sure
Justin:but, honestly, like, the slammer weren't, I think, hit harder, you know, type of thing. Like, and some of that, like, historical stuff, I say historical,
Rick:you know, of, like, 15,
Justin:20 years. Yeah. But, like, I think that was longer outages than what a lot of businesses had, you know, like, most recovered within a week, you know, or a week basis, you know, type of thing.
Joe:Like Financially, though, different impact. Right.
Rick:Financially, the tech was integrated in a totally different way now. Right? The speed to like, reliance on and expectation and speed to transaction is, like, so totally different.
Justin:And to CrowdStrike, I mean, I'm eager to see, like, all the faults that were into that, but to their credit, they quickly pushed out a new patch within hours, you know, type of thing. Mhmm.
Rick:Yeah.
Justin:And they push out as much help as they could.
Rick:That's right. The bad thing happened Yeah. Already, and you can't like, they can't go and touch every end user's PC. But, like, everything that they could fix remotely I mean, they they they did respond fairly quickly.
Justin:Yeah. You know, credit where credit's due, you know, into that. But Yeah. Yeah. I was just thinking, like, yeah, they're we've had a lot worse issues and stuff out there that lasted way longer, you know, with that.
Justin:I mean, ransomware is just something like it is not as wide spread as in happening all at the same time, you know, to businesses, but it's way more impactful than this thing is. Even if you had all the backups, it takes a while to restore everything. You know? Oh, yeah. Yeah.
Justin:Yeah. So that's for sure.
Rick:I like that conversation. It went places I didn't expect.
Joe:I think we beat this up pretty good.
Justin:Yeah. I think so. Should we move on to this deliciousness that Joe has brought?
Joe:Oh, it is good.
Rick:It's malty.
Justin:What do
Rick:you got, though?
Justin:We got rabbit hole. It is construction, straight bourbon whiskey, 4 grain, triple malt, which a lot of us, we had to look up what malt was.
Rick:Spent way too much time on it.
Justin:We had a idea what it was, but we're like, what what is it exactly?
Rick:I was very incorrect.
Justin:Yeah. So malting is essentially taking a grain, soaking in water to allow it to essentially open up so the fermented sugars are more available, and then essentially drying it in hot air to basically it doesn't toast it, but it kinda, like, toasted a little bit, and gets ready for it to basically put into the mash and Yeah. You know, eat up all those sugars and everything. So and the interesting thing was it, Joe, you pointed out that the ingredients, there's a honey
Rick:barley
Joe:So here's the ingredients. 70% corn, 10% molded wheat, 10% molded barley, and 10% honey malted barley.
Rick:And so the triple malt is 1 wheat 1 wheat, 3 barley.
Joe:Yeah.
Justin:Which is very interesting. I expected, as you mentioned, rye. Typically with the 4 grains, you know, it's, you know, with that wheat, rye, barley, and corn obviously, you know, into that. So this, they take it a little bit different slant, you know, into that.
Rick:It is super smooth, though. It is. Like, when I think about the bite that Sunrise have, this does not have any of that.
Justin:Yeah. It's a good balance. So, yeah, it's not it has that sweetness obviously from the corn and everything, but it has a good balance with the other grains. A lot of it, you'll get, like, really bitter or really, you know, a a bite, you know, into that, or that spice from the rye if they put a lot of spice for, you know, into it. This is very well balanced, with it.
Joe:Well, I tasted it before, I put a chip of ice in it and then afterwards, and I enjoy it much more now this way. So, you know, when we're done, we'll need to, we'll need to try it that way. Yeah. Yeah. So hey.
Joe:Cheers.
Justin:Cheers, guys.
Joe:So anybody from Rabbit Hole, if you're listening, you can send us, yeah, you can send us another bottle, and, we'll feature it again.
Justin:Yeah. Absolutely. Yeah. So little thing happened called b sides.
Joe:B sides Pittsburgh.
Justin:We actually promoted a couple of times on both of the other episodes. We mentioned it, for each of those. Joe, why don't you take this since you're well, you're both part of the organizer. Oh, yeah. You're the main Yeah.
Justin:On to the event.
Joe:I've been doing this since 2011. Yeah. It all started with a group of folks who went out to the, 2010 one in Vegas and came back and said we could do this again. And, you know, kind of the first one we did was 80 people showed up. We had 8 talks, raised $12, and, all you needed was a speaker and some people to listen to them speak, get some booze and some food, and you got a Pittsburgh conference.
Joe:Yeah. And so this one, in contrast,
Justin:that's a really good combination. That
Joe:was a good combination. I mean, it's Pittsburgh. Just Right. You wanna get your friends together, you're gonna need food and booze. Right?
Justin:Yeah. So It's the first one up at left field.
Joe:It would left field main space near
Justin:the PNC park. And you wheeled in the kegs there and
Joe:Tap the kegs around noon. Yep. Yeah. This one. So this event, we had probably the best reviews of ever.
Joe:Yeah. I would say this is the best one. There were almost no complaints, and the complaints were actually just good constructive criticism
Rick:Right.
Joe:Which is unlike any other year.
Rick:Yeah. Ideas for improvement.
Joe:Yeah. Just just good stuff.
Justin:The one was, like, the booths were on each side of that hallway, which Yeah.
Joe:A little bit of, new
Rick:space. Yeah.
Justin:We need more flow for the traffic to get through,
Joe:the people to walk through, and, we're already coming up with ideas for that.
Justin:Which, to be fair, this is a brand like, they redid everything in the casino.
Joe:So They redid a lot of stuff, and we
Rick:had a little
Joe:bit of
Rick:a first or second event there.
Joe:Yeah. In this new space. So,
Justin:so still get familiar with, like, what you can do and what's right, you know, and all that. So Yeah.
Joe:So quick stats. We had a 1,004 tickets sold, had about 800 attendees show up.
Rick:Biggest ever. Be pre COVID numbers. Yeah.
Joe:Yeah. Yeah. Yeah. We had, just, under that, right before COVID, and we're striving to hit the, 1,000 mark. Maybe maybe next year, we'll hit the 1,000 that actually attending.
Rick:That'll be the goal. And sooner, hopefully. Sooner? We'll hit the mark sooner. Oh, right.
Rick:Yeah. Yeah.
Joe:We'll talk about that in a minute. Yeah. So we had over, over 20 sponsors and raised, all this is public. We'll we'll share it. So raised, over a $100,000 to make this thing happen.
Joe:So talk about, throwing a party for your, 1,000 Pittsburgh closest friends, with, about 20 different sponsors, talking about their their wears. We had 2 tracks of talks. Mhmm. So about 14 talks all day long. We had a capture the flag, a lock pick village Yeah.
Joe:A speed mentoring. And this year, we had a Newham Radio Village, which next year, I think, the organizing team for that part of it wants to call it the wireless village. They were Okay. Talking about all things all day.
Justin:Mhmm.
Joe:In fact, in that room, I heard some of the folks, and they did a great job. I heard them talking about it was almost as if they were giving hourly presentations all day long, on on various things from how wireless works, how ham radio works, even car hacking. So, you know, if you didn't get a chance to check that out, you definitely gotta get there next time. The talks, are just about ready to be published. So, everything that was recorded that we had permission to record will show up, on the Pittsburgh, besides Pittsburgh website.
Joe:And if you're not already on it, you should go to pitsec.com, pitsec.com, and subscribe, and that'll allow you to, become part of the Pittsburgh Slack channel for the cybersecurity community.
Justin:So you can the b sides, Pittsburgh Slack channel.
Joe:It is a Slack channel that has a channel for b sides Yeah. But it also has a place for like, if you just Justin, if you just wanted to throw a conference and you wanted to get a channel and
Rick:Mhmm.
Joe:Just promote it and talk about it and get a conversation going, you could just request one.
Rick:Security Pittsburgh Security Community Place. Yeah.
Justin:I know you know. I'm in it.
Rick:Yeah. Yeah. Yeah. Yeah.
Justin:But it's more than just a lot. It is probably one of the the best hidden gems in the, Pittsburgh security community. It is phenomenal. Like, the the the collaboration that's on there. I mean, just hitting it up, you know, type of thing.
Justin:John, tells you, like, you know, because it's an open Slack
Joe:and sometimes Kudos to John Ziola for putting it together a few years ago.
Justin:Exactly. But he says, like, some people do a little bit too private of that. Keep in mind, it is, like, you know, the admins can see
Joe:some of this stuff,
Justin:you know, type of thing. Yeah. Don't worry about that. You know, type of thing. But, like, for the stuff you really don't care about or don't care about John, you know, CN, you know, type of thing, it's great.
Justin:Like, I've reached out to so many contacts from Pittsburgh. Mhmm. You know, to see him on Slack and be like, hey. I got a quick question for you, you know, type of thing. Or there's 3 or 4 of us that jump into it.
Justin:Like, it is phenomenal. Yeah. You know?
Rick:Well and just on the talks really quickly, I also wanted to add, more talks submitted significantly more than ever before.
Joe:Eighty. So I must feel like I need to apologize to people who had a great talk submitted that didn't get it in because there were over 80 talks submitted, and we only had room Yeah. For 2 tracks. Yeah. An hour long talk all day, you really only got, like, 14 to 16 slots for it.
Joe:We tried to avoid talks over lunch, just because we don't want people to have their talks scheduled and not have people show up for it. So here's what we're planning to do.
Justin:Like, a time for lunch, and then they put away the food. Right?
Joe:Oh, yeah.
Rick:Yeah. Exactly.
Justin:Yeah. So
Joe:So a couple couple cool things from this year. One of the things that we did was, lots of drink tickets, were out there. Thank you to sponsors who sponsored this and, also bought extra drink tickets. We said to the casino, what is the earliest legal possible time you can open the bars? 9 o'clock.
Joe:Done. Yeah. And
Rick:Do that.
Joe:That opened, which actually beat our first b sides where I I I don't think I'd let them tap the kegs until at least 11:30. Mhmm. So that was fun.
Justin:Reminds me of, It's Always Sunny in Philadelphia when they're they're riding on the plane. They're like, what's the most beers I can order? Right now she's like, nobody has ever asked that question. He's like, great. So there's no policy.
Justin:Yeah.
Joe:I love it. So so for next year. Yeah. And I have the contract on my desk, and I'm really hoping the casino, holds the date for us. But right now, hold the date.
Joe:I believe it's July 11th. It's a Friday.
Justin:Okay.
Joe:Always we've always done these on Fridays. And right now, we're looking to get the casino. Right? The casino. Yeah.
Joe:Yeah. Right now, we're looking to get the entire casino. So the difference between last year is you'd come up the escalator and you could go to the old conference space, or you can walk all the way down to the new ballroom conference space, and then there was a buffet. What they do is they converted that buffet into nice conference space, and I was impressed with how nice it looked.
Justin:I love how the windows were, like, on the back of the speaker in their voice. That that's what they're looking. They did a great job. Yeah.
Joe:So there's that. So we had those two spaces. And, some things we'll we already have ideas what we can do to anybody who was caught up in too much traffic, walking through the hallways. We're we're working on that. But the other thing we're working on is getting all 3 spaces for next year.
Justin:So the 3 tracks? Three tracks.
Joe:Okay. We'll be able to get another
Justin:7 or
Joe:8 speakers, maybe do some, you know, play around with it, maybe get the, capture the flag back to the cool room it used to be in that everybody loved as well. That was another one that had big windows and were wide open and,
Justin:you
Joe:know, they would play music in the background. It was a lot of fun. So that's, that's what's on the horizon.
Rick:And, Ben, if you had if you had a talk that was a great talk and it didn't make it is just there was just space issues, and please submit again because
Joe:we'll have
Rick:some more slots next year.
Joe:Yeah. And, promoting Tris, Tris is coming up, and they are still doing call for speakers. So if it's still open by the time this, rolls out, you should do that. The, oh, tickets.
Rick:Oh, earlier. Yes.
Joe:People had waited, and, it was almost, like, last minute. So we have to get those t shirts and a lot of the, signage and a lot of the food orders in almost 30 days before the event. Right. So that's, like, early June. And we were early June about 500 tickets
Rick:sold. 600.
Joe:Yeah. Of the 1,000. And it was only in the last couple weeks that that happened. So watch out for this. One of the things we're gonna do is have an increasing scale of price Mhmm.
Joe:As we get closer to the event.
Justin:Front end that
Joe:because we wanna get as many people signed up legit who are gonna show up early. Now don't buy a ticket to hold it and then don't show up because nobody likes no shows.
Justin:Right.
Rick:But we
Joe:know what happens. The other thing we're gonna do is open up
Justin:sponsorships, a
Joe:lot earlier as well. And I believe we're gonna have some discounts versus sponsors
Rick:Yeah.
Justin:Early bird.
Joe:If they, are able to get their funds to us. Oh, okay. You know, this calendar year, probably by October ish or so. And if we have that money, you'll be able to take advantage of, a great sponsor slot and at a discounted price.
Rick:Yeah. And we'll be able to do a bunch of cool stuff with it because we know earlier it's there.
Justin:Yep. Awesome. Cool. Next topic?
Rick:Let's do it. What are we talking about?
Justin:So one of the things I thought was cool in the news, know before. You know? Make sure out No. At first. No.
Justin:What? Is this really Justin? What? Because it's really you. Right?
Justin:No. I'm just kidding. Sorry about that. Yeah. So, yeah, they hired, essentially a an employee, did the interview, did the background checks, all that stuff and everything.
Justin:Mhmm. And surprise, surprise, came out to be a North Korean spy. So it was really interesting. I actually thought for a second, I was like, why would they want to be in KnowBe 4? You know, like, is that really a hot level target?
Justin:You know? I mean, they're in a lot of businesses. Yep. But there other than some employee information and not even PII, typically, it's, you know, email and name. What did you know, what are you gonna get out of that?
Justin:You know? I can see you Could be a launching platform maybe because everybody has to go there. So if they were able to get malware to
Rick:the portal Doing delivering bad stuff through phishing tests would actually be pretty nasty. Uh-huh. That would be pretty rough. Maybe that's the maybe that's the integrity attack. It's a human integrity attack.
Justin:Everybody white lists, white lists know before. Right. Email is going through.
Rick:It really could be. Yeah. It really could be something like that.
Joe:So in a person interviewed, did they do a video interview? And were they using, like, some sort of AI in order to mask their appearance?
Justin:No. I think they were doing a video. I'd have to double check, but I think they were doing a video. But then when they present an ID, the ID was fake, but it looked enough like the person.
Rick:Close enough way through.
Justin:Yeah. So it was enough that it went through, looking at that is what my when I read it, what I understood. But I thought it was interesting. You read, basically, the CEO came out, and I thought that was good, what you pointed out. I'll let you point out.
Joe:Yeah. Set me up, and that's perfect. So, yeah, the CEO wrote a blog, released it on 7, or did an FAQ, and then wrote a follow-up blog. The FAQ came out on July 25th, and the blog came out on 27th. And, basically, people were saying, well, you were breached.
Joe:And he's like, no. We weren't breached, wasn't a data breach. It was in fact, they didn't even need to bring this out into the open.
Justin:Mhmm.
Joe:And so as I was reading this, I was thinking, man, kinda really sucks to be know before right now. And then after I was reading it, I'm like, wow. That's a lot of good integrity that he's showing. Yeah. And it's because they he he said he wanted to explain what this what happened.
Joe:Because if it could happen to them with all of their security controls and all their processes being a security company, just think of all the companies this could easily happen to.
Rick:Absolutely.
Joe:And what do we know right now? Do we know how many companies this has happened to? We'll never have an idea. It's probably happened. Right?
Justin:Oh, I honestly, the brilliance I don't know the CEO, but the brilliance of like, this is free publicity.
Joe:Right. And this is
Justin:quite like Yeah. Wow. Yeah. You know?
Rick:You're already in the news. You own the
Justin:You want to talk about, like, a else this thing coming out. You know? Like Right.
Joe:Yeah. He said egg on his face, and, yep, they had egg on their face. But here's what they, did. What were you gonna say?
Rick:Well, I was gonna say, the to the point about, like, how many companies has this happened to, is this happening to? I know both in my prior experience and my wife's prior experience, there are it's actually similar things. Definitely not the same thing. But similar things are not uncommon in certain hiring practices. And so, you know, she had a couple experiences where the developer that interviewed, who happened to be having, you know, video issues at the time and all sorts of things, and got hired, right, through the process was not the developer, clearly based on skill sets and, you know, a a difference in vocal tones and accent, things like that, that that that interviewed.
Rick:And so I think operationally, it happens and can happen and and it hap but, you know, from a nation state perspective Mhmm. That's pretty terrifying because, you know, the the low tech things, they're gonna find someone out, right, when they don't have the skill stuff like that. Like, well, the nation state doesn't necessarily need to suffer from those. Right? The long con, you can you can you can work a while at making someone look like they should fit in and have the skill set and all that stuff.
Rick:So it's pretty wild.
Justin:And I had somebody so I just recently I actually have a developer starting tomorrow. Mhmm. But going through the interview process, there was one interviewee that I had I could definitely tell he was reading from some AI generated Oh, yeah. Answers. Like, I asked him about this question.
Rick:Your answer has 3 distinct sections.
Justin:He was doing very simple. Like, I was hiring for a senior position for this, and he was like, yes. And you can turn the widget green or blue. And I'm like, what? Like
Rick:Why is that irrelevant detail? What do you mean?
Justin:And you can use GitHub for a source code repository. It's like,
Joe:did it start out with, in this ever changing environment, because every AI generated are the
Rick:real parts of that.
Justin:But, yeah, it was so like 101 that you could tell, like, it was something AI generated. You know? So is
Joe:that the guy starting tomorrow?
Justin:Yeah. No. He didn't get the job.
Rick:I I I know I mentioned this to you before. I don't know if I mentioned on the podcast, but I have a buddy, who, they started calling the their junior security person, Chat GPT, behind his back because it became extremely apparent that, you know, that that's where he was sourcing all of his information. And, and so if they had a security question, they'd be like, you just ask chat gpt and they're like, wait, the person or the the site? But, yeah, it's a it's an interesting take on the same issue in terms of the the in the source of the information is not the perceived source of the information. Yeah.
Rick:Right?
Justin:Yeah. Yeah.
Joe:So some some good takeaways, and I respect what no before called out here. And I like some of the controls that they put on in their write up. So if you get a chance, go find the know before, blog, take a read on it. But just some, highlights here. I took some notes on that they do.
Joe:So they were they shipped out a PC, and it went somewhere and somebody got it. But and then and the way it was being used set off some alarms, and that's how they kinda discovered stuff. That year,
Justin:it was remoting in from North Korea or no?
Joe:Yeah. I I what I heard was they took it and put it in some place, in the US, I believe
Rick:Right.
Joe:That has, like, almost, like, a farm for putting machines like this. Mhmm. And then somebody's got a hands on stuff so they could remote remote to it and get access. Got it. I'm not sure.
Joe:I didn't get the details how that works. But what Novi 4 did was they didn't put any sensitive information on the machine, and it's almost as if a new machine that's clean was sent with no data. Yeah. And the only thing they could do
Justin:process from a company in public.
Joe:I've seen it in multiple ways. I saw, you know, you get apps loaded, you get things loaded, in advance that gives you access into further than maybe you did. Yeah.
Rick:If, like, for me drop shipped depending on the service you buy and stuff like that.
Justin:Gotcha. Yeah. And then The data isn't technically loaded on the PC. Yeah. No.
Justin:But access to service is loaded. So if they drop, like, full wiki access to all of your stuff, that's Right.
Rick:Or even, like, hey. You're you're you're, you know, your privileged groups associate you with the finance department. Okay. Here's access to the finance department share drive. And by the way, here's the, you know, here's the link to that right out of
Justin:the gate.
Joe:So that brings up the second thing they did a good job at is they limited the access only to something somebody needs in order to go continue going through onboarding. They had no other privs.
Justin:Oh, really? Yeah.
Joe:So they were locked down
Justin:from no other privs Good.
Joe:On the access control. So when they log in, they still had very rudimentary access.
Rick:I have you said that's unique. I've seen similar things before, but they're often tied to, like, hey. You're not allowed to access x until you get training on y.
Justin:Mhmm.
Rick:And then there's a compliance reason for that. But this is actually an example of, like, that working well for all the right reasons.
Joe:Oh, yeah. And then the third thing they did, or will do in the future, and they're fixing their own process, they said, is for new employee workstations, they're gonna, drop them to a UPS shop that requires a photo ID as you come to pick the machine up. So it's going to help them as another control that's going to help validate that the person who's picking it up has an ID that matches the name of the person who's supposed to pick it up.
Rick:I do think that's good help.
Justin:I well and they got fooled by the ID already. So I guess where where that actual physical ID is.
Rick:UPS, you look at this instead of me. Yeah. So I think it's good. I don't think it's bad at all. But I I I retail retail level identity checks, I'm certainly not suggesting it's bad.
Rick:I don't know that it's foolproof.
Joe:So
Rick:but from a defense in-depth perspective,
Justin:it's very natural. Validate the ID on the onboarding process, you know. So it's really it's good. You know?
Joe:They validated the ID against the the video they had seen Right. Which may or may not match. But when you walk up in real life and you have an ID, that's a little bit harder to
Rick:I don't know, man. I used to have long hair in a pony tail, and I don't anymore. I
Justin:have facial hair.
Rick:Yeah. But but anyway, not foolproof.
Justin:And it's lower level employee not to say UPS. People are lower level employees, but it's somebody working a counter, you know, checking ID. So Mhmm. Would they check it? If it went through one place, would they catch it in another?
Joe:I'd be very interested in, if UPS has a, an actual formal process for that that they,
Rick:I mentioned before retail ID checking, but actually now that I think about now that I think about a little more who UPS is and the fact that they get tons of parcels each and every day, many of them would require authentic, like, ID authenticated stuff. Because it's not just laptops. Right? They Yeah. It's pretty bad.
Joe:I don't know. Well, the difference here is that I
Justin:get a lot of alcohol to my house, and the UPS driver's like, can I just say I dropped it off? I'm like, yeah.
Rick:But that's different than the retail location, though. Right? But that's different than, like, having to go to the retail location.
Justin:Oh, yeah. But he drops him at my house. He's like, can I just Yeah?
Joe:And if you're the UPS driver's boss watching this, it never happened. Anyway, the, you're
Justin:never you're never getting anything again,
Rick:man. It's over.
Joe:Yeah. You better you better enjoy that.
Justin:So I live in Wisconsin. It's Yeah.
Joe:I use FedEx for all
Justin:of that.
Rick:Don't even get us started on these unsecured, lights. Yeah. Right. But but
Joe:the main thing is is that instead of just shipping it to somebody's house and having the UPS guy leave a laptop on their front porch, this is a little bit harder.
Rick:And and I do think it is because I I think the the delivery driving thing has become such a commoditized service. That's a lot more of a gig economy thing. Whereas the retail worker in the store whose job it is to authenticate people and give them their stuff, I I could see there being a different and better process at the retailer. I'm not saying it's foolproof.
Justin:There's a lot of peer pressure, like, are those employees prepared to say no, you know, type of thing? Like, typically, they're junior employees. Like, it has to be very clear. If it's questionable, they're gonna let it pass.
Rick:I think I'm convincing myself that it I don't know. It's likely better than I initially thought it was. I'm certainly not saying it's foolproof. Yeah. Yeah.
Rick:But yeah. But I did But I
Justin:think it's control that can't hurt. Like, if you're a legit person, it's gonna fly through. Like, you have the ID. Everything is gonna go through.
Joe:If you work at UPS and you have details on these controls
Rick:Yell at Justin.
Joe:No. Post post post it.
Justin:Comment. I want to invite them to my house so that I don't know where I live.
Joe:No. Not to your house.
Justin:They already do.
Rick:They do it right now. No. But
Joe:comment on the, comment on the, on the podcast.
Justin:Yeah. Let us know.
Rick:Absolutely. I would like it'd be good to hear. But I do think those are great those are good good added controls to try and stem similar things from happening in the future. Yeah. And it sounds like some things went went really right.
Rick:I mean, obviously, they they found the thing before it got worse.
Joe:Right. And one of the things that they detected was that the the the guy was trying to load, Infostealer software on that endpoint in order to see what they could pull. And, there there was nothing there to pull, so it didn't go anywhere, but it set off alarms. So big takeaways.
Justin:That's probably the biggest thing. Yeah. Because, you know what, I'm thinking about this. Like, let's just, eliminate that. You know, it was kind of a laptop sent sent to a farm and all that stuff and everything.
Justin:Yeah. What if the person was just a, you know, a spy trying to infiltrate a company but acted as his own agent? You know? Like, he has the ID. He has all that.
Justin:If he gets past onboarding Mhmm.
Rick:This is a burner identity.
Justin:Or he has his real ID or whatever it is.
Rick:It's it's real for the
Justin:nation there. Exactly. Yeah. So it matches everything, and you hire the person, you know, who's able to fake his way through the onboarding, you know, or or the interview process. Yeah.
Justin:You know, that's gonna be a lot harder. I think the detection on monitoring what they're doing. And and I love the idea And
Joe:any insider threat, really, right,
Justin:profile. Yeah. Exactly. But, but, yeah, I think the and also I really like the, limiting the access until they're kinda slowly onboarded. Right.
Justin:You know, because once you start interacting with them, you know, usually people wanna get in and out, you know, as fast as they can. They're not Oh, I have
Rick:to take all these trainings before I can steal
Justin:all the data? Yeah. Yeah. It's like so, I mean, you know, most criminals are, like, I wanna be done with this, you know, type of thing. So the fastest way in and out, you know, with it.
Rick:Yeah. I I agree with that in general. If if this was truly, like, a sent by a nation state, they're probably gonna have a little more patience than your typical. But, you know, I I think, you know, all all the controls are good, and it's just like, you know, it's anything else. Like we, you know, we go to b sides and we go to the lockpick village.
Rick:Right? And we we learn pretty quickly, like, well, most locks are great, but they kinda keep the honest people honest
Joe:Mhmm.
Rick:Oftentimes. And so you set up layers of of physical security. And locks are one of those layers, or it's multiple types of locks. And and because you you you delay, you frustrate, and you set up additional instances where they're either gonna get frustrated and give up,
Joe:is to deter all the uninterested Yep. To take the, person who's breaking in and make it as hard as possible. So it takes as long as possible. Meanwhile, what should be happening at the same time is some cameras picking that up. Yeah.
Joe:Motion's going off. Somebody's monitoring that. And the difficulty to pick the lock Mhmm. Needs to be aligned with the amount of time it takes for your local police to actually be able to get there once you call them. Right.
Joe:And then now you can have a response process, before they get to the next layer. And that's the same thing
Rick:for your network. Exactly.
Justin:We use a so, you know, I use work at Diebold. One of the tests we had, we used to run a whole bunch of different tests including, certain explosive is how fast you can get into the cash box in the ATM Mhmm. With blowtorches, with, you know, all the stuff and everything. Like, how fast can you get at the cash, you know, was the ultimate goal, with this. And we it was essentially minutes in every single test.
Justin:But those minutes, hopefully, you have alarms already going off and police responding, you know, into that. And that's what's the whole goal was to and usually, there's limited cash. Like, the most cash boxes are casinos, and they have, like, 30, 40, 50 k in that cash. But most are like 5 k, 3 k, 2 k, you know, whatever it is, you know.
Rick:But I also love the analogy because I think from a digital perspective, there are all these defenders and blue teamers and stuff like that. And and the same thing applies, like, how fast can I get to the cache? Well, how much c 4 do I have? Yeah. Okay.
Rick:But I'm probably gonna get noticed. So the really really fast mechanisms right in. Yeah. Loud, noisy, gonna get detected. Maybe if you have a great escape plane, you'll walk away with whatever's in there.
Rick:You might even destroy some stuff in the process. I mean, the the digital equivalent exists. Right? Like, there are some attacks that are really noisy. You're gonna get detected, but it might be faster.
Justin:There's one there's one attack that was going around, and never really caught on. But if you poured gas down the, like, the front of the ATM and it filled up so much, you know, with that, too much and explode the whole thing. Too little and it didn't, open up the safe, but the right amount would actually blow up in the safe and everything. So, yeah, it was interesting seeing some of the cases.
Joe:There you go. There's your tips. So if we start
Rick:seeing that, I was just gonna say, banks and UPS hate this pun. Yeah. Right?
Joe:So but we we got off, track from the, the hiring process. So the things to think about, and we need a whiteboard here for next time. Process flow, you're hiring, you know, go go through and document your hiring process. Whiteboard it out and look at each step and see at what point you give people enough access, and by then, would you have time to have detected this problem yourself?
Rick:Yeah. Well, I think the other thing that's interesting is even if it's not insider threat, right, like, compromise credentials is a thing, right, bad people using legitimate access. Right? So even if it wasn't like, hey, this was a different person than who you thought it was, maybe it's just a different person using a legitimate account that has gone through the training and has been an employee for years and all that stuff, but now it's compromised. And I think a lot of the same controls apply in terms of the defense in-depth and are they doing something weird, and is there least privilege, and why are they doing this thing today that's out of normal behavior?
Rick:And we should talk about that and look at it, think about it a little bit.
Joe:So Makes sense.
Justin:Yeah. Alright. So we got just a little bit more time for one more topic here. Yeah.
Rick:What are we talking about?
Justin:So one of the articles, that came out, and actually, do you have it in front of you? The article? Why don't you cover this one here? I don't have it right.
Joe:Real high. It's CISOs scapegoats. And the article's point was, how are CISOs being, potentially the scapegoat of these various problems? So you have the new SEC cyber risk disclosure rule. Yep.
Joe:You have various rulings like the latest with, oh, we mentioned it earlier.
Justin:And we'll have all these links, we talked what we talked about today in the show notes. So definitely check it out. You can read it for yourself here. But what I thought I mean, so we talked a little bit before the podcast, and I thought it was interesting. I mean, CISOs are often looked at if the security is not good, it's your fault.
Justin:You know, and oftentimes they get blamed for if you have a breach, what was your CISO doing? You know? And it it happens all the time. Even with the the, the change, you know, the Change Healthcare United. Yeah.
Justin:I was hearing in congress. They were basically blaming the CISO, like, you didn't have a security background and all that stuff and everything. And I don't know the CISO from a hole in the wall, but, you know, they go into the CISO, you know, wasn't qualified, you know, with that. No matter, you know, what what the issues or the facts or anything like that, that wasn't apparent and or clear at that time. But you know, some senator congressman was going on like, hey, CISO, like why weren't things bad?
Justin:And it's it's interesting to me because as a CISO, as if you're following the traditional 3 layers of oversight, you shouldn't have much operational in charge. You're like, you're there as an oversight facility. You're seeing if the first level controls are operating effectively or not, you know, type of thing. Unless there's a failure reporting, like what if you said everything is crap right now? You know, like, you need to fix all this stuff.
Justin:How is that their fault?
Joe:Yeah. No. It's and one of the things that, came out of this article is the dismissal of the SCC charges against SolarWinds. Right. Came out of it.
Joe:Mhmm. But I don't think that they really said
Justin:in the lawsuit.
Joe:Yeah. This is crazy to me. And it and it's and it's unclear to me whether or
Justin:not he is totally out of the water with this, but They cleared some charges onto it. Yeah. With that. There are some that were dismissed, you know, as without standing.
Rick:It's interesting. There's not a lot of precedent yet. So I think, like, lawyer like, all all the the case law is kinda feeling systems feeling its way through some of these things. But it just it seems to me that all you were talking about the the 3 layers of defense. Yeah.
Rick:You know, the Department of Transportation makes the rules of the road. Are they responsible when someone doesn't follow the rules of the road? Right? It's not terribly different than that. Right.
Rick:There are people that make the rules, people that enforce the rules, people that follow and or break the rules, and I think, I don't know, I I just keep coming back to the concept of, like, adversarial attribution is hard, right? Mhmm. Well, attribution of, like, who did the bad thing is also Right. Difficult or murky sometimes, right? Because it absolutely could be a case where, you know, a house falls down.
Rick:Why did it fall down? Well, the architect designed it poorly. Maybe. Right? Or the builder built it poorly.
Rick:Maybe. Or, well, you ran a bulldozer into the wall that's like a structural supporting thing. Maybe.
Justin:And it was crazy, like, was it Experian or Expediant that had that, big breach and everything. Yeah. And I remember the CEO get on, and he's like, it was a Struts vulnerability, you know, one of their, like, you know, exposed websites, and they're like, oh, yeah. The admin didn't patch it. The the CEO get up there, and it's like, the admin didn't patch it.
Justin:It's like, the admin didn't patch it. That's your But you know the point there,
Joe:that was the problem is it was known, it was on a list, and it was deprioritized. Mhmm.
Rick:And it was Someone made that decision?
Joe:Someone made a decision.
Rick:Right. And and and how they made the decision matters, right, in so many ways. Because there's a lot bad stuff happens all the time that, like, you can't account. Again, we talked about the act of God in terms, like, bad stuff happening earlier. Like, sometimes bad stuff happens and everyone did all the reasonable precautions.
Rick:Sometimes bad stuff happens because someone was clearly negligent. And drawing the line between those things, I mean, it's it's definitely a combo of art and science. Mhmm. But I think all this is wrapped up in where are those lines drawn and your point about the 3 layers as well. Like, how do you attribute where why the house fell down?
Rick:Right? And when that happens, well, with the CISO, there's a chief title there. Right? That's typically a decent layer of authority. Saying the admin didn't patch it, That might be true, but okay, was the admin personally account accountable for this thing?
Rick:Or do you need to make the CISO or some oversight function regardless? CEO, whoever it is, risk management, whatever, responsible for evaluating and addressing it. Right? Because it's weird to me to hear, oh, I blame the admin. Right, wrong, or indifferent.
Rick:Because, like, that's probably not an executive level
Justin:Well, the decision is not deciding what's patching and what's not. You know, like, typically, that's not authority. But they can fail a process that's been defined. They can. Absolutely.
Justin:They don't they don't decide whether or not it should be patched. However, there should be, again, the layer of security there, there should be oversight. So detecting whether that patch has made it or not is absolutely in securities oversight. Right. You know, type of thing.
Rick:So are we saying that the CISO scapegoat thing is overblown? Yeah. Absolutely. Yeah. Alright.
Rick:I agree.
Justin:Yeah. And honestly, I mean, I'm of the kind of premise, everything boils up to the CEO, you know, at that point. Like, at any level that there's a failure, it comes down to whatever it is, a prioritization and oversight, and it comes down to he's the chief executive officer. So even if it's a fail of the c CEO or CCO at that point, then maybe he didn't have enough interaction to bubble up that stuff. Maybe the process was failing to get the the appropriate resources there.
Justin:Like, there's a failure somewhere, but at the end of the day, it comes to the top, you know, on they didn't execute appropriately, you know, again, you just have to own it. Maybe you didn't do anything wrong, you know, at that.
Rick:Yeah. Sometimes bad stuff happens. Yeah. Exactly. Sometimes you were a cause of bad stuff.
Justin:You let somebody take too much ownership that didn't have a good judgment. That's what I'm gonna say.
Rick:Like, trust flows down from the top. Right.
Joe:So getting into like a slightly different perspective. Yeah. What does this mean? What I'm seeing in the industry is lots of CISOs who aren't in their job anymore and aren't actually interested in going out and getting another CISO job. They are liking the idea of a v CISO job working in that perspective because it's a different layer of accountability.
Joe:They don't have that, main layer. So I'm seeing that. And what's happening now is that when companies are hiring the CISOs, they may not be getting the best of the selection because The
Rick:people that have been through it are no longer
Joe:gonna be through it. They're no longer interested in wanting to do
Justin:it. Right.
Joe:And so one one of the things I do when I jump on a call with a director of security or a CISO, and I think I might have mentioned this before, on another episode is I asked a question. Let's start here. What keeps you from getting fired? And let's reverse engineer through that.
Rick:How do
Joe:we create an assurance program where you're in a position to demonstrate that you have defensibility of your actions, that you're pointing out when there's a nonconformity of a process, that you have a corrective action process that really goes 5, you know, do the do the 5 whys of a root, cause analysis Mhmm. And figure out what the problem was. And if you stop with it being a person's fault, you're not going far enough. In fact, that's a tenant of an ISO 27,001 implementation. You can't blame a person for nonconformity.
Joe:They might have a part in it. And sure, maybe they're not doing their job well and you have to deal with that, but you need to go all the way down and look at the processes Yeah. And see what failed in the process, in that process, mechanism.
Justin:And to that, you have corrective action to, you know, address the problem. If it is a person or a process, you know, into that.
Rick:I don't think I heard that. I don't think I had heard that before, but I love the question, like, what keeps you from getting fired? Because it's such a great way of
Justin:See how you disagree with it. Oh, I love it. I took issue. Oh, see.
Rick:But here here's why I'll start with why I
Joe:love it.
Rick:Okay. I love it because it comes at a risk tolerance question from a sideways point of view. Like, because you can ask people like, what's the organizational risk tolerance? And they'll be like, oh, maybe they kinda know, maybe they but oftentimes they're like, I don't even know what we're talking about. And even if they're, like, steeped in that, they're like, well, what do you mean by it?
Rick:I'll tell you what I mean. Uh-huh. It's this whole thing. But if you're like, well, what keeps you from getting fired? It's like, okay, well, what would your boss be super pissed about?
Joe:Right. What would cause that problem?
Rick:Yeah. That's risk tolerance. Have you
Joe:read the book Extreme Ownership?
Rick:Yeah. Yeah.
Joe:Yeah. And extreme ownership is everything that you could possibly do to make sure this isn't a problem. Yeah. So if somebody gets on and I would ask them that question and then ask them your question, which is, if I said first, what's what gets you fired? And instead, I would lead into the point of, well, what's your risk tolerance level?
Joe:What are your thresholds? And if they don't know, I'd say, well, the first thing you need to do is take extreme ownership of that.
Justin:Yeah. You need
Joe:to understand what that is. You need to understand who's gonna hold you accountable for what. They're probably already holding you accountable. You just haven't had that conversation yet.
Justin:Right.
Joe:Have that conversation.
Rick:It's
Justin:all about expectation. Yeah.
Rick:Yeah. And then and have the trust flow. So that's why I love that though because it approaches that that question in a very direct, straightforward tactical way that's gonna get you all the answers you need to get to the other places. So anyway, that's why I like this. So you But you didn't like it.
Joe:Didn't like the way I framed that. Tell me why.
Justin:Yeah. So I look at it as, the getting fired as the low bar of expectations. You know? So you're attacking at what I need to deliver and here's the low bar. So you don't have to
Rick:say, like, what's gonna get you promoted?
Justin:Yeah. I want to say, like, what's gonna add the most value, You know, and that You
Rick:know, maybe that was his next question. Yeah.
Joe:Oh, you gotta I I always start with, how do we make sure that the worst case scenario for you, the person I'm talking to right now
Justin:And that could be Yeah. Could be addressed
Joe:because I think and it's a starting point.
Justin:Yeah. When I think of the fired, like, I think of office space as, like, you know, you just have to work, you know, just enough not to get fired. Do you
Rick:really wanna be the person that only wears 4 pieces of flare?
Joe:Yeah. No.
Justin:Maybe I
Joe:need to reframe that. That's that's good feedback. I'll reframe that question a little bit. And it's what is your boss expecting of you?
Rick:I like that.
Joe:But it's it's easier to it's more fun to just start with
Justin:when you're fired. And I get it. And, you know, we're probably more aligned than, different than that.
Rick:Oh, yeah.
Justin:But I hear, like, what's gonna get you fired? That's the that's the bare men, you know, type of thing. And so a lot of employees, you know, not good employees will say, what's gonna get me fired? And I'll work right here. Get out of the bag.
Rick:That's fair. Yeah. My my not exactly the same, but my kind of version of that, like, sideways question that I'll ask in a lot of risk assessment stuff is like, what's your worst Friday afternoon ever look like?
Joe:Right.
Rick:Like, what's the absolute worst stuff that could happen? Right. I have to touch every
Justin:single endpoint before door over the weekend. I I had a call with a client at 9:30 in the morning on Friday, and one of the guys from the security office, he's like, I've been on the call since midnight last night, you know, type of day. I'm just like, then why are you on the phone with me? We're talking PCI. Like, this isn't a priority right now.
Justin:And you dropped it off. I was like,
Rick:if if I had to pull a silver lining out of that event though, it would be testing the responsiveness of like, a live fire test of how aware, responsive, and, and coordinated a lot of these response teams are. Because there are a ton of unsung heroes that made a lot of stuff happen really, really quickly.
Joe:Oh, absolutely.
Rick:And and I think it really those events, you never want them to happen, but, you know, it's like never let a good crisis go to waste. Right. And and I think part of that for this one is recognizing the people that really knocked it out of the park to get stuff back in order versus the people that, like, maybe didn't jump on the horse quite as quickly as they could. Right.
Joe:Hey, I poured you a drink on I
Justin:appreciate it.
Joe:For a reason here, and it's because I think we needed to cheers to those IT people who went well above and beyond.
Justin:Yeah. Yeah.
Joe:And, it looks like we came full circle with this conversation. We started
Justin:with CrowdStrike,
Joe:and we're ending with it. Yeah.
Justin:Hey, here's to everybody. Lead us out here?
Joe:Yeah. Here's to everybody that did a superhero effort over the last week in order to get your companies back up and running. So cheers to you.
Rick:Whether people know it or not.
Justin:To everybody. So I think this is a wrap. We had a good episode here.
Joe:Loved it.
Justin:So That's it. Alright. Thank you everybody for joining us for episode 3. Be sure to like, comment, subscribe, get the word out, you know, of this, podcast here. We're gonna potentially have a surprise for episode 4.
Justin:I'm not gonna mention it because we don't have anything planned right now, but, we're talking about it. So stay tuned for that. It will be a good one, and, please join us again. Thanks all.
Rick:Bye bye. Bye.
