Episode 18: TRISS Highlights, Cloud Chaos & SaaS Lessons Learned
Welcome to Distilled Security Podcast. My name is Justin Lemplein. I'm here with Joe Nguyen and Rick Yocum, and this is episode 18. To start off with, we wanna tell you a little bit about a conference we just went to. We just went to Tris, which was yesterday.
Justin:Ton of fun. We had a booth that we did a little trivia game, and we had a there was a lot of good contestants there. I thought we had a whole range of people from younger people that I was actually really surprised. Some of them actually got, like, some of the answers. Oh, yeah.
Justin:Like, what port is default, you know, for DNS and all that stuff and everything to more advanced people, you know, that, you know, could rattle it off or not rattle it off depending on, you know, the person. Yeah. It fun. But yeah. And we had a a good time.
Justin:What was something that stood out to you, Rick, on the pocket or the conference. The conference itself?
Joe:Yeah.
Rick:I you know, I feel like it's such a cop out, but I just really love the the community events and just getting to connect with everyone. Some some people whom I only see, like, twice a year. So I I think for me, that's that's the thing. And and frankly, we always have fun at the after parties.
Justin:Yeah. I wish they would've opened up alcohol a little bit sooner. Oh, yeah. I know. 04:30.
Justin:But yeah.
Joe:Yeah. They don't they don't do that, but that's okay. And but it didn't stop us from being the last ones there.
Justin:I mean, that's our MO. We got it.
Rick:I feel like we close every event.
Joe:Every event.
Justin:Yeah. No matter if there's young people or not there, you know, like, we're looking over there. They're, like, exiting out, and we're like, well, it's it's only 02:00. For the audience, we don't stay up that late. You know?
Justin:It all the time. Yeah. Just just for pizza. Yeah. Yeah.
Justin:Yeah.
Joe:Right. So but yeah. I think, yeah,
Justin:it it went off with a hitch. I really liked, like, some of their formats. They have a lot of panels. Oh, yeah. And I really appreciate like, the talks are great, but they kinda know what they're talking about, you know, through the presentations and everything.
Justin:But the panels, you know, they get questions ahead of time, but it's I feel like it's more free form.
Rick:That's kind of this format.
Justin:Yeah. Exactly. You know, when they're discussing back and forth and somebody brings up something and somebody responds to it, I feel like it's more organic.
Rick:You know? Conversation.
Justin:I I like that kinda format and everything, you know, with it.
Joe:So Yeah. Did you have a panel? Did you make a panel? Did you get to enjoy a
Justin:little I in on some, but I I've only did like minutes at a time. So unfortunately, had never sat through one thing, you know. Yeah. So it was always like five minutes here, five minutes there, and Yeah.
Joe:I got a chance to sneak into two things. One was the Scott Davis did a talk about how to speak like a CISO. And I thought that was interesting. And I think it was less about how to speak like a CISO, but how to better improve your skills for speaking and how to better engage with audiences. He could've gave this really anything and quickly adjusted it.
Joe:Yeah. But being a person who coaches people professionally on being better speakers, he can probably really easily pivot his talks like that. So he hit the it everything he said made sense, and it nothing was like, wow, I never thought of this before, but you brought it all together in a very nice way. And I know that you interviewed him in advance Yeah. Before that, so it it mirrored that.
Joe:But what he really got into was the rule of three formula where develop your supporting points, make three of them. He you know, two's not enough and four's too many to remember. Right. And it made sense. And then how to connect with your audience was a big thing he covered, and then speaking like a story.
Joe:And I used to be I'm not the best presenter, but I am so much better than I used to be. Because I would take a talk and I'd try to, like, take somebody else's material Yeah. And I'd try to memorize things that just wasn't in my way of thinking. And soon as I flipped it around to become interactive with the audience and start to do those kinds of things and to actually tell the story Right. In a way that I want to, and then you add in some of the tips you get from a guy like this guy where keep it in sets of three, make it really easy, and kinda just go through it.
Joe:And so if you're look not to plug anybody on purpose here, but he actually does this for a living. So he you can find him on the Tris website and you could go find out some information. So if you need help getting better at speaking, public speaking or giving presentations, go check him out. I thought it was good. Yeah.
Joe:The other thing I went to was a panel and it was the I it was called breaking the glass ceiling, and it was the one about how you focused on women. Was that lysis? Yeah. Put that on? Yeah.
Joe:And and that was good. And as I'm sitting there, I'm thinking, what are the things that limit women? And I'm hearing all these things. And as much as I do everything I can to make sure that I'm not, like, intentionally doing this Right. Like, you just start thinking about all the unintentional things you might do because you're just we're we're not women.
Joe:So we just don't have the same mindset. We're not in the same challenges. Right. And it was very eye opening. Even though I've sat in those kind of panels before, every time I'm like, you know what?
Joe:I can I'm I'm gonna I'm gonna try harder. I'm not even trying harder, but I'm gonna try harder. I thought it was very interesting for that. So, you know, that was just good.
Justin:I love that. Interesting.
Rick:Yeah. Okay. On the storytelling thing, it just put me in mind. I I was super lucky to get to go to a kind of a neat different conference like a month ago. And essentially, like, Ed Norton ended up being a keynote speaker.
Rick:But a big thing and then he has some, like, board presentation software with AI and stuff like that, so we're talking about that. But a big thing that he was talking about was exactly what you were just saying in terms of how important storytelling is and how all the executives that he sees at all the board presentations and things like that. It's basically two core things. It's storytelling and it's empathy. And so I think just complete nail on the head in terms of, you know, storytelling being important.
Joe:Oh, yeah. And they hit on EQ a little bit. Somebody asked in the audience, asked a question about EQ or emotional intelligence. Yeah. And so that that resonates.
Joe:But What is that? Emotional intelligence? Well, Rick I guess it's my wife would say, yeah, I she think. But Rick, like like Ed Norton, like as in the incredible Hawk Ed Norton as in Fight Club Ed Norton?
Rick:Yeah. Yeah. He actually cow. He he walked on stage and he, like, was dressed in, like, civilian clothes or whatever. But he walked it was a a conference on AI stuff, and so he walked on stage and went, oh, and he looked confusing.
Rick:He goes, oh, is this AI Fight Club? So it was just like a funny Oh, that is to to get started. But nice.
Justin:Hey. You bring up Hulk as the first movie you cite from him?
Joe:Well, no. It's I can't remember. There's so many other better. I know, but I I just saw a meme the other day, and it's like who played it best, and it showed each of the Hawks and each of the people. Yeah.
Joe:And so it was stuck in my head.
Rick:Oh, that's cool.
Joe:Too much he at? To he they they asked you to rate him. They just had him in a chronological order. Yeah.
Rick:That's cool.
Joe:Yeah. So I didn't get my rating. I don't have one.
Rick:Well well, one housekeeping thing. So just if anyone is interested in some of the takeaways from that conference, I'm gonna be, like, posting them out on LinkedIn. So if if you're interested in
Joe:So you
Justin:went to a con. I went to a con about them on the go.
Joe:I saw a lot things. Yeah.
Justin:Super Not Tris. Yeah. The Correct. I love
Rick:I love Tris. This was just before Yeah.
Joe:Yeah. Yeah. Another con that you went to, and it was sort of an elite con. Right? It had, like, some of the top people from the some of the top places.
Joe:Is
Rick:that what
Justin:I'm understanding?
Rick:Yeah. It was, like, a 100 c CIOs from some of the world's biggest companies and then like 20 to 30 CSOs. And it was all to talk about AI and where things are going and and all of that. And it really, I think, refreshed my perspectives and and changed my perspectives on some things. And so I'm gonna try and make sure I'm sharing out some So if
Joe:I wanna see some of that, just keep following. So everybody follow Rick on LinkedIn if you wanna get some of these tips. Because he was getting them from Yeah. Well I
Rick:mean, like, just a ton of people. Like, well, Ed Norton and, like, the CEO of IBM.
Justin:Can you say the, like, conference or you wanna No.
Rick:I'm happy to. Mean, it was it was it was it was Palo Alto's NextGen Summit.
Justin:Okay.
Rick:So it's pretty cool. And I think you can see the agenda online, but there are some really cool speakers.
Justin:That's not open to the public one.
Rick:I think it's by invite InviteO. Essentially. Yeah. Gotcha. Yeah.
Rick:But anyway, that was cool. But then so I wanna mention that, but also because you're talking about storytelling, put me in that mindset. But then the the women in tech thing was so interesting because my wife, Amber, was an engineer, and she's always been in kind of a male dominated field. And I remember talking to her for the first time about a lot of my because I met Amber actually working with her at Del Monte. And we weren't we we for a lot of that time, I worked there and she worked there, we weren't together.
Rick:And later in our relationship, we had these conversations about, like, all these blind spots that I had at the time. And I just remember being blown away by the difference in experience and the difference of perspective and just if you're not used if someone hasn't shown you or told you, you just there just might be a bunch of stuff you don't see.
Joe:Oh, yeah. That's and I looked around the room and I'm
Justin:Like what? Maybe I'm still shadowed by this, but I've had teams where I've had mostly women on my team and everything. Yeah. But, yeah, I'm trying to think of, like
Rick:Well, so so a lot of
Joe:the stuff that we had talked about before were some of the
Rick:dynamics of decision making because it that that time in that culture, there were certain rooms where certain decisions could be made and other rooms where those decisions couldn't be made or opinions that would be taken into account and opinions that wouldn't be taken into account. And it's not to say that it was all 100% always gender based, but there were elements of it that may have been or at a minimum could be perceived as such. And as soon as sir I've I've found, as soon as some things like that if that perception starts to cement itself, then all of a sudden you have a whole host of issues and challenges with people engaging the way that they would typically want to engage in in providing feedback and all that those sorts of things. So, I mean, effectively, even if the culture doesn't have elements of any any type of ism in it. Right?
Rick:Even if it if it doesn't materially if it feels like it does, it doesn't have all the same impacts, but it certainly has a bunch of negative impacts anyway.
Joe:Yeah. One of the things that came up so give you an example, something that came up, Triss in the conversations, it was very interesting, is that if you have and I noticed this a lot. If you're on a, like, a Teams call or Zoom call or something like that, there will be no qualms of, like, we'll even do it right now, of me talking and Rick deciding to start talking, and we try not to cut each other off, but that happens a lot more frequently. And one of the things that somebody on the panel had called out is that during these teams calls that have men and women on it, the men are more likely just from whatever social thing they do to start talking over
Rick:To interject.
Joe:Just like that.
Justin:Yeah.
Joe:Yeah. Except you would keep going then Right. And and would interject and start giving their opinion while all the women on the call, they said, would wait and like they'd hit the raise hand button, but nobody acknowledged it. That kind of thing would happen. And so hearing them say that makes me think, alright.
Joe:Now I can pay attention to that. So that's one of the things. I looked around the room and I was like, if I only my this would be something I think a lot of women would go to this talk. And I looked around the room and there were a fair number of men in the talk, which I thought was good. And so that's just some of my takeaways.
Justin:Yeah. Yeah. That's interesting. I think that's more personality based, though. Because I've seen women actually interject, you know, oftentimes too.
Justin:So and I I think it's hard to nail that down to a a gender or sex.
Joe:Well, I think one of the points that we're making is that the majority of women Okay. Are not going to interject and the majority of men will. And now what's gonna happen
Justin:That could be.
Joe:Yeah. The men are going to get their opinions heard, and the women are gonna wait as and maybe it happens more. I don't know if it's more of an online thing or if this is in a meeting or whatever.
Justin:Because there is a technology delay oftentimes that you're also battling with, you know, type of thing. So
Rick:Yeah. But I I mean, I think there are and, again, this like, every person is individuals are individuals. Right? But there are, like, behaviors that are biased towards genders typically. Like, I I remember seeing a thing.
Rick:It's prob it may still be true. I don't know. But it was true a bunch of years ago where men were much more likely to stretch resume experience in certain ways to fit a
Justin:job. Yeah.
Rick:But women would be very truthful in terms of their experience and and very, like, specific about what it is and what it isn't.
Justin:Yeah.
Rick:And because of that, right, that leads to maybe men getting an interview that a woman wouldn't get. Right. And it's kind of behavioral. Right? But if you don't create the spaces to make sure that the right resumes are getting to you, however you would do that, right, then you're not necessarily getting the best candidate.
Rick:You're getting a candidate that happened to behave a certain way maybe whereas others did. And I think the meeting thing is the same the same concept. We don't create the space.
Justin:Well, yeah. And I saw it it was something similar to that not exactly where if you had, like, 10 criteria and, you know, let's say you have a guy that only met, like, seven of them
Rick:Oh, he'd still apply.
Justin:He would still apply. Yeah. And if a girl met only seven of heard something like that.
Rick:I think
Justin:that's She wouldn't apply. Yep. You know, type of thing, which in that scenario, that's really hard to then look at the resumes you didn't get, you know,
Rick:in It's like a self selecting. Yeah.
Justin:And that's where, like I mean, there's actually psychologists out there that will train women to basically compete in the business world, you know, with other men and overcome things of that nature. It's like, hey. You're good enough. Just apply for it. You know?
Justin:Right. The worst case you get is a no. You know? But then
Rick:if you're an entrepreneur, like, to your point, like, how do you force people to apply? Extra to encourage people that typically wouldn't apply in certain ways to apply in those ways?
Joe:Yeah. I don't know that. I don't know. I would separate those two things because if I'm hearing what if I understand what you're saying, if you're an entrepreneur and you're trying to get people to apply, like, don't know that you can really do a whole lot other than, you know, maybe just ask people to apply. Yeah.
Joe:But really, the whole like, the whole point of the of that panel was how to break through that glass ceiling. And it was tips about what, you know, not the specifics like you have your occasional people who have already figured this out, but how do the the vast majority of the other women who haven't figured out that they should be doing these things to get ahead, not feeling bad about it. Yeah. Right. Not feeling bad about applying if they're only have 70% of the items.
Joe:Gotcha. And so I encourage everybody, like, go out on a limb. Take a risk.
Rick:Do it. Yeah.
Joe:Yeah. Yeah. And because, you know, I don't want to say this to any you know, just for women or men, but everybody should should always be doing that.
Rick:Right.
Joe:But then the other side of it is if you're a person who's prone, and I am sometimes, to start talking on a on a Teams call, like, be aware and just look around the if the people are on video, look around. Is there somebody who looks like they might wanna say something? Mhmm. Encourage them to it or give them the room to do it? That kind of thing.
Joe:So it opened that kinda summarizes the whole thing. It opened my eyes to yeah. I think we still need to keep paying attention and do what we can. And I'm proud to say, like, I believe, like, at least half, if not more now of my company are actually women. So Okay.
Joe:That's good. Yeah. We have a majority of of women on the team, and they I think they all do a really good job of being heard. Yeah. And I think we have become very conscious to make sure that everybody gets heard.
Rick:That's great.
Justin:Yeah. Yeah. But it's not like we'll talk a little bit later on, you know, like, building a SaaS and everything. But when I put out a job requirement for my first full time developer and everything, I got, like, 900 resumes Oh. And less than 5% were women.
Justin:You know? Yeah. Like, it was just and it was worldwide and all that stuff and everything. But, yeah, the the population is just not there.
Rick:That's what was gonna say. I wonder you you know, because I do I do think there's stats about what genders gravitate towards specific fields too. Yeah. And it doesn't necessarily have anything to do with merit. It's just, oh, there's a lot more people of this gender in nursing or teaching or, you know, technology or whatever that is.
Rick:So your populations will naturally be skewed based on some of that stuff.
Justin:But I think, yeah, as long as you give, like, you know, open and equal opportunity and you're looking at, you know, merit based, you know, at the bottom line and encourage people to improve in any position that they're at. You know? I mean, I think that's kinda bottom line into that. Like, I'm thinking about it. Like, I don't think about this a lot, but, like, my company is actually half and half right now.
Justin:My two full time is a guy and a girl, and my two part time is a guy and a girl. And but I didn't
Rick:look designed to do that.
Justin:It's just they're the right candidates. Exactly. They were the right candidates for the right job at the right time, you know, type of thing. So yeah.
Joe:Yeah. That makes sense. And it's not like you weren't intending to do one thing or another, but you got lucky enough that you had people apply who were qualified. Yep. They weren't being felt like they were held back and and, you know, they they ended up getting getting the position, which is great.
Justin:Yeah. Yeah. Yeah. Absolutely.
Joe:So back on the the game show stuff you did. Oh, yeah. Yeah. What were we talking about?
Justin:Yeah. Yeah.
Joe:I think we're talking about Tris, but this became a really great conversation that wasn't even on the agenda. So we should add this one as a separate Yeah. For that topic. But on for the game show stuff, I saw you recording some of the
Rick:people who are coming
Justin:up there. Yep.
Joe:Are you gonna did they let you any of them let you release
Justin:the first? We didn't really ask type of thing. But I mean, they knew where we're recording Yeah. You know, into that. I think I don't think I'm not a 100% sure, but I'm leaning toward what we're gonna be doing is kind of a Tris snapshot type of thing, like a five minute video.
Justin:The highlight's real. Yeah. Exactly. And have music playing in the background. So we're not gonna actually have somebody sitting down, and we're gonna, you know, listen to all their questions, but we'll have clips of that.
Justin:I love it. And I did some b roll walking through the the conference center and everything with my other camera. So I think we're gonna do something. Maybe I'll do, like, an overlay, you know, into that. But Yeah.
Justin:I think that will probably be better.
Joe:I think yeah. You should feature that one question that I saw. I wasn't there when you were recording all of them, but I saw the one. And the question was, name a name a security podcast. And I think the you had
Justin:Yeah. Our banner right behind right behind you. I'm like, look. Look. Yeah.
Justin:And they
Joe:answered they answered it right.
Justin:Yeah. Yeah. That was awesome. Was pretty good. So yeah.
Justin:But, yeah, we'll do that, you know, week or two or something like that. You know? No rush. But, yeah,
Joe:I think that'll be a
Justin:good promo, and I'll give it to you guys to Yeah. Send out as well, you know, into that.
Joe:That's great. Yeah. I had a great time with the whole thing, and I was super appreciative that they let me plug b sides date in the morning opening as well.
Rick:Just dropped. Okay.
Joe:Great. Yeah. So it's gonna be on July 10, and the other thing I got to tell everybody is that the sponsorship packet is soon ready to leave by the time this cast gets put out for publishing.
Justin:Yep.
Joe:We're hoping the sponsorship package is out. I think it's gonna go to all of last year's or previous sponsors first, and then it'll be opened up to others so that folks can get a chance to get get a hold of some of the coveted one one only slots. Yeah. Nice.
Justin:Great. Alright. Next topic here. Can you really rely on cloud service providers anymore?
Joe:Can you really rely on DNS?
Justin:Yeah. Right?
Joe:That wasn't all the others.
Justin:It wasn't all. So we wanted to bring in and talk about a little bit of the AWS outage here that was on 10/20/2025. And then additionally, just yesterday, Microsoft had a global outage with their front door, which I wasn't aware of, but it's essentially an application load balancer Yeah. At the end of the day with that. So did one of you guys wanna go over a little bit of, like I mean, it was DNS, obviously, with AWS.
Justin:But, you know, a little bit about the the events and what happened. Do you have those notes?
Rick:I don't think I have notes on the details of it.
Joe:I have
Rick:some thoughts about the follow-up. But
Justin:Yeah. So I have it in front of
Joe:me.
Justin:Okay. So, yeah, with AWS, it was, of course, US East one. And if you're not familiar with US East one, they push most of their, like, new features, you know, into that. So if you're, like, bleeding edge, you're usually going to there. That's where they release a lot.
Justin:They're, like, alpha stuff or, like, big features or new services or something like that.
Rick:It's like before it's rolled out globally?
Joe:Yeah.
Justin:Because, I mean, if you look at, like, AWS's design, you know, they have all these regions, US East 1, East 2, East 3. But if you actually look at the the design behind the scenes, a lot of them are and I don't know if it's all, but a lot of them are, like, three data centers tied into, like, the one service. So they're, like, kind of a a try failover into that. Yeah. And the entire stack is basically built in there around that.
Justin:But while most of it is, you know, like, copycat into that, they do release different services in different areas, you know, into this. And US East one, I think I saw a stat the other day that out of the dozen or so outages that AWS has had over the years, like 75% are USC Splunk, you know, type of thing. So, yeah, it it is interesting, you know, into that. So if you're looking for stability, don't do u USC Splunk.
Joe:Don't start there.
Justin:Yeah. Exactly. But, obviously, if you need some features and you're doing some cutting edge application stuff or AI or quantum or something like that that might be, like, you know, bleeding edge and they're coming out with it, you probably might
Rick:That might be the only place that's available. Right? Exactly. Yeah.
Joe:Yep.
Justin:So, yeah, so they came out with this. It was a, obviously, a DNS resolution failure with DynamoDB, and it was wide spread. You know? So I woke up actually, Episki was affected by this. Okay.
Justin:I woke up with a alert. I I figured it was, like, 03:00 in the morning.
Joe:I think
Rick:it went from I do know that it went from, like, 02:30, and
Justin:it was
Rick:resolved by, like, resolved by, like, 05:30, but it's DNS. So replication.
Joe:02:30 so just for clarification, 02:48AM is when Dynamo or, yeah, DynamoDB API errors started to have the DNS failures. And it was 05:20PM when a lot of the other items were were coming back.
Rick:Like, finally back.
Joe:Yeah. Core DNS by 05:25AM. So a few hours of the core stuff, but it had issues all along. Like 3,500 over 3,500 companies affected 60 countries, 17,000,000 users reported items logged on down detector.
Rick:So Flight delays, I think, I significant.
Joe:Yeah. Yeah. And so when something like that happens, you know, who's if if one if your DNS at your company goes down, then somebody's probably in trouble. When AWS goes down and you're at your company, you're like, well, this is affecting everybody. Yeah.
Justin:Yeah. They probably have didn't have a good day. But you know what? I mean, a lot of the times when AWS has issues, they'll go back and do a new process and Mhmm. Correct root cause and everything.
Justin:I mean, I can't personally to me, I can't even imagine running a production line, like, pushing out features on an AWS type scale. You know? All the complexity and everybody's apps that are using it in a different way, and comp like, that is a massive thing to get right
Joe:Yeah.
Justin:All the time. You know? So, yeah, kudos to that. And, yeah, I mean, when once you get DNS wrong because of the, you know, the replication Yeah. It just, you know, the time it gets to actually correct records and everything is, you know, just takes forever.
Justin:So, yeah, we were seeing at least on my side, we're seeing issues up until, like, late Monday. And it was more of a we use a service on the back end that does kind of a Firebase but with post Postgres, you know, database and everything. And that's hosted on US East 1. And so we were seeing while the app was working, the interface of the management, you know, wasn't a 100% and all that stuff. So, yeah, it was it was interesting.
Justin:And then yesterday, I would say not as big, you know, but definitely impacted it. There was a lot of, what, identity authentication issues that were coming out with this yesterday. Disruptions went to Outlook, Xbox, Minecraft, Alaska Airlines, Starbucks, Costco. I mean, I would say, like actually, a quick to speak quite frankly, like, I was actually surprised how many, like, DynamoDB, like, users ever actually, like, into this or affected whatever. Like, people still use that.
Justin:Alright. Okay. But yeah. But yeah. It was definitely interesting.
Justin:And I don't know attributed to a miscommunication in Azure Front Door.
Joe:So is it a change management issue at the end of the day?
Justin:Yes. Yeah. They said misconfiguration. So I'm sure they'll be putting in some type of linting checks,
Rick:you know, to control there.
Justin:Make sure that, you know, that configuration does, you know, doesn't slip by again, you know, into there. Yeah. So with this, I mean, things happen, things break, and all that stuff. The the core that at least I'm kinda curious about is when do you care with the service providers? So if you're a small shop, do you just live with it?
Justin:You know? Like, AVS goes down once every three, four, five years, and that's our acceptable tolerance. Or if you're doing, like, multimillion dollars an hour, do you do a multi cloud strategy to try and balance it out, you know, into that? Yeah.
Joe:What comes back down I always like to get back to basics. Yeah. So what's your business impact analysis? Right. And what are you relying on?
Justin:Yeah. And
Joe:so if you're a smallish company that can totally withstand having, you know, a few hours of downtime, some of the best business continuity advice I saw for some of that is, well, we'll just wait Yeah. Because it'll be back up in a period of time. But while you're waiting, what are you gonna do? And have you planned for, like, what's your what's your tiers of timings? So five hours?
Joe:Well, what if you're at hour three and you haven't heard anything and you're gonna hit your five point? Then what are you gonna do on hour six? Is it time to do something else?
Rick:Right.
Joe:And a lot of companies, it's it gets pretty expensive to be in multiple regions of just the same cloud provider Right. Let alone having your application
Justin:double your cloud cost at that point.
Joe:And then how how much more is it to actually it's not just double your cloud cost, but it's double your engineering and your architecture That's a lot data.
Rick:Yeah. Research knowledge.
Joe:Exactly. In order to be, you know, redundant between two different or three different clouds.
Justin:Yeah. And a lot that you could do, like, with front end. Again, like, I I wasn't aware of, like, Frontdoor, but, like, an application load balancer, you know, if that's hosted into there, you know, that might be your single point, but it could route, you know, into in between different things.
Joe:Yeah.
Justin:But yeah. And I it was actually very funny. My insurance company, why I don't have, like, cloud coverage, I got an email from them after the AWS outage. It's like, here's how to submit a claim.
Rick:I got that same email. Yeah.
Justin:So and it was like, oh, yeah. Like, people, I guess, do take out insurance to
Joe:Mhmm.
Justin:Cover this. You
Rick:know? Because even if it's not full cloud, like, depending on the t's and c's of your policy, like, might have business interruption insurance Yeah. For instance. And there might be an, you know, elements in there that you could file claims even if it's not cloud specific. Right.
Rick:Yeah. One of the things that this put me in mind of was just how things evolve over time and how, you know, it used to be people didn't have backup diesel generators in their data centers. And then all of a sudden, it's like, oh, you know what? We need that because the power was a commodity that you just can't go without, and it causes too many problems when you don't have it. And then it used to be that there weren't redundant, you know, telco lines into, you know, demarks and data centers and things like that.
Rick:And then all of a sudden, oh, well, that's just typical. And and I started to wonder, oh, are these sort of core cloud services becoming more utilities in a way than Yeah. Than an application? Because I think forever they've treated like an application. Right?
Rick:But I think it's we might be getting to the point where it's starting to shift where these cloud platforms kind of function like utility. And if they go down, to your point, it's like, oh, the world's having a bad day or, you know, this half of the country is having a bad day or all these major services are down. And it's not that different than, you know, whatever a major solar player or
Justin:something like government stuff with FedRAMP Absolutely right. Tied in and all that stuff and everything.
Joe:Well, I like the mental model you're creating because I never really thought of it this way. And how did this stuff start to evolve over time? So you have your data center and you have your one your one line
Justin:Yep.
Joe:And then your one power Mhmm. And then you added your second power. And actually, I don't know which came first, adding redundant power I don't know. Yeah. Or having redundant telecom.
Joe:Yeah. So you might have had those. And then what's happening is everything's kinda working so smoothly with it both coming in at the switching layer or your power switching layer is just making that invisible.
Justin:Mhmm.
Joe:So now anybody who's running on top of that, that's invisible. Right. So how do we and as somebody solving this problem, taking the multi clouds and making it invisible
Rick:to be able
Joe:to run on top of that layer.
Rick:This is the the next note that I had was exactly that. It's a you know, there's I'm sure there's probably some cloud engineer. I could imagine this being like a a big thing, which is exactly what you just said. It's like, hey, I wanna be I'm AWS native, but I want to have the redundancy of something else.
Justin:Mhmm.
Rick:So just like it used to be set up multiple data centers, like, is there some sort of semi transparent thing that at some point someone can build and then buy and says, hey, look. I know how AWS works, and I know how GCP works, and I can read your settings in AWS and just replicate them to GCP like that. And we can figure out how to seed enough data or what your critical data is and stuff like that and make it somewhat transparent. So you're still paying for two things. But it's the same as companies would pay for multiple data centers when, you know, some still do, but mostly pre cloud days.
Justin:Terraform or some type of infrastructure as code.
Rick:Yeah. You could. I mean, sometimes it was just you would have bare metal and you could use Terraform to do it, or sometimes it would just be like, no. We have a warm site, and we're just not consuming you know, we're we're we're keeping the systems mostly off except for once a month, we'll turn them back on or whatever Yeah. Whatever the strategy is.
Rick:But I could see something similar with the cloud thing to say, hey. Look. We're gonna spin up a bunch of infrastructure automatically in a different cloud. We're gonna seed all the data from the past week or whatever. Get it bring everything current to some extent.
Rick:And if you have separated compute and swords, it's even easier because you can just move the storage over. And then potentially am I not far enough? Too close? Too far? Oh, thanks, man.
Rick:Yep. And then but anyway, do do the thing transparently as you were saying, but from a cloud native perspective. Because at some point, it feels more like a utility service than an application.
Joe:Yeah. So if you're listening and you are the guru cloud architect who has this. John Ziola. We we need you to be the special guest on one of the episodes and actually talk about how this works. Yeah.
Joe:I love that. That'd be interesting.
Justin:Yeah. Couple will come to mind at everything. So, yeah, we'll talk about afterwards. But, you know, another thing that's actually interesting. So you remember, like, all you guys, like, doing assessments back into the mid to early two thousands?
Justin:Like, a part of this was, like, where's your data center? How far away is it from your backup data center? You know, like Yeah. Yeah. These are the questions you were asking, you know, and this.
Justin:And now that it's like
Rick:Now it's you in you are you in USC one? Exactly. Exactly.
Justin:What cloud service provider are you in? Okay. You're good because they're doing all the backup strategy and all that stuff sort of, you know, type of thing. They're not backing up your data, you know, but could be maybe, you know, type of thing. But, like, I've also noticed it's not a ton, but some people are now, like, boomeranging back into back to hosting their own stuff, mainly price, you know, into that.
Justin:So, like, one of the companies I know that I I follow for years, they make Basecamp. Oh, yeah. Yeah. Like, project management. You know, way back in the day, they're popular.
Justin:It kinda dropped off a little bit. But they were share they shared their story migrating from the cloud. I think they were in AWS, and they saved hundreds of thousands a month switching to their own data center and standing up their own stack. And they broke down, like, all the pricing, everything they need. They even had, like, build a backup, and it was incredible.
Justin:You know? Like, you look at that, and you're like, wow. Wow. Like, that's a lot of money, you know, that's going out the door for cloud, you know, into that.
Joe:Now did that also include all of their people resources they needed in order to support that?
Justin:Or They stood that up. Yeah. They hired the right people or they retrained or something like that. That was all in there as well. You know?
Justin:And they broke it down and was like, yep. So after initial capital cost of getting all this infrastructure and everything, they basically looked at this and they said, we're gonna basically, like, break even in two and a half years, you know, and then Oh, very interesting. You know, pass out. We're gonna do that.
Joe:Yeah. So what what risks are they taking on doing that? I can think of I mean yeah. Yeah. People resources.
Joe:People Yep. Now have the knowledge, and it's a very specialized set of knowledge to have to handle and support a a data center plat a data center environment. Mhmm. And if some of one or more of those people leave, now they have to rehire for that. So that's something you don't have to do, but at what cost?
Rick:Well, did they go from, like, was it cloud platform to their own data centers and they built their own data centers? Oh, so they're not even going with,
Joe:like I
Justin:don't know if they built their own, like, physical wall
Rick:because that's what I'm centers. Yeah. Because you could like, mean, there's still a bunch of data center providers out there. Yeah.
Justin:So That just gives
Joe:you Like, coding.
Rick:Yeah. Rackspace, Expedia, blah blah
Justin:blah blah. Yeah. So they might have done that. I don't recall that specifically. You know, the more important thing is that, like, they're not shelling out for all the compute time, you know, into that, and they built their own stack That's interesting.
Justin:Of tech. And they probably I mean, why wouldn't you, you know, outsource that to somebody else that already has generators and UPSs Yeah. Yeah. The infrastructure to do that type of thing. But, yeah, I thought that was really interesting.
Justin:Story. And it it really got me thinking. It was like, okay. So who is AWS really for then or any cloud service provider at that matter? It's like, you know, it's great for the small person that, like, you know, I'm trying to spin something up.
Justin:I'm a new startup or or something like that. Like, I'll tell you, like, a Pisky's, like, back end right now, I pay $30 a month. The entire back end infrastructure is $30 a month. You know? It's like,
Rick:the the digital version of WeWork or something. Yeah. It's like you you can commoditize and, like, share the the fixed costs associated with all this infrastructure. Yeah.
Justin:Exactly. But as soon as you get to a scale, then it makes sense to pollinate it in house. You know? And it's and I'm sure there's a big gray bleed area, you know, into that that Yeah. You know?
Joe:A lot of stuff.
Justin:Alright. There's a lot of work to actually flip over. So when is the burn worth the effort, you know, type of thing.
Rick:So Yeah. And I imagine it's like any other, like, pendulum that happens where one of the things popular it's just supply and demand. A thing's popular and is probably more expensive because there's a bunch of people vying for it. And then people start to migrate away in the other direction, then that other thing's popular.
Justin:Yeah. Yeah.
Rick:And all of a sudden, that's the expensive thing. And, you know, I think these pendulums happen in in pretty much every kind of technology or or, you know, shift that we see. Yeah.
Justin:So, yeah, I just thought it was interesting. I mean and then then the creation of, like, micro cloud service providers, like Oh, yeah. You know, like, with Vercel, and I'm trying to think of another one that I use. But they're basically, like, mini ones that give you, like, a lot of the edge compute, a lot of the, like, easy stuff to do, but they're not flow full blown AWS. You know?
Justin:Like, they don't give you every single bells and whistles. It's like, if you need a fast website that's built on node, we gotcha, you know, type of thing. Outside of that, you know, it's like, that's not the right, you know, play. So
Joe:No. That makes sense. And just thinking about the the the when when when do you have to think about it? And what is the right time for for somebody? And as somebody who's, like, building building your application, at some point, does it make sense for you to use one of the services that are out there?
Joe:And there there are services that do the multi cloud. They can do the like Kubernetes, multi cloud clusters. There's an open source project, Carmada, if I said it right, enables you to run this kind of stuff. And then some vendors are very specific to to doing this stuff, but it's not something you hear a whole lot about. And when you start to look at your your own impact analysis Right.
Joe:And and thinking about like a Episki, for example, if it's down because a w you know, AWS is down, then is that tolerable? And for your target audience of customers, is that tolerable? Well, I mean, what's what's a Pisky solving at the end of the day? It's not making sure that the heart surgeon's able to perform that surgery. It's making sure that you have reliable evidence for your your various frameworks.
Joe:Mhmm. And if you have to wait a few hours to be able to demonstrate that, especially even in an audit situation, say, today is your audit and the customer's relying on that and AWS is down Yeah. Can you have a meaningful conversation with the auditor that that's just okay. We're just gonna cancel this part of the interviews for today,
Justin:and we're gonna do that tomorrow. We'll do other interviews for that. Everything.
Rick:Well, a thing I hadn't thought of before too, just thinking through and I fully agree with your note about, like, it kinda starts with BIAs. But third party risk management. Like, I mean, is are you asking your SaaS providers to what's your multi cloud strategy like and things like that? Or or how integral are those SaaS providers to your work streams and and all that. And think it does it you know, like everything else we talk about, it comes back to risk management.
Joe:Well, it's really funny because one of the topics that we didn't select today was the impact of, you know, your third party risk management.
Justin:Oh, yeah.
Joe:Yeah. We kinda touch on that almost every time because it's so important and it keeps coming up. But you're you're absolutely right is, you know, to what degree are you, you know, are you looking for in your supply chain that kind of resilience and where do you really need it? And so, you know, with 3,500 companies being impacted by the the AWS issue
Justin:Yeah. Mhmm.
Joe:And and airlines being impacted and people's flights being delayed, certainly, were some there that were very impacting while there were probably tons that weren't. So
Justin:Now have you guys ever looked into the insurance aspect of this?
Rick:Of, like, multi cloud strategies or, what happens
Justin:when No. If you insure, like, on like, if a cloud service provider
Joe:Business interruption insurance.
Justin:Business interruption insurance. Yeah. Have you ever looked into that?
Joe:Not specifically for a cloud outage like you both were mentioning. I didn't get a notification because I'm not relying on
Justin:something Yeah. And it was like a general term. I I'm just curious because I I have not as well. And I'm curious on the affordability and the use case of when that is appropriate. Cool.
Justin:You know? So obviously, you would have to be, like, single region or something like that or maybe not.
Rick:Like It depends on what you're doing. So, like Yeah. At Del Monte, I remember, it was a big deal. We definitely had business interruption insurance because Did they cover technology into that? Oh, yeah.
Rick:Yeah. Because it it it the heart of it I mean, if you put a perishable product if you work with a perishable product and your business gets interrupted, spoilage is a legitimate concern.
Justin:Yeah. Yeah.
Rick:And there's a whole bunch of other logistic issues in that. But I think, you know, when is it appropriate? I I think it really just depends on what's your what's your business model like. Like, how elastic is your supply chain and how elastic is your, you know, the supply side and the demand side. Because, like, would imagine if you sell the majority of your stuff over the holidays season, you would definitely want business interruption insurance at least to cover that period of time.
Rick:Because if there's a tech issue during that time, it, like,
Joe:potentially cripples you. What's coming up? Black Friday.
Rick:Yeah.
Justin:Yeah. So I I Maybe I guess it goes down to how expensive it is and whether it's worth it. You know? Yeah. Like any other insurance.
Justin:Yeah. And how easy it is to violate the terms of the insurance. You're like, what do you need to actually do a successful claim? Because, like, when we're when I was at gift cards, you know, like, you know, you know, three, four months, we were making, like, millions of dollars a day, you know, into that. And we had a loose, like, no push, you know, into that.
Justin:But, yeah, if we're down the millions of dollars, you know, that we're not collecting over our website, you know.
Rick:In my experience, and it's aged experience at this point, so that that might have changed a little bit. But from what I'd seen, a lot of those more prototypical insurance types like business interruption, those things that have been around for a while, it's you still have to, like, justify the claim and make sure you didn't do something stupid and, you know, all that stuff. But it's easier to make a claim for, like, business interruption insurance than it is to do, like, cyber insurance. Like, cybersecurity insurance is notoriously hard for reasons I think we might even have talked about before on the podcast. But business interruption and things like that, they're just more reasonable.
Rick:It's just like Yeah. It's like, you know, house insurance or whatever. Like, bad stuff's gonna happen. And so, you know, actuaries risk rank that and spread the cost out.
Joe:Yeah. I would look at it similar to well, are we in a hurricane zone? And was there a mass set of destruction that happened? And when AWS is down and it affects
Justin:Are you in US East 1? Yeah. Yeah. Yeah. Are you in US East 1?
Justin:Is that the only one?
Joe:Man. Everybody who uses DNS, you're not gonna get your claim. Sorry. Yeah. You have to memorize
Justin:Yeah. Your IP
Joe:Only those who have memorized your IP addresses Yeah.
Justin:Right. Type them in directly
Joe:can get, you know, coverage.
Justin:Yeah. Flooded. So wrap up Cool. Into this. Yeah.
Justin:Don't use DNS? Is that what I'm getting out of it.
Joe:So Yeah. Yeah. Just stop stop DNS. Actually, was funny because one of the news sources was incorrect, but it when I when I looked it up on my feed lead to see what the whole ADA or Azure issue was yesterday, the first thing that came up was incorrect, but it said it was DNS. Oh, that's And it's like made the joke of it's always DNS even funnier.
Justin:Yeah. Right.
Joe:Even when it isn't, it's DNS apparently. So but yeah. Lessons learned, you know. I don't know. If you do your impact analysis, understand what you're relying important.
Joe:On look at your supply chain. I mean, you can find previous discussions. You can find discussions all over the place in supply chain. You know, supply chain is so important that when they did, you know, even the CSF two o from one o, they Mhmm. Added supply chain.
Joe:Right? So it became a very critical thing. It's years past now since that's come out. And then, you know, you have to decide is your cloud service that you're selling to your customers so important that you need to actually figure out the multi cloud strategy. And if you're somebody who's a buyer relying on it, look and see if that's in your model.
Joe:Look at your threat model. Yep. See how it blends in.
Rick:And like most businesses, I mean, think about it in terms of when your business does certain things. Like, I mean, if you're heavily reliant on AWS to run payroll, but you only run payroll twice a month, and that's the only critical service. Well, you need it when you need it and the rest of the time you don't. So how do you think through
Justin:Yeah.
Rick:You know, because because I guess my point there being the criticality of various services fluctuates over time in two ways. One, there's times of the year, times of the month, or whatever that things are more important or less important, but there's also this concept of the duration of an outage matters materially as well. And so you might be fine in four hours, but hour five is really painful. Yes. Or four days is fine, but day five
Joe:is you know where your pain threshold sits. You may not. And if you don't, well, that's that that's tolerant of an appetite. Yep. And you gotta figure out where, you know did we talk about this in the previous one?
Joe:You have your interrelation between tolerant appetite, and that's how much risk you have an appetite to have. Yep. Your tolerance, which is what are your thresholds? And then their third part is when when in the tolerance continuum, do you have to start to react?
Justin:Yeah.
Joe:And you need to know when that trigger is and what causes that trigger. Right.
Justin:Yeah. Yeah. I saw something the other day that your defined appetite is not always your actual appetite, which I thought was absolutely true Yeah. You know, into that. So, like, we're okay being down for two days, you know, or something like that.
Justin:And then on, like, half a day on day one, you one of the execs get a screaming customer in their ear. They're like, we're not okay with this. Like, it needs to be up right now. Well, I think that has a Our biggest customer just called.
Joe:People are so forgiving of what might happen in the future and they are not forgiving about what's happening right now. Emotional. Yeah. It's just a common psychological Yep. Way that we are as as people.
Joe:I mean, just think about it. I'm thinking, you know what? Future me is gonna be totally much better in shape than current me and is gonna eat much better. But future me, you know, I never get to that point.
Justin:Oh, cheers.
Joe:Yeah. Yeah.
Justin:Well, that's alright. Yeah. Cool. Alright. Going on do we wanna do why don't we do alcohol?
Joe:Yeah. Let's do that and then and then we'll hit on you know, we're talking about Episcus, so maybe we can talk about that afterwards.
Justin:Oh, that's great. Yeah. So today, we decided to go with the black dark bottle being that Halloween is tomorrow here. This is actually a Penelope version. Just came out.
Justin:It's project x.
Joe:What is it? Penelope?
Justin:Penelope. Yeah. And we've had Penelope on before.
Rick:Architect maybe?
Justin:I think the Toasted.
Rick:Oh, Toasted. Oh,
Joe:that was delicious.
Rick:That was good.
Justin:It was very good. And we actually gave one of those bottles out yesterday at Chris and everything. But this one here, they're basically doing a experimental, you know, thing. A lot of the bigger distilleries, they'll do, like, you know, their staples. Mhmm.
Justin:And they'll expand their staples, you know, into certain things. But then they'll do, like, small batches to see where people like. And, you know, quite honestly, it also attracts a different audience. Like, a lot of the bourbon whiskey spirit, you know, community, they like the new. You know?
Justin:They like to chase the new taste, you know, what's coming around the blog, what's people experimenting with, what are they doing with Pete or double or triple casing or you know? And this one here is finished in a Olarozo sherry cask, which was coming out of Jerzez, Spain, J E R E Z, Spain. I'm probably butchering that. I'll I'll leave it to you. I'm not gonna try.
Justin:Yeah. Exactly. But essentially, it was double cask, like so they age this. It doesn't say the age onto it. It's a 108 proof.
Justin:And then they finish it aging in a La Russos sherry cask. So instantly, like, when I got this, it's a little hot. Mhmm. So it's a 108 proof. You definitely taste that.
Justin:It's above the kinda 90 is my sweet spot. So Right. Can taste it a little bit hotter, but then it has, like, a lot of sherry in the aftertaste. And it's alright. You know?
Justin:Like, it's good. You know?
Rick:I will say it has grown on me Yeah.
Joe:Yeah. As
Rick:I've been sipping it. Like, at first, the amount of sherry on the back end felt a little overpowering or cloying.
Justin:Okay.
Rick:But as I've been sipping it
Justin:You get a lot of sherry on the the latter taste.
Rick:Yeah. But as I've been sipping it, I like it more and more.
Joe:Yeah. I I love the nose on this when it comes after I drink it. And the and I saw you put a little bit I put a less chips of ice in there.
Rick:Yeah. Put a little cube in there.
Joe:And I only had a couple of small chips, but I opened it up and it It did. Kinda cut the burn. Yeah. And I liked it.
Justin:Yeah. So definitely recommend to try out. Maybe not stock your seller, you know, with this, but it's worth a try, I would say.
Rick:I like it enough to have a little more.
Joe:Yeah. What what's this what's a bottle like this go for? It's under 100, isn't it?
Justin:You asked me the hard questions. Yeah. Project x.
Joe:And that's that that's a fully black bottle. You can't see through that at all, can you?
Rick:No. No. Which is like, I like the black bottle, but it's kind of a shame because it's a really
Justin:nice color boost. Sorry. $70.
Joe:Oh, okay.
Justin:I got two, so I was like, cheers, guys. Cheers.
Joe:I was gonna say, while that lid's off, you should pass it over this way. Yeah. Yeah.
Rick:I'll I'll put the lid back on to pass it over Justin's laptop. He's he's working off one these days.
Justin:Now, do we wanna dive into f five here?
Joe:We can talk about it quickly. Yeah. Yeah. So f five breach is what we're talking about.
Justin:Yes.
Joe:And that was announced on October 15. So October, a big month for outages and breaches. And this was interesting. Lots of the Department of Homeland Security, CISA, directives coming out of it. And in summary and so if you haven't heard of this, in 08/01/2025, f five discovered a nation state threat actor, widely believed linked to China according to the notes we have, had long term persistent access to its internal systems, including the big IP product.
Joe:So what it did was it gave a lot of exposure of, you know, potentially foreign adversaries being inside of networks and it's heavily relied upon by lots of parts of our government. So that's Yeah.
Justin:That's why it's private sector too.
Rick:And private sector. Yeah. Yeah. Like a ton of the Fortune five
Joe:hundred is.
Justin:And my impression, like, just to interject, like, real quick, my impression of what they attacked, they didn't say exactly, but it sounded like it was, like, some type of wiki, you know, or internal kinda documentation site. Because they said they got snippets of source code and, like Some
Rick:customer configs.
Justin:Yeah. Exactly. So it almost seemed to me like they got access to some type of portal where they put a lot of this stuff up there, you know, type of thing. Because they were very adamant on saying, like, there was no compromise to the source code. They you know, the at least that they couldn't confirm
Joe:Right.
Justin:And everything like that. And yeah. And they said for source code, portions of BIG IP source code were stolen. You know? So I bet that's probably some type of internal documentation site, you know, that they probably got access to.
Justin:Yeah. My best guess.
Joe:So by the time you're listening to this, if you haven't already taken care of this, if you're using f five and you haven't patched, you're really late.
Rick:Yeah. You're gonna wanna do that yesterday.
Justin:Here's the thing. And I always thought that it was kinda odd. They confirmed nothing was compromised, but they released a whole bunch of new patches.
Joe:Yeah. Another one's compromised, but there was a lot of patch right away.
Justin:Yeah. Yeah. So What does that mean?
Rick:Tough to draw conclusions.
Justin:Yeah. I
Rick:mean, because it could it could be one of those things where, oh, we have an identified issue. Let's highly scrutinize all this stuff. Oh, we found a bunch of other things along the way.
Joe:Well, how many times have we heard this story and that's exactly what it's been, nothing compromised. And then six weeks later, which we're not to yet. Right. You get the next announcement of, oh, it's bigger than we thought or it was deeper than we thought. Not that we're paranoid on this podcast, but I think we're a little paranoid on this podcast.
Joe:Yeah.
Rick:But I think it's I mean, it's common. It's a common approach because in investigations are crazy.
Justin:And Yeah. But I I guess I just look at it from a logical perspective. Like, nothing was changed, what new code are you pushing out? You know? And I don't think and it it could be my fault.
Justin:I didn't dig into it that deep, But I saw that they're pushing out a whole bunch of patches.
Rick:Oh, yeah.
Justin:What do you push out then?
Joe:Right. You know? They they they have some things that they said were vulnerable. And part of it, 600,000 f five BIG IP devices, potentially potentially, keyword, vulnerable, and over 250,000 of those exposed in The US alone.
Justin:Yeah. Right. Yeah. That I'm
Rick:sure that language was carefully crafted, but I can't pierce what it would mean. Like, potentially vulnerable based on what? Like, there's like a bunch
Justin:of things. Yeah. I mean, it could have been in that that Wiki that they had access to. Maybe they're discussing potentially undisclosed vulnerabilities.
Rick:They I think I did read that that was something that they had access to Yeah. Was undisclosed vulnerabilities. And so maybe that's what all these patches were. So they're like rapid patching of
Justin:all those patch things. All those, you know, so that, yeah, they didn't have a chance to actually
Rick:Yeah. I could
Justin:see that. Yep. That could be with that. But, yeah, I think everybody's fear, especially when it first came out, was is this another solar wind, you know, type of thing. Yeah.
Justin:You know, where now, you know, we've had, you know, nation state actors into a whole bunch of devices all over the place and everything. And it appears not today, you know, into that. But yeah. And I also thought it was interesting. One of the things I you know, because I follow regulatory and all that stuff, The they were compromised in August.
Justin:They didn't release until when did they released? October 15. October 15. And the reason that they came out and they were actually cleared was because the government had involvement into this.
Rick:I thought I thought I read that the attackers had been in the system for over a year though.
Joe:So yeah. No. Just just a clarification. It was discovered in August '20 Okay. '5 Yeah.
Joe:For what Rick said. So you were right on the on the date as well. And so what you're saying is because they they weren't guilty in the public eye of not disclosing sooner because they The Department of Justice grant it. Yeah. Government Oh.
Joe:Government investigations were happening inside, and they told them probably because of how much the government uses it
Justin:Yeah.
Joe:They wanna investigate first before this is being public.
Rick:Yeah. But I I thought that was an interesting point too, how the attackers were, you know, the the systems had been breached for over a year. They were just sitting there listening or not doing anything,
Joe:which
Rick:I always think is such a high risk, high reward tactic. Because at any time, they could probably, like, get kicked out and not Right. Maybe get the things they're looking for. But if they make it past that one year mark and all the logs roll over, then all of a sudden, well, how they get in becomes much more difficult to answer.
Justin:And one of the things I don't think it it's been in any of the news, at least from what I saw too. I I always want them to give us, like, what was your IOC, the indicator, you know, of how you found this?
Joe:Right now, was looking. I couldn't find anything on that. Yep.
Justin:Yeah. Because, like, it's in over a year. So what triggered you finding this? You know? What changed?
Justin:What popped up? You know, like
Rick:Because we're talking about nation states and the DJs involved in stuff, I'm pretty sure that's guarded information in case those same IOCs are elsewhere.
Justin:Yeah. But CSO is pretty good at, like, blasting those out when relevant, you know, into that. They'll release IOCs and stuff of that nature.
Rick:They'll hold some too Yeah. Depending on what they're doing.
Joe:Yeah. Even even a quick search right now says that for exactly how it was discovered, there's no public disclosures yet. Mhmm.
Justin:And I find that unfortunate. Because that's the stuff I wanna like, because as people advising companies for protecting companies and all that stuff, I'm like, tell us how you discover that.
Rick:It would be good to know if it's applicable.
Justin:I want to, like, adopt a
Joe:yeah. Exactly. So what they say that they do know is internal monitoring. So not nothing that's gonna be surprising. Right.
Joe:But the only thing written internal monitoring, security investigation, forensics, that kind of stuff all came together. So somebody probably was looking at the right thing at the right time, noticed something, started looking a little bit harder, realized something is a little odd, and then it went It might have been
Justin:an IP address, maybe, you know, pinging up, you know, at odd times or something like that. Because the actual activity I don't know the story, but if they're just looking at a a wiki, you know, that's really hard to track down. Right. You know, type of thing. That's probably not a system that has access reviews.
Justin:That's probably not a system that, you know, would be flagged on anything. It it depends on what they're doing. You know? But it might have been their sock be like, what's this, you know, IP coming from Beijing? You know?
Joe:It could be.
Justin:Four in the morning. You know? That's weird. You know? And then look at him to do it and then find the root of the compromise.
Rick:Yeah. I've also seen a couple times in the past situations where a like, change management essentially or change management of a legacy system that needs to be reverse engineered before it can be migrated elsewhere results in higher scrutiny because the IT teams or the InfoSec teams are like, well, how's this even being used today in the enterprise?
Justin:Yeah.
Rick:Because we need to move it from this place to this other place. And they start looking and they go, hey. I this is weird traffic. What are you doing, business person? Here.
Joe:And I go, well, I'm not doing that. Hey. Well, what do you mean? And then they look and they look and they
Justin:go, oh, no. Right. Yeah.
Joe:I don't know if we'll ever know, but, you know, it's a good story.
Rick:It's not not not to rely on, but, yeah, I've seen change management be a a tactic for, you know, like, threat hunting effectively.
Justin:Yeah. So yeah. So some of this story I mean, I don't think there's really too too much we can gain with this. I mean, they they haven't shared a whole bunch of details into it, but obviously, making sure you're patching to your infrastructure.
Rick:And everyone's like, do zero trust and network segmentation and least privilege and, you know, all all your standard stuff, which people have a hard time doing because it's hard and no one has money and
Justin:So do you remember the advice, like, way back in the day where they they said you should always have, like, two different vendor firewalls?
Joe:Oh, yeah.
Justin:Yeah. You know, within your DMZ or something like that. You know? So a compromise of one wouldn't actually lead to a compromise of another, you know, into that. Potentially, that was the thought process.
Rick:Yeah. I I do. You know? But it makes me think if I were to manage that today, I would want a transparent layer to make the config changes to both of those at the same time. And then I'm like, oh, well, does that what if if I make a config error then and it bricks it or opens something up in the wrong way, the redundancy no longer matters because, like and I think that's the thing that gets missed sometimes, and this applies to the prior conversation and this conversation.
Rick:But if you have a redundant solution, but you manage it centrally, right, the cost of a mistake kind of disables the fact that you have redundancy in the first place.
Joe:Yeah. You're not literally just defense in-depth because you remove the depth, you make it equilateral.
Rick:Yeah. You need, like, a layer one, you know, team to deal with the one firewall and a layer two team to deal with the second firewall.
Joe:Yeah. Now I can see the benefit though. So if you know that you're using f five on the line and you're using something else, soon as you hear about this, you can just kill it. Just turn it off, unhook it, and then go and deal with your patching.
Justin:Figure out what's happening. Parallel. I was talking about doing it two different layers. Serial. Oh, I
Joe:get you.
Justin:Yeah. That's that was
Rick:I was talking about parallel management and configuration.
Joe:Right. Yeah. Okay.
Rick:Yeah. But yeah. In serial in terms of traffic flows.
Joe:Right.
Justin:Yeah. Where like the border firewall would be at five and then your
Joe:Oh, well, yeah. You could still just like bypass it and but still have your firewall protection and you just don't have two of them for the time it takes to Right.
Justin:Right. But let's say, like, if an f five compromise happened, and let's say they had full access, your back end firewall wouldn't open up your entire network. So your d DMZ might be wide open, but it's generally open anyway. Yeah. You know?
Rick:It's just like anything else. Like, how many how many internal doors do you of your house do you lock?
Joe:Yeah.
Justin:It's like, I don't I
Joe:don't know.
Rick:I lock the outside door. Don't I typically don't lock like the door to the basement and then the door to the bedroom or whatever.
Joe:Yeah. You don't lock all the doors
Rick:on your way in? Yeah. One after the other. Yeah. Amber would kill me.
Joe:Yeah. And it's just like Maxwell Smart going down the hall.
Justin:Well and the thing is, like, well, you guys know, like, with my house and everything. So we obviously locked the door. But one of the things we do on the security system is anytime an external door gets open, it beeps. And I had them actually install it was funny. The security guys said it never happened before, which I thought was really odd.
Justin:So the alarm panel is next to our mudroom, next to our garage. You know? And I was like, well, if there's an intruder, like, I I have to walk all the way downstairs to disable the alarm. Was like, I want I want one in my bedroom. Yeah.
Justin:Yeah. I had
Rick:one button totally makes sense.
Justin:Yeah. He he said he's like, oh, I never had that before.
Joe:Well, wasn't my customer in yours apparently because he would have had that.
Justin:Yeah. But, yeah, we have a a panel, like, in our bedroom that, you know, that I can actually look to see what door opened and stuff of that nature and then the thing. But we also have that it goes beep beep, you know, like
Rick:It doesn't play the John Cena music?
Justin:Yeah. I wish. That would be awesome. It'd get a little old after, like, all our kids
Joe:Oh, yeah.
Justin:Get out of
Joe:the house. Do you have turrets in both both places or just turrets for the outside?
Justin:Yeah. Right. You know, the what is that? Idiocracy when they, like, shoot each other.
Joe:Oh, that movie's more true and true. Oh, yeah.
Justin:Yeah. So we should do, like, a solid movie podcast.
Rick:Oh, that'd be a good time.
Justin:Yeah. By the way, favorite Edward Norton movie?
Rick:It's probably Fight Club.
Joe:Same. I really really
Justin:So why he was excellent with that? I think I would have to say Rounders.
Joe:Oh. Is great. That was the one about gambling. Right? Yeah.
Joe:Yeah. Oh, I did love that movie.
Rick:He told a cool story about Rounders.
Joe:Oh, that's true.
Rick:Well, so he was talking about empathy, and he was talking about how you've in his experience, you find different he called them access points to, like, identify with the emotional state of the person you're trying to be, and it's different at different times and stuff like that. And and I think for for rounders, it was it was the jacket. And it was one of those things where he's like, oh, I know who I know who I am supposed to be as this character because I got the jacket, and the jacket's everything. And he said he said, like, sometimes it's a walk or sometimes he was telling this story about another a short I think it was a short film that he did that had to do with him being an inmate at the prison, and it was a voice. Like, he actually was having trouble figuring out how to be the person, and so he interviewed a ton of prisoners, and one of them just had this super gravelly voice.
Rick:He's like, and as soon as I heard that, I knew who the character was and how to act as that character. So anyway, I mean, going back to storytelling a little bit, like, out your access point for the story that you're telling to to drive and build the empathy.
Joe:Gotcha. It was cool. Anyway, yeah.
Justin:So you're not like you're going in with a gravelly voice. Is that what I'm hearing? Right. Right. Right.
Justin:Come on. The metrics.
Rick:Right. It's like, this is just going as bane.
Justin:I don't know. If if you
Rick:have a Halloween board meeting, it might be worth
Justin:They always had that weird voice,
Rick:Oh, no. You're right. That wasn't gravelly.
Joe:It was it was the one that was kind of like theirs. Yeah. Yeah. It's like he's on the
Justin:border of, like, puberty or something like that. It's like, what what is this voice? You know?
Joe:Yeah.
Justin:Yeah. But I loved him in that.
Rick:And Browns.
Justin:Know what? Another one that he did for Nolan was The Score with Robert De Niro, where he's like that retarded janitor. He's retarded the right word. Well, anyways, he played that, like, special needs, you know, janitor Yeah. But then was not that outside the way to get in to steal, like, some of the jewels and everything.
Justin:Yeah. Oh, yeah. I remember that. It's like yeah. When you have characters that are that dynamic
Joe:Oh, yeah.
Justin:And can play and you totally believe their character. Like Yeah. You just bring them life. Worm in rounders. Like, he was that character.
Justin:Oh, yeah. You know, type of thing.
Rick:It's it's Yeah. Incredible to watch. Yeah.
Justin:That'd be cool. Yeah. Yeah. Last topic here. How to fail at a SaaS?
Justin:And now No. No. No.
Joe:We can't let you do that. I won't let you do that, my friend. So the topic is actually launching a sack.
Justin:Oh, right. Right. I got that wrong. Yeah. Yeah.
Justin:You you're
Joe:not failing. You're succeeding, man.
Justin:Oh, man. The the times that, like, I failed at everything, like, so
Joe:How many failures does it take to get the success?
Justin:Right. Exactly. Yes. Yeah. How many light bulbs is it?
Justin:You know, failed light bulbs. You know? Right. And just be right with one. So, yeah, TV, a little background and everything.
Justin:So about and it's almost embarrassing to say how long. I started off with a beta probably seven years ago.
Rick:And you've and you've and you've doing this full time since then. Right? Like, that's all you've been doing for seven
Justin:oh, okay.
Joe:You'll be you'll be an overnight success. It only takes ten years.
Justin:Yeah. Yeah. Exactly. Yeah. My first iteration so I had let me back up a little bit.
Justin:So with Hapiski, I had an idea at Diebold at a previous company as I worked at with that the GRC market was serving basically the upper echelon of organizations. So ones that had a whole bunch of complexity and can afford all this money to actually solve it. And I actually remember, like, it was way back in maybe 2010. I worked at Diebold. We bought Archer.
Justin:You know? Archer was the premier GRC tool at the point at that point. We bought four modules. Mhmm. 50 licenses, you know, for people I
Joe:thought you're gonna say 50 people in order to run it.
Justin:50 people licenses into it. You know? Could be 50 people. But we bought four modules, and it was like policy compliance audit and a missing one.
Rick:Risk or something. Yeah.
Justin:Maybe something like that. We paid a quarter of a million dollars. And Annually. Right? No.
Justin:This was there was a separate $60 license annually, but we're self hosting.
Rick:But a ton of money.
Justin:Yeah. So this is that capital
Joe:of software, not just to get it all ex
Justin:Exactly. So we had to the and yeah. This this was just a license of the software, then we had to put it on our own infrastructure. So we had to get our SQL Server together, all that stuff, front end. And then then we had to hire some people to
Rick:Professional services to implement.
Justin:Yep. To do that. We went out to where was it? Overland Park in Kansas for a week for training. Sounds right.
Justin:That's where Archer was originally before Archer.
Joe:Charge you for training to go to the training class.
Justin:Right? They include it.
Joe:They include it. Yeah. There's a
Justin:line item somewhere. Know, quarter of a million dollars, you get a free week out. Well, no. We had to pay for our plane tickets out there and maybe hotel, but the class for was free, you know, into that. And I'm like and you get into it and you're like, all I'm dealing with is controls and evidence and, you know, like,
Rick:it's just a workflow system.
Justin:Yeah. It's just it's shouldn't be that hard. Why is it a quarter of a million dollars,
Joe:you know, having, It's same thing that we can do in Excel.
Justin:Yeah. Basically. And so I was like, yeah, there has to be there has to be an easier way, you know, type of thing. And So I started doing a prototype. I think at gift cards, I started doing a And it was all one big PHP monolith application.
Justin:So back end, front end, everything was, you know, kinda one thing. And it worked to for a time. Like, we actually I'd moved over at at that point to TrustedSec, and we're utilizing the tool Mhmm. And we got rocks out through the tool. It wasn't customer
Joe:What's a rock for those who know what using?
Justin:Report on compliance. It was a For PCI. The end result for doing a PCI audit. You know? So the full blown
Rick:Very specific report format.
Justin:Writing all of it, all the evidence, everything into the report. We did it all through the portal. It wasn't customer facing, but I produced status reports out of it on a weekly basis saying, you still owe me this evidence. You still like, here are the fine So
Rick:the assessors were using it to run the engagement?
Justin:Yep. And we did it probably half a dozen times, and it worked awesome. We submitted basically, we printed to PDF, the rocks, and we submitted to the banks, and there was no issues whatsoever. And so we got a little bit of validation into that. But one of the things we were seeing through this was dealing with all these controls and it being a monolith application.
Justin:You had to wait for it to load. And you could do some things with caching, but it was one of those things that, like, hey, this isn't gonna work out, you know, in the long run. You know, dealing with this amount of information. There's modern architecture, you know, for some of this stuff. And we then went into a different way of doing this, and we split it up.
Justin:After I left TrustedSec, we basically did kinda like Episcop two point o. Mhmm. And we split it up into a back end and front end. And the back end is actually run by a service called Supabase. It's actually open source, but we're using their service to kinda do the management into it.
Justin:It's like Firebase, if you're familiar with that. You know, a lot of different calls to database, almost seamlessly API driven, but it's run on Postgres, which is a known database and you can control it with that. It does all the authentication so we could do OAuth to, like, 20 or 30 different services, you know, into this, you know, which is great. We only do two, Microsoft and Google, you know, because that's what it is. But you could do Slack.
Justin:You could do a whole bunch of stuff. You we can do real time, you know, so we actually have real time syncing. We have local caching with the like, the modern architecture is awesome. The the problem of rolling out SaaS, there's there's a lot of different things that I even go over, like, analyzing the market and doing all that stuff. They're like, look at a problem and pulling at this stuff.
Justin:But the biggest mistake I would say I made with a PISCY is thinking I could part time roll this out myself, you know, into this. Like, it is a lot. And my primary revenue is still kinda is still consulting. So I'm still out there helping customers and everything, and that's paying two full time and two part time employees. So, like, my salary paid out.
Justin:Right? If you can not do that, 100% recommend to do that, you know, type of thing. Like, the the problem is that straddling and when you're helping customers out, you know, going back and forth, it is such a hard time Mhmm. Doing that, you know, into it. And it it's, you know, it it like, can you imagine doing an audit for a full week?
Justin:And then after you're done with that audit for eight hours, going in and actually developing more code. You know? It just it's not sustainable.
Joe:It might be easier to get up in the morning super early, drink your coffee, and then develop code Yeah. And then go do the audit because it sounds like the audit's gonna be the easier thing for your mind after a long day.
Justin:Yeah. Maybe. You know? And I, you know, I help customers on front end, you know, onto the audit, like, the interface in between for, you know, a number of times. But, yeah, it's just one of those things, like, if I'm coding and this is just the way I'm built, I need at least two, three hours.
Justin:I need another you know? A good time to start troubleshoot, get it out, get it done, get it done. Well, and then
Rick:you also have a family and you have industry events and all sorts of things like that that you're doing also. So, yeah, just nights and weekends Yeah. Would be really difficult.
Justin:So, yeah, that's one thing. It's like, you know, it's so much easier if you had the time of and that's why, you know, honestly, I I've taken zero funding, you know, so far, which is nice. It will be nice, you know, once this thing starts launching. But at the same time, it's really hard because if I was funded, then I wouldn't have to do consulting. You know?
Justin:Right. And it'd be a 100% of building this app.
Joe:So there's pros and cons of this.
Justin:Yeah. But then, like, I've had a number of conversations, but if you're doing kind of a pre seed, you know, which is where I'd be into that or angel, you know, depending on where you, like, split it up, they're taking 40
Rick:Oh, yeah.
Justin:Plus or minus percent of your company. Yeah. And no doubt, like,
Rick:I don't so high on there. Exactly.
Justin:Yeah. I don't I don't blame them for that, but, like, do I really give up half my company for a 100 k? Right. You know? Right.
Joe:Sounds like Shark Tank deals in.
Justin:You know? Exactly. You know? And that never made sense to me. Like, at some point, I might take some investment, but that's to put, like, fire on what is already working.
Justin:Right. You know, once we get a validated product, customers coming in, you know, money's looking good, we got a good team, then it's gonna be to, you know, just expand how fast we're getting stuff out there. Supercharge it. Yeah. Exactly.
Justin:You know, type of thing. But but it won't be at 40%. You know? It'll be at, you know, 10% or somebody.
Rick:And there's probably I I'm interested in your perspective on this. I I would suspect if you had taken, say, money early on when it was a monolith application
Joe:Mhmm.
Rick:You might have gone down a path or it might have been harder to turn around because you would accelerated so far in one way that then you recognize an updated architecture because you've learned things along the way and you go, oh, well, you you had the ability to pivot because you had some time to do it. Yeah.
Justin:Yeah. And to do that yeah. Because with money comes demands. Right. You know?
Justin:Yeah. Like, there's expecting a return at that point. So Yeah.
Joe:You're not your own boss.
Justin:Yeah. Exactly. So you're absolutely right. It would be way harder to pivot at that point, you know, unless they're, like, really good at understanding advise you know, investors.
Joe:I I love what I'm hearing. Won't remember all these points at the end, I'll just say when I'm hearing them is, you know, one is decide, are you going to want a boss or be your own boss for it? And that'll help you decide whether you want somebody else's money or can you find a way to get it done bootstrapped long enough to make it where it needs to get to? Yeah. And the other thing I'm thinking about, I'm not sure what your thoughts are or how much you consider this, is thinking about like the end game.
Joe:So I was like to think about where am I gonna be when I'm done and how big can it be? How many users can possibly want this thing and need this thing like market? Yep. And based on that times the number of how much they would pay for it, what does that look like? And, you know, for a lot of people, 50% or having only 50% of that amount is gonna give them the lifestyle they need Right.
Joe:When they sell versus, you know
Justin:And I'm not living on
Joe:all that.
Justin:So I yeah. I I've considered that. I'm not looking for a lifestyle company. I'm estimating that I have a number of spreadsheets, and I've done some analysis on different sites and all that stuff. But the GRC market is huge.
Justin:It's like a $127,000,000,000 market, you know, into that. Obviously, that's nowhere near what I can, you know, make out of this. You know? But I was estimating with the market and with some of this stuff within the next three to five years, I can probably get to and it's a big range, but 10 to $50,000,000 a year, you know, into that.
Joe:In recurring revenue.
Justin:In recurring revenue. Mhmm. You know, into it. Yeah. I think that's a doable number, you know, into that.
Justin:We'll see, you know, into there because right now, we're doing a very simple, and I always wanna make it, like, simple for the smaller businesses because, like, I hate when, like, tier c companies are like, here's a module and then it's per user, but it depends on the user and the access and, like, you can't even figure it out.
Joe:You're not even trying to compete with, like, an Archer in this case. No. Trying to find the
Justin:In fact, I tell my my team, our primary competitor is spreadsheets. Yeah. That that is our primary competitor. You know? Because
Rick:because the the smaller organizations are so underserved in this space.
Justin:Yeah. So yeah. And they're probably still on something, you know, like, they're still on, like, spreadsheets trying to deal with two or three compliance realms or just don't have a clear picture of what their program looks like. You know, spreadsheets are all over the place even if it's on OneDrive, you know, type of thing. It's still nobody knows where
Rick:Or they're growing their business, and they have a situation where, I don't know. We just landed our first big client, and they're asking me all these questions. Yeah. And I don't know what to do. And I
Joe:get that time. What's your ICP like employee count or your average customer that you're targeting?
Justin:So it it depends. It was funny. I was just having this conversation. It depends on the industry.
Joe:Mhmm.
Justin:The higher regulated they are or scrutiny they get, the lower it'll be. You know? Because they realize that pain point sooner, you know, into that. The less regulated, it could be way big.
Joe:A bigger company.
Justin:Yeah. So, like, manufacturing. Yeah. Or something like that. Like, they're not under a ton of scrutiny to get security right.
Justin:And, you know, they I could come up to them and they could be like a $2,000,000,000 manufacturing and have one person for security. Perfect. That's that's our ideal customer.
Joe:Yeah. And they're probably not CMMC at that point because like that. But they so not highly regulated even if manufacturing. Alright. That makes sense.
Joe:And so
Justin:But, yeah, it could be a 100% fintech. Like, one company I worked with, had trusted tech. They were pre revenue, all VC funded, trying to get into the financial industry. They were a 40 person company that had to get their product PCI compliant before they could even start to market it. And one of the problems that they had was, you know, we went in, we did a gap, problems everywhere.
Justin:They got you know, helped them out, got them fixed, went through the rock, got them compliant, you know, into that. We came back year two. What you guys were supposed to do a pen test. You guys were supposed to do quarterly scans. You guys are supposed oh, I thought we were done.
Joe:Yeah. As soon as you as soon as you went away, like, that that
Justin:went out of my mind.
Rick:We're trying to build a thing.
Joe:Come on, guys. So so a a tool like a PISCUI can help them Yeah. Remember what they're supposed to do, help them keep on track.
Justin:Yeah. So all the tasks and evidence are built into it, tied to the controls you need to achieve. We're actually working on we're doing a whole bunch of stuff with, like, of course, AI now, you know, because, you know, it's a thing. But one of the things that we have on the use case deliverable is based on the control set, suggest all the tasks that you need to do, and then it'll be an easy click button to actually say, yes. It says, I have to review my policies every year.
Justin:Yes. Create that task. You know, type of thing. Yes. I have to do quarterly access for you.
Justin:Yes. Create that task. You know, type of thing. It would just look at the controls and then say, oh, yeah. There should be a task for this because it is highly manual and you somebody needs to do something, you know, into that.
Justin:And then you can assign it to the right person Right. Or group and, you know, then track it, you know, on whatever schedule you need.
Joe:No. I love this conversation. I'm getting, like, two different framings we can do for this conversation, and I'm gonna let's let's solve them both here. One is, like, going back to Tris. There was a I I talked to a significant number of college students at Tris.
Joe:Mhmm. And I don't know how many of them, but some were thinking what's I wanna build I wanna build something. They wanna go to a near path. And if they're gonna do that, the kind of takeaways that we're getting from this conversation that I'm hearing is these students, like, listen to this. They probably signed up yesterday.
Joe:They probably came by and Yeah. Took a QR code.
Justin:They were
Rick:forced to do trivia. Right? Yeah.
Joe:And and they did that. So if you're listening, tell your friends that are developing software that there are a couple things to consider. And Justin went through a lot of this. So don't repeat some of the things that he had already figured out. Yeah.
Joe:But decide, do you want to get an investor and have the luxury of somebody, you know, giving you the money to get things done. And as a student, you you the you know, you have so many years ahead of you. So you can you can do it and if it doesn't work out, that's not the the thing. And the the other side for a student though, they don't have the ability to do what you do. They do not have the experience to go out and consult and put money on the table to to handle all this stuff.
Joe:So, you know, that's the other side. So
Justin:Yeah. And it's hard to identify pain points if sometimes you're not in, like, deep in an industry, you know, into that. The and it's a it's a interesting dynamic that kind of the the entire, like, SaaS industry is in. So one, there are a ton of stuff coming out because everybody with AI Mhmm. Can generate an MVP, a minimal viable product over a weekend.
Justin:Oh, yeah. Know, type of thing. They have an idea. They'll ask AI, Claude, whatever it is, like, generate this website, do this, do this, do this, this, and all of a sudden, they have a a basic product that just shows the what you can do and everything like that. And that's phenomenal.
Justin:And, you know, in this, like, kinda environment, you should fail fast. You know? Yeah. Get in the customer's hand and see what they like.
Joe:Well, some of these were PhD level students.
Justin:But it doesn't matter.
Joe:Like, have little bit high school. They could have I mean, adjusting your experience level. Yeah. Yeah. They have a little bit more knowledge than say you're just a four year college grad into it.
Joe:And so I could see them having a little bit more industry knowledge that they had to do and get in order to get to the PhD level. Yeah. But then being able to figure out if they're gonna fail fast. I love that. Yeah.
Joe:So that's the one part I was thinking about. But I'm gonna go back to just a Pisky in general. And you gave so much, but you were talking about things that made me feel a little confused at first, but then I think you got rid of it, which is you talked about all the various technologies and all the various, like, you know, stuff with the databases and things behind it. But your
Justin:buyers sometimes stop myself from nerding out after that.
Joe:But your buyers don't have to care about any of that. No. That's all back end stuff. They have to just go to the front end
Justin:Yeah.
Joe:And they get it. And And
Justin:you guys have both, like, experience it. Yeah. Like, it is lightning fast. It's like a native application on
Rick:And that's the key. That's like the what makes it great. Well, here's all the acronyms and things that I
Justin:use in the back end and how I've pivoted. Yeah.
Rick:But this is why this is part of why it makes
Justin:it great.
Joe:You did a lot of stuff in the last year alone. Right? The the cache thing so that would just be really fast. It would actually start, like, preemptively pulling data in knowing where you're going. Right?
Justin:Oh, yeah. Like and we've even done, like, native key mapping to it, like hot keys into it. Like, it is something like, if you're going control by control, you can actually hold in a key, and it'll go like it's like, again, like, it's running on your computer. And this is all over the web because we have, like, prefetching going on local, like, dataset that's actually stored into your browser. We're thinking about actually implementing a SQL lite into your browser at some point.
Justin:It's backlog ticket. Mhmm.
Joe:This gotta be super expensive though, but you probably don't tell anybody the prices either. They feel like set up a sales call. It's super hard for them to be able to get the information right. There's no way they're gonna be able to quickly just go online and quickly know what to pay.
Rick:He's teeing you up.
Justin:Yeah. Exactly. No. We have one flat cost and everything. It's $3.50 a month, 20% off if you pay by year, you know.
Justin:Oh, wow.
Joe:So it's super easy to buy. Yeah. And it solves problems, and it gets rid of the spreadsheet stuff.
Justin:And you can have unlimited users,
Joe:like Unlimited users.
Justin:And in fact, that was one of the things we were talking about earlier on, you know, paying for users. I never went to a point where if somebody's collecting evidence, say you have a task for every three months, somebody needs to collect evidence, and you assign to somebody that's that's not their daily job to be in a GRC tool.
Joe:Right. Like, oh, I forgot that I was supposed to do a Pentest together. It's like
Justin:the network guy. You know, like, hey. You know, you need to do an ACL review every six months, you know, type of thing. Well, I don't wanna pay for an extra user license, you know, into that. I just I I never wanted, like, dollars to be like, should I sign the user or should I just ask them to send me the evidence and I'll upload it?
Joe:No. They just automatically get access.
Justin:Yeah. I mean, that's the thing. Other GRC tools, they're like, oh, that's another user license. You know?
Joe:So Now let me ask you this. How does that engineer get to know that it's their time to do something?
Justin:They get a notification.
Joe:How do they get that?
Justin:Email right now, we're working on Slack. Yeah.
Joe:So they're already working in email. And in the future, they're already working in Slack. They'll just they'll get the reminder. Yep.
Justin:Oh, that's awesome. Yep. Yep.
Joe:So it's easy for people to remember what to do. You don't forget your stuff, and you don't have to misplace the spreadsheet.
Rick:Yep. It it puts me in mind of like one of my favorite Steve Jobs quotes, which is like, it's simple to make things complex, but it's complex to make things simple. And so you talk about all these technology on the back end Yeah. And all of the experience you have to have put together the architectures and the patterns and, like, how do you do assessments in a way that makes it simple for the end user to succeed. Yeah.
Rick:Right? And that's all the knowledge and experience and tech that goes into this. And I think the thing that I love about the output is, yeah, this is this is designed to be simple so that, know, it can scale for a big organization, but it's kinda designed for organizations that might have one person or one half time person, but they have these heavy compliance requirements. Because that's I think you probably see a lot of clients like this.
Joe:The the
Rick:small ones that with heavy regulatory burdens are the ones that have a really tough time dealing with that.
Joe:Right. Mhmm.
Rick:So I I I love that sort of mission, at least my perceived mission.
Justin:And I love that. Yeah. There was a Steve Jobs it was out on Facebook or something like that. It was a combative person. He took a question from the audience.
Justin:He's like, you should be using this technology. This technology is superior than what you're using today, like, has been out there for years. Like, why aren't you doing that and everything? And he had, like, a long pause, and he came back. He's like, you know, he's like, I failed at this probably more than anybody else.
Justin:Mhmm. He's like, you gotta start with the user experience and then work your way backwards. Yes. He's like, this person's probably right. We're not using the technology optimized for this.
Justin:But that, at the end of the day, does not matter.
Rick:But what are you optimizing for? You're optimizing for the end user, not
Justin:for the specific for the end user, and then you work backwards, you know, into that. And that's important. Like, I haven't done I haven't perfected totally, you know, into this. But, like, some of the stuff we incorporated this, like, I wanted to do key back shortcuts. We have a universal search.
Justin:It's like command k or control k depending on what system you are, and you can just start typing to go to wherever you want in the application. Because one of the annoyances I have with other GRC tools is it takes 11 clicks to get to where I want, you know, into this. And I wanna be able to navigate around and have kind of a universal search to, like, just start typing and go down and enter or, you know, click on it.
Joe:So basically, tell it what you want to do. Yeah. So I discovered this in Microsoft over the last couple years, but not before that. And it was such a pain to go figure out, like, how do I go and find the setting? You know how some of the Microsoft Office tools, you have to click that one little corner button and it pops up a new window.
Justin:Yeah. Clippy clippy needs to come
Joe:up and tell you what to do, and and then you kinda you go through that. And now in the search bar at the top of the window, you can actually say, you know, give it the command of what you want to do. Oh, right. And it will actually just do it Yeah. Right there.
Joe:And so you're I love that you're doing you're given the ability to quickly get things done Mhmm. Like, by telling it what you what you want.
Rick:Yeah. Like infinite use cases. Like, oh, a prospect a client prospect just asked me what my password controls are. And maybe there's like one sales guy at this company, you go, oh, well, if he has access to this thing, you just type password into universal search, and guess what? Yeah.
Rick:And they call come up. Like All the controls.
Justin:You can
Rick:do the thing.
Justin:All the issues, all the yeah. No matter what it is, like, I mean, we're feeding all the sources of that to make it sure it's, like, at least clean and relevant and waiting, know, into that. But, yeah, I mean, that those are basic things, you know Right. Into that. And when you experience, like, bad tools, you know, like, you know, if there's better ways to do this.
Joe:Yeah. So this will save you a lot of time. And I know you weren't looking to, like, actually sales pitch Episki, but we've talked about some stuff that's so exciting. How does somebody what what's the what's the website?
Justin:Yeah. Go to episki.com.
Joe:How do you spell it?
Justin:Episki.
Rick:Maybe you can
Justin:put it in
Rick:the show notes.
Justin:Yeah. And yeah. It's actually a lot of people ask me where I got episki from. It's actually a Greek root word episkopos. It's actually a biblical word, meaning elder or overseer.
Justin:You know? And I thought that was a good kinda overseer because you're ever seeing kind of the government's program and everything into that. And then I kinda shortened it and put an iandput.com, and that was available. So I bought, you know, type of thing. So Well, there you go.
Joe:So so the topic here today was launching a sash, and there's so many things we learned about what you should do and what to consider if you're going on this path.
Justin:Yeah. And I will say, some of the end stuff into this, I'll just wrap it up and put it, Cherry. Like, one of the things that we're you know, when you're deciding a business structure and how to put sales on top of it, one of the things is because we wanna appeal to smaller organizations, we couldn't really have a sales staff Yeah. You know, behind it because they go on commission. They expect decent commission checks.
Justin:And at a small price point, you can't really have Right. You know, type of thing. So the way we designed this was all kinda self self guided, you know, into that where you can sign up for a trial, you can play with it yourself, kick the tires
Joe:Yeah.
Justin:You know, then sign into it. And we have everything kinda automated at the back. So we have a in line chat where you can chat with, you know, myself, another person, you know, get you, like, what you need. It actually has, like, demos where it actually leads you through the application and say, you wanna add a new framework? Click on this thing right here.
Justin:Mhmm. Now click on this thing, you know, and it just kinda guides you onto So
Joe:it costs you nothing to go check this thing out.
Justin:For how many weeks? Two weeks. Two weeks.
Joe:Can check it out for a couple weeks.
Justin:And, like, if you need more time, you know, just Push the button. Ping us. You know? Yeah. Ping us.
Justin:We'll gladly give you more time. We just wanna put a little bit urgency, you know, behind it and not say, hey, you got it for three months and you forget about it. You know? And then, you know, we'll make sure we, you know, we look at a lot of data and the tracking of, hey. Have you used this feature yet?
Justin:I'm not like, hey. Here you should use this feature. Here's how to use it. You know, we'll send you reminders, email, and chat, and all that stuff and everything. But, yeah, our goal is to kinda get it very kinda easy from an onboarding perspective.
Justin:Just click some buttons. You got the the framework of choice. You're into there. If you need to upload stuff, you just drag and drop it, all that things. And eventually, you know, I I think I you know, we've talked about this.
Justin:Getting other companies to actually help you kinda manage
Joe:the company. To go there. So if so if if you're a company and you're small and you're a startup and you're getting pushed to get your compliance in order and you need a tool and you can't afford to buy an expensive tool, but you can't afford to go hire somebody, you have a choice. You go hire that full time engineer or you and you probably don't need a full time engineer or you can get some fractional help, is exactly the kind of stuff Rick does and the stuff I do. The stuff you're still doing and you're working Yeah.
Joe:Your way out of
Justin:Exactly. Yeah.
Joe:And so you're looking for folks like us. So, you know, start with a PISCY and figure it out. See if you can do it yourself. And if not, call one of us up Yep. And we will put some engineers on this at a very affordable fractional price
Justin:Right.
Joe:To run your GRC program using this tool.
Justin:Yeah. No problem. And that could be, like, just get you set up and kinda, you know, in a good state or ongoing, you know, which it usually turns into that because people realize, like, oh, yeah. There is a lot of work here that I don't typically wanna do.
Joe:As an entrepreneur running a company, the last thing I wanna do is stop doing the things that are going to get me new customers, make my current customers more successful. Yep. I wanna be out actually, the part I still love to keep doing is VC so stuff. Mhmm. I love to keep talking to the top leaders in the companies.
Joe:Yep. Helping them understand what to do. And every minute that I'm spending doing bookkeeping something else is a minute I'm not helping them succeed at the thing I love to do Right. And they want me to help them with. Mhmm.
Joe:And so I'm not a bookkeeper. I I've outsourced that.
Justin:Yep.
Joe:And my accounting company is doing a fantastic job of keeping that going. And at first, I thought, wow, this is expensive. I'm paying this much a month to have these people do this stuff. And now I'm like, it's done and this is done. Perfect for
Rick:your time. Yeah. Yeah.
Joe:Oh my gosh.
Justin:Yeah. Guess I went through a few companies. I, you know, I shared with you some of the, like, the bad, you know, stuff I went through. But now I have a company of, like, every single week, I get a report. You know, I get end of month, you know, like, reporting and everything.
Justin:And that's, you know, that's the biggest thing if you're are like, we got this question last night about starting your own company and why'd you do that. Oh, yeah. And, you know, a lot of us are freedom. We wanna set our own culture. We wanna, you know, basically, you know, fail or succeed on our own merits, you know, into that.
Justin:But you're absolutely right. Like, that doesn't mean I wanna do every aspect of the company. You know? In fact, as soon as I can afford it, I'm pushing it off to somebody else, you know, type of thing. Like, the only reason I'm still doing it is because haven't found the right person or I'm, you know, not that state where I
Joe:can Right. Know? ROI is not quite there yet.
Justin:Exactly. You know? Like, the doing the the the podcast here, you know, like, Buzzy, like, I did the cutting Oh, yeah. Of our first episode. I'm like, nope.
Justin:Yeah. Yeah. There's people off the
Rick:somebody to do this. They do the perfect thing.
Joe:Can I have
Justin:a little bit more before we Absolutely?
Rick:Cheers. So, Justin, while
Justin:we're pouring
Rick:that, what's been the most the the most satisfying moment or, like, the thing that makes you happiest about having started on this journey with a PISCY?
Justin:I mean, a ton of stuff. It's you're talking specifically with SaaS or
Joe:What makes you happy about the decisions you made to build a product?
Justin:So I like being my own boss, know, minus all the the minutiae of stuff.
Joe:I can see that because I don't think I could ever have hired you and manage you.
Justin:Yeah. I know. You'd be impossible. I'm pretty difficult and everything.
Joe:I don't
Justin:think so. You know, other people say I am. You know? Oh, you
Rick:have you have strong and well reasoned opinions.
Justin:Yeah. I mean yeah. I at some point, we could should go over some of my firing layoffs, you know,
Joe:and everything.
Justin:But I stand by all the facts of the case, you know, in in that. Like, I
Joe:You're much easier to work with as a peer. Yeah.
Justin:I'll tell that. Yeah. Yeah. But, yeah, I I'm very opinionated. Not that I step into anything.
Justin:Like, I've never, like, I'm not gonna blow up in front of a customer. Like, that's taboo, you know, that type of thing. But I will pull an executive aside and be like, what you're doing is wrong. You know?
Joe:Oh, and I love that transparency. And I actually encourage every person on my team to tell me that I I had a really good candid conversation with a customer last evening about what we could do better on this stuff. And I got great feedback, and I super appreciated it.
Justin:See, I've gotten fired over that. You know? No.
Joe:Where I've like world, that one happened.
Justin:It was, you know, some, you know, sexual and I'm not not getting anywhere to the company, but, like, sexual improprieties were going on. Everybody's rumoring on the back end, and I pulled the the the executive aside. We're like, hey. Like, you're ruining morale in the company. You are ruining it, you know, in here.
Justin:And a month later, was fired, you know, into that. You know? Like and and that's the thing. It's like, I'm not afraid to do that because, like, if I do get fired, okay, I'll find another, you know, like, type of thing. And now I don't have to, you know, I just have to, you know, critique myself on how Right.
Justin:Right. Bad I'm doing.
Joe:So, yeah, independence. That's that's one
Justin:of the
Joe:things of running your own company. Yeah. Well, that's great.
Justin:But, yeah, I say, you know, from a, you know, a a happy spot, like, I would love, you know, to hit some of the marks with a pissy. You know, get this ass out. We just started advertising for like, we got a whole bunch of leads about last year. We paid for marketing. And we I just got my stat, I think, late last week or early no.
Justin:Early this week and everything On some of our stats and everything, we put in about a thousand dollars from Wednesday to Wednesday. For, like, for advertising? For advertising on Google. We're gonna also do to LinkedIn, maybe a little bit less than a thousand. We were getting I just got got the report, 34% click through rate.
Justin:Nice. Which if people aren't yeah. Exactly.
Joe:That's super high.
Justin:So that's and if people aren't familiar, so it's kinda broken out into impressions, which is what Someone looked. Somebody saw it on their web page Yeah. You know, type of thing. And then click through rate is they looked and they clicked. So broken down between somebody searched for GRC tools or something like that, and we're bidding on good keywords.
Justin:And I have a an excellent SEO girl that's like, she's phenomenal, you know, and everything. But with this, we critique, like, we're running, like, kinda three ads with the number of keywords and everything. We're getting a 34% click through rate on first, and it's still kinda ironing out. It's super high. Right.
Justin:Like, people will be happy with 5% Right.
Joe:Yeah. That's amazing.
Justin:With this. And now, like, I'm redoing a little bit of the marketing website to make sure we're one, we're catching all the AI keywords because, like, just saw a stat the other day, like, 30% of Google searches are done by, like, ChatGPT and all the AI Yeah. Things. So they're basically going to ChatGPT for their initial search for anything. Yeah.
Justin:You know? Oh, yeah. It's only it's the new Google. Yeah. So but you have to now craft your website in a certain way to then get the answers to what they're asking.
Joe:Yeah. I've had a couple customers recently tell me that they found us because they found us through asking ChatGPT for help on getting the various frameworks that we support.
Rick:Fantastic. Was awesome.
Justin:Yeah. And it goes to, like, a lot of the content, but you can also format your content to be more AI digestible. Friendly. You know, type of thing. They recommend, like, almost like FAQ type stuff, you know, like so, like, the question that's asked and then you answer it, you know, type of thing is, like, very AI friendly when they can see, a question answer aligned directly with that.
Justin:But anyways, like, we're we're redoing it to get more conversions into that.
Joe:Oh, that makes a lot of sense.
Justin:And more content. But So
Joe:as so as we wrap this up, what kind of pitfalls would you say that people should if they're gonna build a product, they wanna start thinking ahead. Like I said earlier, I wanna know what my end state's looking like so I know where I'm driving to.
Justin:Yeah.
Joe:Knowing what pitfalls to avoid as I get there is a super powerful thing.
Justin:Yeah. What would
Joe:you suggest?
Justin:So I would say, you know, if you can develop fast, fail fast, you know, into it. Try to get a product as soon as you can into a customer's hand and start getting feedback out. Like, that was never a problem for me getting feedback, you know, but it's taken so long without actually have that. Like, the the full time developer I first hired, he started August, you know, into that. So a little over a year.
Justin:And since then, we've been putting in tons of features
Joe:That's awesome.
Justin:Night and day, you know, from where it was to where it is today. Yeah. And I I wouldn't been able to got there without him, like, coming on board and helping me and focusing on it day in and day out, you know, into that. That was a realization of my failure trying to straddle, you know, the businesses into that. The other thing is get a good feedback loop, you know, into this once you have a product and you get some feedback with customers.
Justin:Obviously, like, a lot of that may be, like, manual, like, doing conversations with customers on kind of your first dozen, two dozen, three dozen, whatever it may be. But you then need to fit figure out how to get a good built in feedback loop to address stuff, you know, pretty regularly and make sure it's part of the, like, build process. One of the things that I just shared with my developer and and I love, we use a a Jira like software development product called Linear. Oh, yeah. And they do a lot of good stuff out there, but they actually recently did a blog post on their zero bug policy.
Justin:So all bugs don't go to the backlog. They get assigned in the current sprint. And they either deal with it then or they dismiss it and say, we're not gonna ever fix it.
Rick:We're never gonna fix it.
Justin:Yeah. So it might be, like, very small. Does you know, like, it's, you know, affects maybe one user. It's an edge case that it's like, it it's just not worth it, you know, type of thing.
Rick:That's a really interesting
Justin:But they have to decide Yeah. Never fix or this sprint. Like, that's the choice that it comes into it. And Now clearly, are some things that are, like, structural. Like, this
Rick:is gonna just take longer, but does it it starts on
Justin:this sprint? Bug.
Rick:Feature. Okay. Okay.
Justin:Yeah. Right. So Beating a
Rick:bug if it's, like, architectural systemic.
Justin:Yeah. But usually bugs are like something's not working as expected. Okay. Fair enough. But, you know, they had to before they actually implement it, they had to clear out their backlog.
Justin:They had a whole bunch of backlogs, you know, into it. So close it, fix it, whatever it is. They had to get down to zero, but then they incorporate it into their normal process. And, like, then there were show show co like, on Twitter, like, hey, I reported this bug and it was fixed within a day.
Rick:That's wild.
Justin:I know. And there was customers, like, attesting that. Like, this is phenomenal.
Joe:Because either do is do it now or do it never.
Justin:Yeah. Exactly.
Rick:That's super cool.
Justin:Yeah. And it's just like that customer experience, that's what you wanna kinda
Joe:It has like zero inbox version.
Rick:Yeah. Inbox zero, bug zero.
Justin:That's the stuff I love, like, and getting into that customer.
Joe:So have you adopted that?
Justin:We're getting there.
Joe:No. That's
Rick:super that's Yeah. That is really I
Justin:mean, we don't have a ton of customers right now, so there's not a lot of bugs being reported with that. But we do have monitoring software where we're getting bugs in as we experiencing them, and we're actually triaging them and then actually fixing it, you know, into that. Wow. So yeah. But that's the thing.
Justin:It's like, you know, eccentric customer focus. Like, if they're reporting something that's inter interfering with that, you know, then let's get on the thing. And I'm not talking about blow up your, you know, your road map every time a customer is like, I really like if you add a button to do this, you know, type of thing. You do have to pass it through kind of a But
Rick:to your point, bugs not features.
Justin:Right. Exactly. A bug is something like, it should work
Rick:like this.
Justin:Thought it did work this way. You know, that
Joe:type of thing.
Justin:But a feature is you got a screen of, like, is this something we want to go down the road with? Does it make sense, you know, at this time? You know, there's a whole bunch of kinda considerations when you're building out your road map. Are we putting this ahead or keeping it at the behind from, you know, prioritizing this type of thing. But, yeah, just being, you know, kinda attuned to where Yeah.
Justin:You know, the majority of your customers are, you know, is really where you're gonna succeed. And the bottom line is, you know, like, anybody who's trying to build, like, a company, solve problems. That that's the end of the day is
Joe:your problems.
Rick:You know? I don't wanna keep bringing it back to Ed Norton, but he said one thing that I which stuck with me since then. It was like the the the desired state isn't profit. The desired state is providing a solution that's so necessary. Profit is the natural outcome.
Justin:Yes. Oh, I love that.
Rick:Loved it's just it's never gonna leave my brain.
Justin:Yep.
Joe:Yep. Good It's way to think about it.
Justin:Yeah. So, yeah, if you're on the idea, like and when we're finalizing this and looking, you know, started to push it out, we're looking for wow moments. You know, it's
Joe:like Yeah.
Justin:When will the customers say, wow, this is exactly what I'm looking for or wow. And building a use case around those moments and then making sure all those features
Joe:So in the future, we'll be able to see go to your website and see the wow moments place.
Justin:The Goodyear?
Joe:The Wow Moments place on your website?
Justin:The Wow Moments? Yeah. Yeah. The highlight reel. It's the highlight reel.
Justin:Yeah. Kind of. Oh, I totally wanna go to all the Pisky.
Joe:So here's the future URL. At pisky.com/wow. Mhmm. You go there, you'll see the best stuff. Yeah.
Joe:Yeah. This is what you'll expect to get when you use this tool.
Justin:Yeah. So yeah. Just the stuff that's like, you know, like, you go in there, you click a button, and you have all these frameworks. Or you go in there Yeah. With some of the stuff we're building with, like, automation, and you get all the tasks suggested to you just bring in a framework there.
Justin:You know? Those are the wow moments like, wow. This is really easy. Yeah. Wow.
Justin:This is all in front of me. You know? Like, now I have a complete picture. Like, and then you build off of that story and then kinda work behind it like the Steve Jobs thing. He's like, you get kinda the end customer experience and then work backwards on what features need to be there to support that, you know?
Justin:Alright.
Joe:So go to opiski.com and check it
Justin:out. Sign up. Absolutely free. You don't even have to drop a credit card to try it.
Rick:I love it. Hey. That's awesome. Cheers.
Justin:Subscribe. We actually listen to your comments here, so don't forget to say if you like the episode or if you have suggestions for us for next time to talk about. And cheers. Have a good one. See you, everybody.
Justin:Bye.
