Episode 23: Nobody read the report
Today on the episode, the Delve scandal blows the lid off fake compliance as a service, and we break down what it means for the certification industry, the future of compliance automation, and the third party risk management. Plus, a special guest joins us. This is Distilled Security Podcast. Somewhere
Speaker 2:right now, someone is digging through folders trying to find the right version of an evidence file for the third time this week. Controls scattered across a dozen system, owners who left the company months ago, due dates that already passed, nobody noticed. It's how it's always been done, and somehow, we just accept it. There is a modern, better way. Episkey.
Speaker 2:Visit us at episkey.com/dsp for a special offer.
Speaker 3:If you're leading a company today, you're not short on ideas. You're buried under them. More tools, more frameworks, more initiatives that were supposed to help and somehow made everything harder. Minus Partners exists for leaders who don't need another thing added. Minus Partners helps leadership teams remove friction that's been normalized over time.
Speaker 3:So execution gets easier, not harder. Less noise, clear decisions, faster
Justin Leapline:Welcome back to Distilled Security Podcast. Thank you for joining us. Before we actually dive into some of the topics that we have today, which is an excellent episode, I'm really looking forward to this, We have Matt Chavone. Matt, thank you for joining us, man.
Matthew J. Schiavone:Thanks for having me.
Justin Leapline:So to start off with, get the audience to know you and everything. Why don't you introduce yourself? Where are you working now? How you got into the industry?
Matthew J. Schiavone:Oh, yeah. So my name is Matt Chavone. I work for Sikich. Sikich is a top 25 CPA firm. I lead the third party attestation practice.
Matthew J. Schiavone:We focus primarily on SOC two, SOC one, ISO, CMMC, and PCI.
Justin Leapline:Oh, okay. All the certifications?
Matthew J. Schiavone:Most of them.
Justin Leapline:Yeah. Yeah. We we try. We try. Gotcha.
Justin Leapline:So what got you into this? Have you always been doing SOC two, SOC one, ISO?
Matthew J. Schiavone:I started my career actually in the Department of Defense. Oh, okay. I'm a CPA, so got into internal controls mostly around financial reporting. Later in my career shortly into my career, got into more security related controls. And 2012 is when SOC two was released and got into SOC two a little bit shortly thereafter.
Matthew J. Schiavone:Okay. It didn't wasn't popular right after
Justin Leapline:'70 and all this stuff. They started splitting it out and everything. Got it. Yeah. Yeah.
Matthew J. Schiavone:It snowballed from there. I got introduced to Joe. Joe got me on got involved in ISO. Okay. That was, like, 02/1716.
Rick Yocum:Something like that.
Matthew J. Schiavone:Yeah. Yeah. So and then just kept snowballing from there.
Justin Leapline:Okay. Great. And how big is your team that you do all this stuff with?
Matthew J. Schiavone:The risk advisory practice as a whole is about 60 people. Okay. And then across PCI, CMMC, anywhere from 15 to 20
Justin Leapline:Okay. So the certification. So decent size. You do a lot of them every single year and everything. How many clients roughly, you know, do you do a year?
Matthew J. Schiavone:We do
Justin Leapline:A lot. Yeah. Yeah. Yeah. Now do you promise SOC two in a week?
Justin Leapline:Most of
Joe Wynn:the time.
Justin Leapline:Yeah. Right? Yeah. Yeah.
Matthew J. Schiavone:Or sooner.
Rick Yocum:Yeah. Yeah. That's only for startups that have no security whatsoever.
Matthew J. Schiavone:Yeah.
Justin Leapline:Right. It's easy
Joe Wynn:to fail a SOC two.
Justin Leapline:So yeah. So, yeah, really appreciate you joining us and everything, and I think it's absolutely fitting for what we wanted to talk about mostly here. So to let everybody know, if you haven't heard about Delve, Delve is a GRC automation tool. They've been I don't know. It's only been about five years they've been on the market here.
Joe Wynn:I actually think it's much less. I actually looked at this right before we got here. I think it was they went through Y Combinator in, like, 2024.
Justin Leapline:I thought it was earlier than that. Okay. Yeah. Either which way.
Joe Wynn:Yeah. Not long Very new
Justin Leapline:on the industry, all that stuff, and they got a reputation. And I don't know if you guys you you follow Troy Fine and everything. He he's great on, like, blowing up people that promised the world in what we know is unrealistic timelines ever working with any client whatsoever, you know, type of thing. But these were one of these companies that were known for sock in a box. You know?
Justin Leapline:The quick compliance, if you just need a SOC two, we got you. Just pay us $510, whatever it was, you know, type of thing. Well, about two weeks ago, week and a half ago, I think it was just a week and a half ago, two Fridays ago with this, they got breached a little bit. I don't know the particulars of the breach, but essentially, a a Google Sheet got released out with all their clients and linked to documents with SOC two, ISO, and SOC ones into there. And somebody was it Delve die deep deep delver?
Justin Leapline:Delver or delvediver, something like that. Deep delver. Deep delver.
Rick Yocum:Yeah. Substack article. Right.
Justin Leapline:New new publisher. This is all they've actually published into this. They went through and did a pretty big hit piece onto them. It it was detailed. Their part one went through all the stuff.
Justin Leapline:Yeah. It went through their reports. It went through how they were basically copying and pasting stuff, and all the reports are, like, available. I actually went in and downloaded all of them, and you could actually see all the reports. And they put it up into publicly downloadable format and all that stuff.
Justin Leapline:It's really interesting, you know, and you start looking at it. And a lot of it is they're SOC two reports. So, you know, and some of the ISO. But they pull out that it's basic copy and paste, you know, into that. They what were some of the other things they had?
Justin Leapline:It was about 500 clients, and some of them were known customers into their, like, Lovable. I don't know if you guys know. Right. Yeah. They they do that.
Justin Leapline:Their name was in there. Bland, Cooley, Eleven AI. Like, some of those I don't know, but a lot of them were just, you know, listed in here. And some of it was, like, garbage stuff. You can tell they're internal only into that.
Justin Leapline:They also not only they looked at the reports, they also looked at who's rubber stamping these Yep. And followed up on some of these companies, and it was it was interesting to say. They say even to, like, Google mapped, like, some of the companies are like, what's their location in Pakistan? Oh, yeah. It's some right above a marketplace, you know?
Joe Wynn:Right.
Justin Leapline:Was there a headquarters and all this? I mean, it was pretty big. I guess, you know, first off, before we dive too deep into it, you guys' opinion on this, like Well,
Rick Yocum:first, full disclaimer. This is a person doing a write up. There's been no independent verification that this is that. Everybody that that keeps seeing responses sees Delve saying this isn't true. We're just talking about the article.
Rick Yocum:Yes. Talking about the claims that there
Justin Leapline:wanna get sued. Yeah.
Joe Wynn:So retroactively and henceforth in this podcast, just assume we say allegedly before every Just
Rick Yocum:just assume that because, you know, until until this is verified as proof, we don't know. Yeah. But that's a very well put together piece of research that somebody did. You can click into the stuff like Justin Yeah. Was mentioning, and you can kinda walk through the process and the person puts together quite a convincing story.
Joe Wynn:Pretty compelling.
Justin Leapline:I mean, it's least it's enough to deserve a response. And I don't think Delv has like, there's a couple of LinkedIn posts that where their CEO, like like, we're coming out with a bigger response, but it's not really true. Like, some of the like, he he basically said, like, the reports weren't, like, finalized yet or something like
Joe Wynn:I think I saw a slightly more formal response. Did you? Okay. That more or less shifted the blame to the CPA for, oh, we just do software.
Justin Leapline:Yeah. Yeah. That's right.
Joe Wynn:It's a CPA. It's their fault. They didn't look enough. Yeah. So we we'll get into that.
Rick Yocum:Yeah. And even looking past all that, one of the things I'm excited to talk about tonight is how does reliance on evidence really matter and make a difference, and how do you get something you can trust. So when we leave here tonight, I'm hoping that we have some good takeaways for people for what they should do when they're looking at this stuff and when they're evaluating software and auditors and even vendors that rely on these audit reports.
Justin Leapline:Yeah. So I guess, first off, I mean, Matt, seeing this, did this surprise you?
Joe Wynn:Absolutely not. Oh, no. That's a terrifying answer, actually.
Rick Yocum:Why not?
Justin Leapline:Because of Delve specifically or something else? Both. Okay.
Joe Wynn:Did you know of Delve before this?
Matthew J. Schiavone:Yes. Okay. Well, only through I had a few client interactions where two noteworthy. One lost the SOC two client in December because they were moving I can't say whether or not it was Dell, but it was a very similar situation.
Justin Leapline:Okay.
Matthew J. Schiavone:This company promised them, this is in December, their SOC two report by their deadline of May March 31. So already having not They already
Justin Leapline:signed up with you?
Matthew J. Schiavone:We were in the midst of renewing this year's engagement.
Justin Leapline:Oh, okay.
Matthew J. Schiavone:Things were really slow, and I knew something was kind of a kind of a miss.
Justin Leapline:And everything. But they've already had a SOC two through you guys. They have. Okay. That They were looking at just cheapening it, I guess, getting a discount on doing that SOC two.
Matthew J. Schiavone:Well, I Yeah. For less than our SOC two, they got a pen test, ISO certification
Justin Leapline:Oh.
Matthew J. Schiavone:And vCISO services. And I questioned, you don't even have an ISMS yet in December or a statement of applicability, and they're promising you an ISO certification.
Joe Wynn:That deadline was also by March. The ISO March 31.
Justin Leapline:Yeah. Okay. Okay.
Matthew J. Schiavone:Didn't know what a statement of of applicability was. So nor did they care. What they cared about was they can have Give us a stamp. Everything by March 31. Oh, that's
Justin Leapline:that's today. Of a price.
Matthew J. Schiavone:That's I should have checked in to see if hey, guys.
Justin Leapline:How's that going? We remote Zoom in, you know.
Joe Wynn:Another guest is joining us.
Matthew J. Schiavone:Yeah. So how I mean, how how many ISO programs have you stood up in three months from initiation to certification?
Justin Leapline:Not to certification. You know? I mean, remediation, yes, if it's most of the way there. I mean, documentation is usually the biggest lift. You
Matthew J. Schiavone:know? Stage two.
Justin Leapline:Oh, it's stage two. Yeah. It still takes a while.
Rick Yocum:No. Everybody I talked to that has a fairly sophisticated no. These aren't startups where you have one application, one very small ISMS Right. And really writing security policies. In that case, we would never just rubber stamp them and just pull out a template.
Rick Yocum:But, you know, they don't have to be super complex because the organization is simple. But any organization that has couple 100 of peep couple 100 people are selling to health care and financial services, those kind of organizations, they're, you know, I'm like, look, let's just set a timeline of eighteen months to figure out where we're gonna be at, and then we'll start shortening it.
Joe Wynn:Right.
Rick Yocum:And we need to see what
Justin Leapline:you have. Right. What scope you're setting?
Rick Yocum:Yeah. What is you what's in scope? What do you wanna be compliant for? If you're putting an ISF MS in place from an ISO's perspective, you're gonna do more things than I think you even have to do just to hit all the SOC two stuff. And so maybe along the way, if that is the goal is to get both, you know, we can probably cross the SOC two finish line a little bit sooner, at least get to a type one situation where you can have your, you know, your your your design down and test of one in place.
Rick Yocum:And then after that, let's keep going and being able to show that you have have the stuff in place. But three months is that's typically a timeline and given these more mature, more complex companies just to do the first gap assessment Right. Lay out the remediation road map and get some traction on it.
Justin Leapline:Or as you pointed out, a very simple scope. Like, if they have one web app in Vercel or something like that, you could you could probably push through that, you know, type of thing. You still have to do a gap, but if you're not doing any infrastructure stuff and all you're doing is coding and policies and stuff, you could probably get them ready, you know, at that point.
Rick Yocum:Oh, yeah.
Justin Leapline:You know?
Rick Yocum:Quicker than that. Quicker.
Justin Leapline:Yeah. Exactly.
Rick Yocum:You'd probably be in a good situation in, like, two to three months, and then you get all that evidence burn in.
Justin Leapline:Yeah. Exactly. Yeah. So but, yeah, the the the promises and I think we talked about this. We had a talk that we did for ISAC a few weeks ago.
Rick Yocum:Oh, yeah. That was huge. Plus, it was free drinks if you missed it.
Justin Leapline:Yes. Exactly. Yeah. We're sponsoring with that. But one of the things that we talked it was actually on point here.
Justin Leapline:It was actually before this delve incident, you know, that came out. But we were talking about how the incentive structure is basically set up to the grade, you know. Right. You got these companies that they're trying to get bigger deals that the bigger companies say, well, not without a SOC two or ISO certification. So what do those companies look for?
Justin Leapline:They look for the cheapest way to get a SOC two or ISO certification. And without a, basically, a an anti incentive, like, quality pushing it up, you're gonna get it to the grade to the worst possible point, you know, into that. And I think that's we've all known it for a while. You know? You know, you've seen it, you know, with these players are like, one week suck, you know, or your money back.
Justin Leapline:You know? I've seen that a lot with some of the players and everything. But, yeah, it's it's an issue. It's an issue, like
Joe Wynn:Yeah. The incentive alignment issue has been around forever, particularly with, like, third party reviewers. Right? I mean, like, the whole Enron thing and socks came out of that. Yeah.
Justin Leapline:Yeah. In fact, I was recently I have a buddy going through a cheap provider. Mhmm. And I I think I told you this. The funniest thing, they sent them, like, the evidence request packet.
Justin Leapline:And one of the things that they had in the policy, they're like, if you don't have your access control policy, just copy and paste this into a chat GPT prompt.
Joe Wynn:Yeah. Here's your prompt to enter
Justin Leapline:into your prompt
Joe Wynn:Yeah, Jenna.
Justin Leapline:To get the policy out. It was like, I've never seen an evidence request. Yeah. Yeah. It was like, oh, wow.
Justin Leapline:Yeah. Yeah. I mean, we we do that, you know, but you you don't say the deal.
Rick Yocum:I wanna go back to something we talked about just a couple seconds ago, which is, you know, how did this all come about? And I I can remember, you know, what is it? Like, fifteen, twenty years ago. And we're just starting to get into the parts where you really need to start you should always have been vetting your vendors. But it became a little bit more mainstream.
Rick Yocum:I remember this in in that time frame. And the things we would do is we would create these
Joe Wynn:Oh. Big
Rick Yocum:questionnaires. Who who's the big questionnaire maker?
Justin Leapline:SIG. That was SIG. The SIG Lite. Yeah.
Rick Yocum:SIG Lite. And so you get the SIG Lite. You'd, you know, you'd put that together or you make your own list of questionnaires or questions for the questionnaire, and then you'd you'd make the high, medium, and low risk vendor versions of it and get that out, and it would come back. And you'd be spending hours and hours, even days going through these, and then you'd have all these vendors you're doing this for. And it got to the point where you're like, how can we sustain this?
Rick Yocum:This is a lot of work. We have to hire a lot of people in order to review all this stuff, and some companies even still would do that once they got a SOC two audit Mhmm. Attestation report
Justin Leapline:Yeah.
Rick Yocum:Or saw an ISO report. But you'd get the reports and you would say, well, how many of my questions can I not ask now because I can rely on this report? Right. And so if you start thinking about, oh, that's that's so good. It's gonna make my life so much easier.
Rick Yocum:Can I get the report? And then you get the report, and then you start giving it to somebody who would say, hey, do you got a SOC two report? Yes. Alright. Checkbox.
Rick Yocum:Did you did you did you read it, analyst? Right. Oh, we have to read those.
Justin Leapline:Yeah.
Rick Yocum:Yeah. Yeah. You gotta read those. You need to know what's inside of them. You need to know what the controls are.
Rick Yocum:Do the controls in the application that it's even referring to? Is that the one you're buying?
Joe Wynn:Yeah. Is the scope even aligned?
Justin Leapline:Yeah. Exactly. Yeah.
Rick Yocum:Yeah. And then my favorite part is skimming through and reading through the control and the testing at the end to see were there any exceptions, and then going and following up on those. So all of a sudden, you had your own list of questions, and then you also got the SOC two report that you also had to read, and then you went through and you had a follow-up on the exceptions. Now, I just created more work for my analysts because we weren't sure that we could stop asking the questions in the questionnaire if we could get the vendor, if they were you know, we were important enough customer that they would fill out my questions and pay and get us the SOC two report. At the end of the day, who is the actual customer of these SOC two reports?
Rick Yocum:Just the person who's not paying for it. It's the person who's receiving it Mhmm. Because they wanna val vet that the Yeah. Service provider is actually strong enough to to support them. Right.
Rick Yocum:And so it got all the way through that and now we have these reports and we're up to what we do now in the industry Right. Which is what the So you thousand dollars report.
Justin Leapline:Or worse than it was?
Rick Yocum:Oh, think it's worse.
Joe Wynn:It's way worse. It's way worse. Well, because it's It can be way worse. Yeah. And it's it's the illusion.
Justin Leapline:Just be careful. We're attacking Matt's all, you know, service line here.
Matthew J. Schiavone:I can defend it. I, you know, I hear it all the time. It's all over LinkedIn, SOC two criticism Yeah. Yeah. Poor quality, lacking framework and security.
Justin Leapline:But I don't think it's necessarily the the framework that's bad. It's the enforcement of the quality.
Joe Wynn:Right. Because when it's bad, it's the illusion of security or the illusion of compliance. And it'll trick people that might not be savvy enough or you're an entrepreneur that has no idea about any of this stuff, and you're just trying to figure out who to rely on. Right. And you do you ask generative AI, what should I look for in a vendor?
Joe Wynn:And it says, oh, make sure they have a SOC report, and you do that. Right. But you don't know, you know, how to dig into that necessarily. So I think the illusion of it, into your point, that comes from potential issues with quality, right, or the lack of enforcement Right. Of quality.
Rick Yocum:Yeah. Let's maybe we talk a little bit about unless it's coming later, about how do SOC two auditors how are they reviewed, and what's that peer review process like?
Matthew J. Schiavone:Yeah. So the peer review process, CPA firms have to undergo peer review once every three years. Okay. And SOC two, SOC one is considered an a high risk area. So peer reviewers love to drill in.
Matthew J. Schiavone:And, basically, when peer review time comes, you give the peer reviewer a list of all your SOC two engagements over the past three years. How does that
Justin Leapline:work exactly? So AICPA pairs you up with a peer or you seek out another peer?
Matthew J. Schiavone:Seek out your own.
Joe Wynn:Okay. Oh, I didn't
Justin Leapline:I know that. Then you have to basically do an engagement to say, here's my total population, and they do a sampling, and it'll cost, I don't know, $20, whatever the the Depending on the sample, how many binders they have to review. Gotcha. So they you engage somebody to audit you, and it's tracked by AICPA. They do they submit it back to AICPA when they're done and you or you?
Matthew J. Schiavone:And now there is well, we'll probably get to this, but the AICPA has an advanced oversight committee for SOC two because of some some of these problems.
Justin Leapline:But who submits that? Is it you that submits that? Or Peer review. The peer review submits it directly to AICPA.
Matthew J. Schiavone:And then on the AICPA website, you can go see letters of acceptance or Gotcha. Yeah.
Justin Leapline:Yeah. Then they hand it back down to you and say, pass fail findings, whatever. Here's your remediation plan if necessary, all that stuff. And one thing I was reading with the Delve, you have the option to make that public as the person that got reviewed. Like, they have a website that shows reviews, but you don't necessarily have to
Joe Wynn:So the reviewee can publish their the results of their peer review? Yes. But they don't have to from what I read.
Matthew J. Schiavone:Don't think they have to post the letter of acceptance. I think the results pass or fail have to be and I should I should know this a little bit better, but I think the results are automatically posted, but the letters are optional.
Justin Leapline:Oh, okay.
Joe Wynn:And the letters have,
Justin Leapline:like, some additional detail about the whys around passing or failing? The management like, the management response.
Matthew J. Schiavone:So if a CPA firm underwent peer review, if they post one of their letters of acceptance, letters review, it'll, like, it'll say SOC two or SOC one was included in this peer review. Right.
Justin Leapline:Got it. Okay. And then they'll have three months to fix and all that stuff and everything. Right? Yeah.
Justin Leapline:Candidly, I don't know. I've never So
Joe Wynn:I was this is actually related. Again, don't wanna jump too far ahead, but I know the the CPA involved in this had just failed an AI CPA period.
Justin Leapline:Yes. In January.
Joe Wynn:In January. Yep.
Justin Leapline:And so I You can actually go look at that on the portal.
Joe Wynn:Yeah. So I was curious about and and maybe you know or maybe I we need to look it up. But I was curious, like, what happens when you fail? Because I was surprised that they failed in January, but then continued to issue reports.
Matthew J. Schiavone:I didn't think you were able.
Joe Wynn:That would have been my assumption.
Matthew J. Schiavone:Yeah.
Justin Leapline:Yeah. So I got that details for Delve specifically. So they went through a peer review in January of finalize. Jason F. Claus PC was the one that did it.
Justin Leapline:And quote unquote That was a peer reviewer? That was a peer reviewer. Yeah. Found, quote, unquote, pervasive deficiencies across all engagements reviewed. No isolated incidents.
Justin Leapline:And some of the specific findings, there's no serviceable assurance that SOC engagements were planned and performed in accordance with professional standards. There's actually a quote out there. Controls were not suitably designed, appropriately tested, or supported by sufficient evidence. Engagement teams couldn't demonstrate understanding of the systems. They were supposedly examining independence violations.
Justin Leapline:Threats to independence were not identified, evaluated, or documented. Oh, yeah. Monitoring process failed to catch any of these before the reports were issued, and deficiencies were described as pervasive. Yeah. I've already said that.
Justin Leapline:The corrective action by 05/31/2026. So essentially, what is that? Four months? You know? Four or five months?
Justin Leapline:You know? Depending on where you count. All staff must complete sixteen hours of SOC CPE training. Preissurance review required for their next SOC one and SOC two engagements. Must remediate all nonconforming engagements.
Justin Leapline:Must submit evidence of remediation to an independent reviewer. That's what the
Rick Yocum:And just to clarify, that wasn't Delve that was being, audited here. That was a different auditor. Right?
Justin Leapline:No. It it was against Delve, but the this is the auditor that audited Delve's engagement.
Joe Wynn:The peer reviewer was reviewing Delve's program, and those were their findings.
Matthew J. Schiavone:Like Delves auditors.
Rick Yocum:It was Delves auditor, not Delves themselves. Delves was working with
Justin Leapline:Oh, yeah. Yeah. Yeah.
Rick Yocum:Yeah. Just for clarification. Yeah. I see. Delves isn't a CPA firm, but the CPA firm that was issuing these were peer reviewed by the by the person you mentioned.
Justin Leapline:And have you heard of the so the order was a corp? C c a c corp. Yeah. A c That was o r p.
Matthew J. Schiavone:One that had the adverse finding.
Rick Yocum:Yes.
Joe Wynn:Yeah. That was the the firm.
Justin Leapline:Have you heard of them before? I haven't heard. No. So, yeah, they're the one that I forget where they were located. I thought it was, like, Pakistan or something like that.
Joe Wynn:You know? I dug into when I was looking at the numbers, I dug into the the the note about independence a little bit. Yeah. Yeah. And I looked up AC Corp, and so they they claim, you know, around 700 ish clients.
Joe Wynn:Mhmm. And they were I used the vast majority of the, like, 485 leaked reports they were the stamp on. Okay. And so and it was an eighteen month period, so, like, not a ton of those are gonna be repeats necessarily. So if you have 700 plus clients, right, and you could practically speaking probably three to three fifty of those as a lowball Yeah.
Joe Wynn:Is all through Delve as one source of revenue.
Justin Leapline:Right.
Joe Wynn:Yeah. You're probably not terribly independent. That's a ton of concentration risk.
Matthew J. Schiavone:Yeah. I would be very curious to see how many partners signing these reports A Corp had because if you figure 700 reports in what? An eighteen month period?
Joe Wynn:Yeah. I well, 700 clients, it was their total claim, but it was, like, 500 ish reports in an eighteen month period, and they said they have, like, 55 staff. Do the math. Oh, yeah.
Justin Leapline:Do the math. Do the hours.
Matthew J. Schiavone:I mean, 700 reports for one or two partners. I mean, it takes more than a few partners to actually review the engagements Mhmm. You know, sign the reports, with credibility. Yeah. So I I don't know how many partners signing the reports they had, but 55 staff.
Matthew J. Schiavone:That's
Joe Wynn:Yeah. That's a bit suspicious Yeah. To to rock through that number. Yeah.
Justin Leapline:Yeah. So I mean and, you know, obviously, Dell dealt with this. It's more of a confirmation, I would say, in the industry, you know, that way you had that. We know there there've been the sock mills essentially, you know, just churning out sock for the stamp of it. But, like, what do you guys think, you know, like, there are a number of other companies that do it this way.
Justin Leapline:Oh, yeah. You know? I can count, you know, on one, maybe two hands and to varying degrees and everything. Like, there are, like one of those famous that Troy Fine often beats up on is, like, Comp dot ai. You know?
Justin Leapline:They're the ones that, like, saw two in a week, you know, all the claims and everything. But then you get the more automation platforms like Vanta Andrata, and it was actually interesting. The original article called out that there are public declarations of, like, anti competitiveness and, like, fair practice that, like, Vanta and Drata actually have, which was good. Like, they're not gonna peer up with a an otter and do cuts, you know, with that. Like, you can use our tooling
Joe Wynn:Oh, I see.
Justin Leapline:Whomever you want out of that. The the article actually called out some of those companies to say, like, hey. They they at least have a standard that they're not gonna get kickbacks on
Joe Wynn:Right.
Justin Leapline:Some of the stuff here.
Rick Yocum:Yeah. That's very important, especially in what we do at CISO when we're looking at compliance automation tooling. Mhmm. We're primarily focused on finding tooling that can truly use AI, use integrations
Justin Leapline:Yep.
Rick Yocum:To gather evidence because, I mean, what's better than doing screenshots of every endpoint
Justin Leapline:Right.
Rick Yocum:And trying to submit that and then doing that, you know, once a quarter or whatever it is when you can truly automatically at the click of a button or just within seconds, have it going gather all this information and Yeah. It's always there. And so, like, that's the first step, getting the evidence. And then being able to rely on the process that gathers the evidence, which should also be tested as part of a an audit. Mhmm.
Rick Yocum:You know, can you rely on the tool that's gathering the evidence? And then, we always like to have the human in a loop. That's like a big AI thing these days. You know, what's gonna be there to make sure that you're not getting hallucinations or or whatever. And then at that point, you know, we you know, our team, what we wanna do is make sure that somebody is validating.
Rick Yocum:This evidence makes sense, a true second line of defense role, and then but just rely on automation, rely on tooling to bring it to us, look at it, make sure it's good, tidy it all up, and then have the auditor come in and review the process for gathering it, review the tooling to make sure that it makes sense. And then at that point, make sure that the evidence is actually proving that the design is effective.
Joe Wynn:So Yeah. There's a huge difference between, like, using a car to go fast and letting the car drive you. Right? It's just a completely different level of trust in terms of, like, you use tools to go faster on things, but Yeah. How much you trust it matters.
Justin Leapline:Yeah. And the level of complexity and
Joe Wynn:Yeah. Absolutely.
Justin Leapline:That go into that into there. And, yeah, I mean, as a, you know, a creator of a GRC tool, I really love all that, you know, integration and AI. I mean, there is so much waste in the GRC aspect that can be lightened from a load perspective. But they're not they're not perfect because, like, one of the things I've seen commonly used in automation tool and then trying to go for an audit is, do you have the right scope? Like, okay, I included AWS, but I only did one instance, and there's three that should be included in this Yep.
Rick Yocum:I I've heard about that.
Justin Leapline:Yep. Like, those things aren't gonna caught by a tool. It's gonna say, yeah. Everything looks green, you know, type of thing. It's like, well, you only pointed to one thing.
Justin Leapline:So Right. Yeah, it looks green, you know, you know, into that. And that's where, like, you need the human in the loop, you know, into that. And there is validation on, like, some of the automation techniques. Like, yeah, I need to get trust into that's pulling the right evidence.
Justin Leapline:It's looking at the right config. It's looking at the right parameter that it's set. But once you get that comfort level, then you can repeat that thousands of times and say, yeah, it's a valid test.
Joe Wynn:And there's also a huge difference between AI doing an evaluation and AI building code to do a robotic style automation. Right? Like, if AI's building traditional code to evaluate a screenshot Oh. Well, that evaluation is gonna be done pretty consistently, more consistently most likely.
Justin Leapline:So, yes, in a context. But keep in mind, AI always has to translate into text. So we did some testing in, like, in Oh, yeah. Yeah. And everything.
Justin Leapline:The it it's easy it's better to pull parameters and configs. Way better to pull that than the screenshot. Yeah. So the screen but
Joe Wynn:the screenshot's not the point.
Justin Leapline:Like, the point is basically, like, if you use AI
Joe Wynn:to build code and, like, the traditional code is doing the evaluation of the control Mhmm. That's different than just
Justin Leapline:Oh, yeah. Yeah.
Joe Wynn:Having AI. I mean, you you told me this Yes. A while back, which I love because you were doing some, like, data manipulation. You were like, well, I asked AI to do it, you know, I don't know, a 100,000 rows
Justin Leapline:or whatever. And you're
Joe Wynn:like, it didn't do great. It did okay. And you're like, but then I asked AI to build me Python to do to fix the data. And you're like, and then that worked perfectly. Yeah.
Joe Wynn:Exactly. And that's such a clever thing. And I I think people get stuck, like, in what AI is sometimes Yeah. Because if you if you kinda shift the AI left a little bit and you end up with traditional code that can be QA'd and it's just gonna perform like traditional code consistently, you can use that really effectively to evaluate compliance.
Justin Leapline:And and the well, I wouldn't say new, but what a lot of people are leaning towards so the model context protocol, the ACPs. Yeah. It eats up a lot of tokens, and it has to give kind of references when you're referencing AI. A lot of different people are actually moving more to the CLI. So the actual console, it's having AI run console command.
Justin Leapline:So it's running AWS commands Right. To actually pull back the saves. Right. And then just evaluate those configs and everything. It's way more efficient.
Justin Leapline:Yeah. It's actually it's way better. It eats up less tokens. You get a more accurate picture because Yeah. It's pulling it straight.
Joe Wynn:It's not gonna hallucinate the inputs. Like yeah.
Justin Leapline:Yeah. Yeah, it's actually yeah. There's a big move. I I saw a meme the other day on LinkedIn. It was like, you know, 2001, make a CLI.
Justin Leapline:You know, web web web, you know, 2026, make a CLI. You know? You're right.
Joe Wynn:But I think it's an important point that people in this sometimes, like, if you're evaluating vendors that do compliance software y things. Right? It's like, well Mhmm. If you say you use AI, help me understand how you're using it specifically. Mhmm.
Joe Wynn:Like, is it the AI itself that's evaluating my stuff and making a call, or is most of that kinda traditional code, but it's AI enabled in various other ways?
Rick Yocum:Yeah. And and to build on that, when you're getting a tool that's gonna do that stuff, one of the things that like that article that we're talking about on Substack, he he really pointed out some things to think about, which is as you're picking a vendor, as you're picking a tool, how is that tool doing the things that they're saying they're doing? So for instance, if it says it's using AI to generate help you generate or help you review stuff, delve into that. Shouldn't have said delve into. Jump jump into?
Rick Yocum:Dive into that. Dive into that. I'll I'll try to be better. Dive into that and understand what's happening. If they're saying they do integrations and the claim that was made is that the way, you know, Dell did integrations was actually not the way you would think.
Rick Yocum:It wasn't go enter your creds for a read only account to pull the data. Yeah. It was it's giving you forms to go fill out and you still collect all the information is what he was claiming. I haven't ever used Delve, so I'm not sure.
Justin Leapline:But He had screenshots in there to Right. Legit, you know. And so, you know I've never used Delve as well, but
Rick Yocum:And if you're gonna go test out a piece of software to see if it's gonna do this stuff, ask the vendor, how exactly do you pull Right. This? What are the tests? Are they automated tests? Show me how these tests work.
Rick Yocum:Mhmm. Can I actually look at the how the tests are are built so I can actually rely on these tests to make sure they're pulling the right stuff? Right. And is that all happening and pulling it back? Yeah.
Rick Yocum:Then that's where the human loop, you gotta have your GRC team check it first, make sure it's accurate before you get audited.
Joe Wynn:Right. Matt, are you are you at liberty to talk about whether or not you're using AI to I was actually just
Justin Leapline:gonna ask them to.
Matthew J. Schiavone:I mean, who isn't?
Joe Wynn:Yeah. Right. Right.
Matthew J. Schiavone:Yeah. So we don't rely on it. It just speeds things up.
Justin Leapline:Yeah. Exactly.
Matthew J. Schiavone:We have evidence collection. We have testing procedures, report generation, mostly for, you know, just proofing, grammar, formatting, things like that. Why waste hours reading? Right. I mean, we read them.
Justin Leapline:Yeah. Yeah. No. No.
Joe Wynn:But like copy editing is different than content generation.
Justin Leapline:Right. Yeah. Yeah. Yeah, writing up a finding like, here are the facts, here are the evidence, write something up, you know, real quick and then you proofread You know? Yeah.
Justin Leapline:That's one of the things we've talked about AI obviously on the podcast before, and one of the big things is like, okay. Somebody's always responsible. Yes. It's not AI, you know, type of thing. You can't just say AI did it.
Justin Leapline:It's you, you know, at the end of the day. And I think, you know, doing these type of reports, we just, did something the other day. It was a report we generated off and it was like, it was great, you know, type of thing. It needed some modification, but, you know, you take the ownership at the end of the day, whatever it spits out. Right.
Rick Yocum:Well, speaking of you oh, go ahead. No. No, please. I was gonna say, I was gonna change the topic a little bit to talk go back to how CPA firms and how ISO 27,001 audit firms kinda get their ability to be accredited, so to speak.
Justin Leapline:Yeah.
Rick Yocum:But were you gonna hit on something relevant to the last topic?
Matthew J. Schiavone:No. No. I I think that's a great
Rick Yocum:So so I so full disclosure
Justin Leapline:about AI and its practice.
Rick Yocum:So so full disclosure, I used AI in order to create a comparison table here. So I'll run through and tell you what it says about the SOC two and then and the ISO one, and let me know if there's any thing that's different. So if you look at these areas, so what does a CPA firm do? Their oversight model is peer review, and the frequency you said this earlier is every three years, and they're audited by other CPA firms. And one of the things that I was unclear about that you clarified was that you get to pick your own auditor, so to speak, or Yeah.
Rick Yocum:Essentially. And and, you know, professional standards are the way that they're they're being enforced. And then on the ISO side, it's more of an accreditation model, and that comes down from, you know, the what is it? The AICPA? No.
Rick Yocum:That's on the CPA side. On the ISO side, they they're working through an accreditation. They they usually have an an ongoing or annual
Justin Leapline:Yeah. Yeah. Yeah.
Rick Yocum:And it's done by the, like, the global organization, and then they take the, like, the ANZ or and the other Accreditation bodies. Yeah. Accreditation bodies. ANAP. And they go through when they are able to certify the local ones, and then those are, you know, get accredited.
Rick Yocum:And then the other thing that they do
Justin Leapline:Rewind back. I didn't hear how they did quality assurance though.
Rick Yocum:Oh, I'll get to that. And so Okay. One of the things they do is they actually send auditors on an audit with an actual customer to sit there with the auditor.
Justin Leapline:So they send another auditor. Yeah. The shadow. Another peer auditing firm. Yep.
Justin Leapline:And who organizes that? Or yeah. Well, the that
Rick Yocum:that's organized by, like, ANAP. And we'll assign the local
Joe Wynn:So it's assigned though. It's not you choose.
Rick Yocum:Yeah. Yeah. That's a sense I'm getting of it. It's assigned and frequency. They will yearly, like multiple times throughout the year.
Rick Yocum:Oh, yeah. Yeah. They're getting they're getting that done. It's happening a little bit, you know, throughout the year. They get a and I'll find the name of what they call that audit, but they're they're sitting there with them, and they're basically auditing the auditor real time.
Rick Yocum:And so and then they also audit by getting the reporting that comes out Right. And sampling it Natural. And then checking the
Justin Leapline:So they're they're with it throughout the entire engagement. So I've done a handful of ISO audits, you know, into the the helping companies. Mhmm. You've done more, Joe. You've done several as well through your practice.
Justin Leapline:Have you ever had somebody shadow you?
Matthew J. Schiavone:I have not. We've just started certification, but
Justin Leapline:Oh, okay.
Matthew J. Schiavone:Gotcha. Most of you was on internal.
Rick Yocum:No. Luckily, none none of my audit customers when we were sitting through audits Yeah.
Justin Leapline:Actually had And I've gone through several. I've never had that. So I'm curious. You said annually, which seems very frequent, but I don't know. Yeah.
Rick Yocum:Yeah. It's on that's what I meant by ongoing. It's annual that they get a review and they're recertified. And I think twenty seven zero zero six is part of what the standards are for for doing some of that. And then and then the oh, it'll come to me.
Rick Yocum:I'm trying to remember what the what it's called when they have the auditor audit the auditor Okay. During the actual customer engagement.
Justin Leapline:It's just interesting to me. I've I've never heard the peer review and never ran into it. I would think I would have run into it at least once
Rick Yocum:or Yeah. Some of auditors
Justin Leapline:run into it. Yeah.
Rick Yocum:Yeah. Some of auditors tell us that and and they tell us the stories about how they're sitting there and, you know, they find something that might be marginal, like, should they call this out as a nonconformity? Right. But he's like, you know, every time that they're in being audited during a live audit,
Joe Wynn:you know, they they have to or they're gonna get totally written up. So I I can tell you, I've seen it once.
Justin Leapline:You have seen it?
Joe Wynn:Okay. But it wasn't on the tech side, so I was only, like, loosely related. Oh, okay. So one of the companies I was running security for, was getting a they're a manufacturing company. They were doing a quality audit, like, the 9,000 side.
Justin Leapline:Oh, okay.
Joe Wynn:Gotcha. The assessment process generally similar, and I and I remember specifically getting the request. It's like, oh, there's gonna be, like, three more auditors on-site.
Justin Leapline:Yeah.
Joe Wynn:Like, can they have accounts where it's like, oh, this is weird. Like, last year, was only this many.
Justin Leapline:Yeah.
Joe Wynn:And Got it. Okay. So, anyway, I've seen it once, although I wasn't, like, really directly. Right.
Justin Leapline:Right. Was more the client's name. Thousand. Yeah. Yeah.
Justin Leapline:Got it. Yeah. It was just okay. So they do perform those and everything.
Matthew J. Schiavone:I think I think those two steps in the process is what should differentiate ISO Right. SOC two. However, we've seen some implications on the ISO certification bodies in the Dell scandal. Yep. But I think having that stronger barrier to entry for certification bodies governed by the accreditation bodies, but then also that ongoing potential ongoing review.
Matthew J. Schiavone:You could be audited. The auditor could be audited at any time Mhmm. Not just at the conclusion of the process, the annual or the triannual peer review. So you have stronger barrier to entry, risk of audit at any time, and then, you know, annual renewal or whatever the renewal is.
Joe Wynn:It is an interesting timing thing for the Delve one. Right? Because if it's an annual review and Delve and was is
Justin Leapline:Oh, okay. You're you're I think you're confusing SOC two.
Joe Wynn:No. You're right. Because it wasn't the same Yeah. Yeah. It wasn't the same CPA firm, was it?
Joe Wynn:Those two That's
Matthew J. Schiavone:ISO ones?
Justin Leapline:No. It's two different companies. Yeah. So I forget who the ISO one was, but it was a different company.
Joe Wynn:Okay. Because I would I I was gonna say, I would have expected the annual one, theoretically, the annual QA process checking the ISO stuff to flag issues in in the eighteen months where the reports were from, you know, before the AICPA one would have.
Matthew J. Schiavone:Well, that brings up a good point, Joe. You can check you don't have to check me, but I think peer review has to start eighteen months after your first issued report.
Justin Leapline:They said that in the thing. Okay.
Matthew J. Schiavone:So a CPA firm could theoretically operate for eighteen months issuing low quality reports before peer review even is, you know
Rick Yocum:And you can get hundreds of reports out in that time.
Joe Wynn:485 or
Justin Leapline:at least 500 and Just need 55 people.
Joe Wynn:Right. Interesting.
Justin Leapline:Yeah. So, I mean, from a fixing perspective, is there something broken that needs to be reworked from AICPA and or ISO governance body?
Joe Wynn:I mean, if you have pervasive issues like, I'm not I'm not saying they were or were not allowed to keep issuing reports because I don't know. But if you have a peer review
Justin Leapline:that They fix what they they've found, then yes.
Joe Wynn:But even that time frame, fix this within, what was it, four months or something? Yeah. If you have pervasive issues, like, you should be forced to to hop immediately in my like, are you gonna keep issuing reports before your stuff's fixed? I mean and again, I'm talking about pervasive systemic I'm not talking about, like, minor deficiencies.
Matthew J. Schiavone:Right.
Joe Wynn:Right? There's probably some risk here, like, some risk rationale we can apply here. But if you're like, 82 of these reports that went out weren't actually about the client's system. Yeah. They were about Dell's system.
Joe Wynn:Yes. Like, that's bonkers. And and frankly, that's the type of thing also where I'm like, did the clients not recognize it? Or did the did the client's customers Nobody's reading it. Nobody in the whole
Justin Leapline:chain knew this? Right. The reports go out, nobody's reading it. What did
Rick Yocum:they say? A dentist office was audited or something like that? Dental was audited, but or their their report description.
Justin Leapline:Why is it dental office being a SOC two?
Rick Yocum:Yeah. Or or is it dental dental software maybe?
Justin Leapline:Oh, okay. That
Rick Yocum:makes more sense. And it was audited, and their system description, you know, was actually Delvs.
Joe Wynn:Yeah. 82 of them. 82 of, like, the 480 were that were that. So that's pretty egregious. Like It was glo glockcert?
Joe Wynn:Yeah. It's l l
Justin Leapline:o c e r t was their ISO 27,000.
Joe Wynn:So Okay. Got
Matthew J. Schiavone:it. That was a certification body?
Justin Leapline:The yeah. That's the company they used to issue out the ISO. Yeah.
Joe Wynn:Now I actually don't It's
Justin Leapline:a Indian cert mill according to the article.
Joe Wynn:Okay. I was gonna say, I don't have the detail on this. So I know I know the SOC reports had some egregious issues. Did the ISO reports also have egregious issues? Do we know?
Joe Wynn:Or was it just they were implicated in the the Google Drive?
Justin Leapline:Yeah. So they're implicated. They they weren't as bad. I'm looking at the article right now.
Rick Yocum:The the article didn't really delve into, dive into the the ISO reports itself. And yeah. And all of the strangest that they found that was repeated in the SOC two reports. But they went into looking at the criteria of and who was the auditor for the ISO certification and where are they based. And, I and I if I recall, they all kinda had this model where they had a US Shell company name.
Rick Yocum:They were actually owned and run by an operating out of maybe India. And those were they were very similar in nature. Yeah. And I think there might even even have been some tied together in people who were highly owners or high up individuals between both the CPA firm and the ISO shop.
Justin Leapline:Sure. Sure.
Rick Yocum:So it and as they started to pull these pieces together, they were making connections between them all Mhmm. Which now you have your little ecosystem.
Joe Wynn:Yeah. Yeah. I mean, so one, if if there are pervasive issues of that nature, it feels like you should be forced to stop, like, as opposed to keep going. Mhmm. Two, I it's just interesting to me that you could go 18 I don't know if this is necessarily wrong, but it does feel broken, and I don't know what the fix is.
Joe Wynn:Yeah. Like, going eighteen months with no real review on something like that feels kinda weird. And then also, I don't really know what prevents these parties from, yeah, you know what? You're right. We'll close-up shop, and we'll just start up elsewhere.
Justin Leapline:Honestly, I think I don't know how effective it would be, but there should be consumer reporting, you know, into that. Like, if I get a SOC two and let's say, you know, they charge me $5 for it. Right. And then I get it. Maybe I'll be happy with it, but maybe it's like they misspelled stuff.
Justin Leapline:They did all that stuff. Like, I should be able to report them and be like, this report is kinda garbage. You know? Now Well, now your SOC reports $6, but they have a
Joe Wynn:whole bunch of bots
Justin Leapline:Right.
Joe Wynn:Messing with the numbers.
Justin Leapline:Right? You know, at least from that perspective, like, it it should you should be able to at least report and then maybe get a little bit more like, hey. We need to audit you now because we're seeing this, you know, a a valid, you know, not just any public comment, but
Joe Wynn:Right.
Justin Leapline:You know, a customer that came and it's like, hey. They they basically described Dell's environment, not ours. You know? Yeah. Right.
Justin Leapline:Type of thing. Like, those type of things is like, okay. There's a quality issue here. We need to address it sooner rather than later or axe it, you know, type of thing. Yeah.
Justin Leapline:Because they're in charge of maintaining reputation. Absolutely. You know? So there's if AICPA doesn't really do anything, I I don't think they can continue to do what they're doing today. They have to get a little bit more strict and tighten up some of those stuff, because they're getting a bad rep.
Justin Leapline:And if you get a bad rep, they're either gonna go away or something's gonna replace them or both.
Joe Wynn:Yeah. The trust erodes. Right?
Justin Leapline:Yeah. Exactly. And I think you even said you started to see some clients come back to you because of this issue. Is that right? That is correct.
Justin Leapline:Yeah.
Matthew J. Schiavone:Yeah.
Justin Leapline:So so obviously, like, clients don't like, the clients want the SOC two, but they don't want the bad rep from the bad companies also. You know? Like so they don't want that noise as well, you know, into it.
Matthew J. Schiavone:And it's as much as you can or you can try you just can't educate a client on why they shouldn't go with somebody who costs less. Mhmm. Well It just
Joe Wynn:if you're selling the service, you're not in the right chair to educate Exactly. Like, I mean, you can try but
Matthew J. Schiavone:We can we can Yeah. You know, talk about independence and then, you know, why we can't do this and they shouldn't be able to do that and, you know, quality and it just doesn't register until something like this happens. So I I think this is going to be beneficial to, you know, SOC and ISO and and the credibility, we'll get there. And we look to other frameworks, look at CMMC and HITRUST. There's additional steps along those processes to issue these reports, you know, undergo those engagements, barriers to entry.
Matthew J. Schiavone:So I'm hopeful to see AIC payable figure it out.
Justin Leapline:Yeah. Yeah. Oh, by the it's We
Rick Yocum:shall see. It's called a witness audit. Oh, that that's what happens when the auditor comes and audits the auditor. And I think every you know, from what I'm reading is every every auditor so if you have an ISO shop and you have lots of auditors, each one of those auditors might get a witness audit every, like, two to four years. And and so if I'm thinking about this the right way, then one certification organization will have multiple witness audits because and that they'll be spaced out.
Rick Yocum:So you might have a couple a year, but not the same auditor. And so and so if you and then if you think about the way it works is you're getting an annual audit. You're one week maybe of of the fifty two weeks a year that that's happening for that auditor. Maybe they're not do maybe they're doing, like, 40 a year or something. And as they do those, you know, you you might just get lucky.
Rick Yocum:So you won't be in the one that has witness audit.
Matthew J. Schiavone:Right. And that's enforced by the accreditation body.
Rick Yocum:It is. You know, and
Justin Leapline:you But I'm curious who pays for that audit. I think the your fees to the
Matthew J. Schiavone:accreditation body probably fund those
Justin Leapline:You think? Activities. Okay. I mean, because those auditors aren't going for free out there.
Matthew J. Schiavone:Right? Yeah. I I don't know for certain, but I I I would imagine.
Justin Leapline:And in your case well, in the AICPA case, you're funding that directly for the peer review We are. You know, that kind
Rick Yocum:of thing. The certification body pays for the witness audits part of
Justin Leapline:the I
Joe Wynn:was gonna say, makes sense because again, you wouldn't want the funding source to potentially muddy. Yeah. Exactly.
Justin Leapline:Yeah. Or be gouged or something like that, like, if it's not your choice and everything. Has so you gotta pick your own peer review audit. Has or Wait.
Matthew J. Schiavone:I'm sorry. So just to confirm, the certification body pays for the witness audit?
Justin Leapline:He just had?
Matthew J. Schiavone:Yeah. But they can occur at any time based on the accreditation bodies.
Joe Wynn:Yeah. Decision. I'm sure there's, like, any time within, you know, not to exceed this frequency.
Rick Yocum:Yeah. I I think I kinda get I'm
Matthew J. Schiavone:a little fuzzy on that process. It was difficult. Okay.
Justin Leapline:Yeah. But, like, has there ever been a scandal before where, like, you know, two CPA companies kinda do the handshake? Like, I'll do your peer review, you do mine, and we'll just sweep everything under the rug. Has that ever, like, come up before?
Matthew J. Schiavone:That I know of? No.
Justin Leapline:No. Okay. I'm sure. Because you look at that, you're like, well, why not? You know, like, you could see it happening.
Justin Leapline:Yeah. You know?
Joe Wynn:It's all a spectrum too. Right? Like, to to what extent is it, well, I know you and I trust you and you know me and you trust me and we can cut each other like friends and family rates versus like, hey, you're just gonna say yes. Right? Because I'll just say yes to you if you just say yes to me.
Joe Wynn:Like, it's a totally different
Justin Leapline:It is. You know, type of thing. But
Matthew J. Schiavone:I think something reciprocal where you do mine, I do yours is probably, you know, there's independence considerations there that
Joe Wynn:I would hope that there were prevents lags.
Matthew J. Schiavone:But I there are certainly, I'm sure sure handshake deals that, hey, you know. Because if you look at some of these peer reviewers who are reviewing SOC two firms and you go to the peer reviewers website, they don't even have SOC two as a service of theirs.
Justin Leapline:So there Really?
Matthew J. Schiavone:So there are I find that in but there's nothing that
Justin Leapline:Their service is peer reviews?
Matthew J. Schiavone:I what I can't do is prove that they haven't hired an outside practitioner to help them with peer review of those applicable SOC two reports. So I can't make that claim. I'm just saying there are instances I found where CPA firms conducting peer review with SOC two don't have SOC two on their website.
Justin Leapline:That's interesting. Yeah.
Rick Yocum:So they may not be trained in what all the pieces are.
Matthew J. Schiavone:But again, they could hire outside consultants.
Rick Yocum:Somebody they might have to go and get somebody with competency to competency to do that.
Joe Wynn:Subtracted out. Subcontracted out to the same firm you're reviewing. Hey. Can you review your own
Justin Leapline:Yeah. Right.
Joe Wynn:Program through me? Yeah. Same person that did the report. I'm not suggesting anyone that has done that.
Rick Yocum:Yeah. Now is there a place in the audit where you needed to clear any potential independence issues?
Matthew J. Schiavone:Very early. I mean, we yeah. We have a new engagement acceptance procedure where we consider other services we have performs or with the clients or are engaged currently to perform. We have to document that all before signing the SOW, sending out the engagement letters, then throughout the process of the audit, we've Gotcha. Document.
Joe Wynn:Is any of that documented in the report itself, or is it just part of your, like, internal risk practices that you must do these things?
Matthew J. Schiavone:In the internal risk practices, in the audit binders and work papers, we, as a firm, we include SOWs if we are providing Oh, I got it. Other services, for clarity for the peer review. And in the management rep letter, which no one sees, the client signs a rep letter and a rep that they took responsibility for all non attest services that they
Justin Leapline:may have performed. Yeah. So is there anything that's, like, really preventive? Like, if you did a pen test for them and coming in and do the SOC two, even though that's part of your evidence, like, that wouldn't prevent you from doing that. Right?
Matthew J. Schiavone:We have to be able to demonstrate just our firm's standards and policies that they have appropriate subject matter expertise to evaluate the findings. They have people on staff who essentially understand. No. No. No.
Justin Leapline:I'm talking about if your company did the pen test and now you're coming in to do a SOC two, you know, into that, that independence wouldn't be a conflict or would it be? It does
Matthew J. Schiavone:It could be.
Justin Leapline:It could be?
Matthew J. Schiavone:It could be.
Justin Leapline:Yeah. So
Joe Wynn:But you'd have a you'd you'd have to disclose it and a partner would potentially make a call one way or the other as to whether or not you're gonna do that work.
Matthew J. Schiavone:And we have to document. Yeah. If we do it, we have to document why we're comfortable.
Joe Wynn:Yeah. Why here's why it's okay.
Justin Leapline:Right. Yeah. Yeah. I'm just curious. Like, a lot of times, at least in the PCI realm, they just they have a spot for independence, but we usually just say it's another team.
Justin Leapline:We're totally separate. We don't report, you know, nothing's going through them, you know, type of thing. Like, I have no say on their findings. They have no say on my findings. You know?
Matthew J. Schiavone:That yeah. That that doesn't matter to us. Yeah. Because, I mean, I would never conduct a pen test. So, it's gonna be performed by a separate team.
Justin Leapline:Yeah. That's what I'm saying, like Yeah. Into that, like, that that there is a kind of internal independence, you know, type of thing. Like, even though truth be told, you could have a partner maybe muscle, you know, like, give them a clean bill of health. I mean, it
Joe Wynn:It totally depends on structure and everything. Yeah. Like, because, I mean, all the firms
Justin Leapline:are set up differently.
Joe Wynn:But you you could very easily say, oh, yeah. This partner is responsible for this type of technical testing. This other partner Right. Is responsible for this other thing. They're fully independent for these reasons.
Joe Wynn:Here's how we do peer reviews. Here's how we
Justin Leapline:do other cross checks. Like
Matthew J. Schiavone:I I would also make the argument though of as an auditor, I wouldn't I wouldn't care if the pen test necessarily came back riddled with findings as an auditor. I mean, I guess I would. I should probably shouldn't have said that.
Justin Leapline:I I guess it depends on the remediation, you know, type
Rick Yocum:of Yeah.
Matthew J. Schiavone:That's what I'm there's additional steps in the process that I should care about as the auditor. But Right. I would
Rick Yocum:expect you'd be interested in, hey, was that pen test a quality pen test that somebody that it really exposed the risk that this company has. And if it did uncover flaws and vulnerabilities, are why are those there? And what is the process that gets those remediated? But also, what's the process that fixes the underlying cause?
Justin Leapline:Are they not doing change management well, patch management well? Like, it could show
Rick Yocum:Baseline standards.
Justin Leapline:Yeah. Exactly. You know? Or did
Joe Wynn:you just or did
Justin Leapline:you just buy the software from someone else and it's your first one and this is the process where Yeah. This is your first pen test, then there you're gonna have findings because you're not aware of maybe some of the Yeah. Configuration baselines you needed, you know, type of thing. Yeah. There could be a lot of reasons why.
Justin Leapline:No. It's a great point. Yeah. Yeah. But, that should be separate from the SOC two audit versus the pen test.
Justin Leapline:I mean, they're they're different skills. You know?
Matthew J. Schiavone:What I meant by that is Yeah. There isn't an there shouldn't even if it was perceived as an independence impairment, it I don't see how it could be our pen testers would be incentivized to give them a clean report due to my audit.
Justin Leapline:Yeah. Oh, yeah. Yeah. Right.
Joe Wynn:Right. And I'm sure there's some threshold too where it's like, oh, we're doing VC so, and we're doing the pen test, test and and and and and. There's eventually one too many ands and you go, well, let's not do this next thing.
Matthew J. Schiavone:Yeah. Like VC so we wouldn't even
Justin Leapline:Right. Yeah.
Rick Yocum:Yeah. That's that's because that's more on advice. That's more on getting configured the right way Mhmm. Build. And so, you know, one of the things you because you
Justin Leapline:could be control design Right. Even just Exactly. Control. Right. Yeah.
Rick Yocum:Right. Yeah. So, I mean, one of the things Matt and I had talked a lot about over the years is, you know, what's what's the separation? And there were times when it might have been easy from a contract standpoint for me to bring, like, Matt's company in to get it done. And as that kind of stuff unfolds, it's like, no, we really need to have our contract signed independently by the customer.
Rick Yocum:That way, there's independence just because I know Matt and I know he does quality work Mhmm. That becomes the the the most beside besides me vouching for him, that becomes the most, potential independence issue that there is. It's like Right. I I'm fully independent for Matt because I I know him for many years now. So when I suggest that he's a good auditor, his reports are gonna come out, they're gonna be something you can rely on, and in fact, this is something we talked about when we were presenting to the we did the fireside chat with the, you know, Isaaca.
Rick Yocum:Isaaca. Yep. Is, you know, when when you're getting that audit done and you really wanna be able to look at that report, go to your customers and say, yeah, our controls are good. And you can believe this because look at the quality of the auditor Right. That I have here doing it.
Rick Yocum:And when that all happens, you know, then then you're gonna have something you can rely on. And But what if your customers don't care? Well, I guess it depends on you're you're probably not selling up market enough yet. Yeah. Someday, when you start selling to more sophisticated customers, they that are processing that, you know, higher level regulated data Yeah.
Rick Yocum:They're they're probably gonna care.
Justin Leapline:So I I think I told this story before. I might have mentioned it in a second. So when I first went in to head up the GRC practice at Diebold, a big fortune services company, Fortune 500 into that. I came into the role. We just did, at that point, our first SOC two, and we're finishing it up.
Justin Leapline:Mhmm. And it was within my first week or two. We hop on the phone with the CPA company. They're like, you got some qualified opinions into it. And for those who don't know, qualified opinions are we didn't meet the control fully, and now you need to do a management response and all that stuff and everything.
Justin Leapline:And it was so we did the manager's response. We, you know, stamped the the report, you know, got it all done, and it was up to me. It was like, alright. Do we hurry up and do it like a three month testing time window, clean this up, and get a clean report, you know, which is the minimum time to get, you know, clean? Do we hold off until next year?
Justin Leapline:Like, it was solely on my shoulders on what to do. And I decided after thinking a little bit, I was like, I'm just gonna send it out. I'm gonna send it out as is. In a year, we'll clean it up, and we'll see how it goes, you know, type of thing. So we'd send it out with the bridge letters, all that stuff.
Justin Leapline:It took almost ten months for a customer to come back and be like, you had some deficiencies into this. I was like, yeah. We cleaned it up, and they're like, okay. Yeah. I mean, it's like, I got zero stuff back, like Yeah.
Rick Yocum:How many customers did it go to and you got one? Hundreds.
Justin Leapline:Hundreds. Hundreds.
Rick Yocum:So, yeah, so hundreds either looked at it and said, oh, well, these deficiencies don't matter to me, or more likely, nobody read the report.
Joe Wynn:Right. Well, there's something interesting here too, because I can tell you, if you're at an organization that has an issue, right, whether it's a breach or even it's Mhmm. Something where so, like, rewind a bunch of years. I'm at a company called Black Box, and sometimes we would we would have this, like, sort of DMZ network where clients would send us network equipment that wasn't working, and we fix it. Yeah.
Joe Wynn:Sometimes it was riddled with malware. Right? So we'd get a piece of network kit on a network that we owned. Mhmm. Right?
Joe Wynn:And it would start posting, trying to fall back and everything. So then the the bit sites and the Upguards and all the the the tools would start to be like, hey. We see weird traffic here. Your network, your score is going down. Right.
Joe Wynn:Right? But I just think it's interesting. So, like, a SOC report finding is bad ten months before someone says something. But I can tell you, as soon as you drop a piece of network kit into the hardware, the security nerds, right, not necessarily the GRC nerds, the security nerds are like, hey. Our threat groups are telling us that you have an issue on your network.
Joe Wynn:Do you wanna look into that? And that happens, like, fast. Right. That happens, like I'll tell you, like, six or seven years ago, it was happening within a week or two. Now it's happening in a day or two.
Justin Leapline:Interesting. But it's not perfect there. Like, I remember the Oh,
Joe Wynn:it's not perfect at all.
Justin Leapline:AWS in there as a vendor, and it's like, oh, they're a d. I I will tell you, well, that's because they're a hosting company and all the bad stuff coming out. I was like, you know.
Joe Wynn:I am certainly not vouching for those systems.
Justin Leapline:Yeah. Right.
Joe Wynn:But but I will say it's interesting in terms of the the perspective, and I think it kinda also speaks to how overwhelmed to your point before, Joe, right, in terms of, well, now we have SOC reports to review and questionnaires to review, and I have way more vendors than I ever used to have. And I've been pressured to increase the ratings of what's high and critical based on these other standards and regulations.
Rick Yocum:Yeah. I wanna come back to that after the break.
Justin Leapline:Yeah. Yeah. Yeah. Yeah. And talk about solutions.
Joe Wynn:But it's just interesting, like, how fast the GRC teams can react to this stuff and versus how fast the the technical security teams, which are a bit typically a bit closer to automation, are reacting to other signals that
Justin Leapline:are coming out. Yeah. Yeah. I guess, you know, to your point of, like, collecting and getting good stuff, I mean, how many times have we run out across customers? They collect the report.
Justin Leapline:They throw it into the folder, and they checkbox the we've done our due diligence for this vendor.
Joe Wynn:They have one. Yeah. We received it.
Justin Leapline:I remember, like, I did another story. I was doing a PCI client, and they got I I think it was actually a SaaS 70 back in the day. But they had a data center vendor got the report got the report back, and all the findings and the, like or the controls and the description and the the testing of that was deleted out of the report. It only had the description of the environment into it. And I'm like, guys, where's the rest of the report?
Justin Leapline:They're like, well, this is what they gave us. You know? And the customer didn't know any better. Like, they didn't know there was a whole, like, listing of controls and whether it, like you know, what's the statement like no no not nothing noted or something like that. Yeah.
Justin Leapline:No exceptions noted or something like that. Yeah. There was none of that in the report. And then so we're like, you know, we reached back out to the data center. We're like, guys, what's up?
Justin Leapline:They're like, that's confidential. It was like, you got a SaaS 70 to say it's confidential. It's like, come on, guys. You gotta find it. Like, we know you gotta find it, you know, into this.
Rick Yocum:That's what the NDA is for so you can share the confidential stuff about the controls I'm relying on in my company that you're providing.
Joe Wynn:And that's when they invented a SOC three.
Justin Leapline:Yeah. Right. So yeah. And it's just like, you know, we got, you know, a bad education problem on, you know, the on the vendor side, you know, just basically trying to shortchange, you know, the system. And then the customers don't know any better because, again, we talked about the incentive structure of, I just want this big deal, and they're saying I need a SOC two.
Justin Leapline:I don't care the in between. I just wanted the cheapest, fastest, you know, way to get that, you know, type of thing. Well and I've I've seen so just I'd interested
Joe Wynn:in if if you can speak at all to the clients that you have seen in your career on this, but I've seen a bunch of organizations where it's the sales team that's sponsoring the SOC report Right. Out of their budget as opposed to, like, the IT team as a for instance. And again, there's some incentives there, right, in terms of how fast you go and who's receiving it and stuff like that.
Justin Leapline:I'm just asking, are we bringing back the trustee certification on the website? Just slap it on your website. They're like, everything's good. Yep. Yep.
Rick Yocum:You're gonna have to drink with that comment.
Justin Leapline:Yeah. Right. That's what I'm gonna do next. I'm gonna invent the industry, bring back the trusty trusty two. You know?
Justin Leapline:Right. But I'm but I'm just saying, like I'll do it for $4 instead of $5.
Joe Wynn:But organizations have pressure internally to get their own SOC reports because they wanna they their sales teams wanna stamp it or they can't engage with their their, you know, downstream clients unless they have the thing. Right? Even if they're not gonna look at it. Right? It's it's a bar they have to clear.
Matthew J. Schiavone:I've definitely seen the push from the sales team. Whether or not I've ever seen it hit their budget, I can't speak to, but it's very interesting. Yeah. They the the pressures that have the report often come from the sales team, but
Justin Leapline:I don't think it would it hit their budget.
Joe Wynn:Is it? I've seen I I've worked at a it might not be terribly common. Yeah. But I have worked at a place where when I landed there, they were the sponsor of Yeah. The the stock reports and all that stuff.
Rick Yocum:Yeah. Because it helps them close deals.
Joe Wynn:Yeah. Absolutely. It's it's it's yeah. It removes friction. Yeah.
Joe Wynn:And to and to some extent, I think at that place in particular, it was one of the distinctions at the time when I got there. They were doing a lot of what I would call checkbox compliance. Mhmm. It's like, well, who's paying for it? Who cares about it?
Joe Wynn:They're not gonna read the thing. They're just gonna pass it forward. Right? This is the sales team thing. And then we eventually took it over and stuff like that, but I have seen it.
Joe Wynn:Interesting.
Justin Leapline:That is interesting. But is this a good breaking part? Love it. Yeah.
Joe Wynn:Alright. That that was definitely enough to drink.
Justin Leapline:Yeah. Exactly. Let's refill here. Matt, some more. Please.
Justin Leapline:So Matt was gracious enough. We don't often bring our guests force our guests to bring bottles, but we're we don't condone it as well. You good?
Matthew J. Schiavone:Yep. I'll take it up.
Rick Yocum:That's good.
Justin Leapline:So, yeah, Matt brought us ard Ardbeg? Ardberg. It's there's no r.
Joe Wynn:Or is it Ardbeg? It's a I've pronouncing it wrong for years. Arbeg. Yeah. I've been pronouncing it wrong for years.
Justin Leapline:Yeah. Arbeg. It's a five year old scotch. It has a pretty decent smoky taste, but not overpowering Yeah. Into this.
Joe Wynn:You Sweet and smoky.
Justin Leapline:Yeah. You definitely notice the the smoke onto this. It's the Wee Beasty Mhmm. Single malt scotch, the ultimate, non chilled filtered.
Joe Wynn:Yeah. It's quite good.
Justin Leapline:Sorry. Yep. Yeah. It's actually What's the proof? Proof is 47?
Justin Leapline:I don't see it. 47. Yeah. Nice. I think that's it.
Joe Wynn:When if you follow the cocktail subreddit
Justin Leapline:Forty seven four. Yeah.
Joe Wynn:You will note that it was recently voted on Reddit for whatever it's worth on that subreddit. Like the The what? The the cocktail subreddit on Reddit.
Justin Leapline:Okay.
Joe Wynn:Voted it. I think most underrated scotch, and then there was a second category at one in as well for like I forget. Most versatile maybe or one of those. They they did a whole Okay. Thing.
Rick Yocum:I'd just say it was 47% alcohol or 47 proof? Percent alcohol. 94.
Justin Leapline:Yeah. 94. Yeah. 94 proof.
Rick Yocum:Yeah.
Matthew J. Schiavone:Okay. Sorry. I
Joe Wynn:yeah. Which it's super smooth.
Justin Leapline:Yeah. It settles really well.
Rick Yocum:Was gonna say, wouldn't be surprised if you told me it was 47 proof because I was only 23% because of how smooth it is.
Justin Leapline:Yeah. So, yeah, you get a good smoky but not overpowering, you know, into that. And then it settles into a nice, you know, scotch whiskey flavor into that. So yeah.
Rick Yocum:Yeah. Thank you very much
Joe Wynn:for bringing it.
Justin Leapline:Thank you.
Joe Wynn:It's great.
Justin Leapline:So cheers, guys. Cheers. The table's so much bigger now. Yeah.
Rick Yocum:We'll just have to pass it down. Quick
Speaker 7:break to hear from one of our sponsors. If you own security, compliance, or risk, and it feels like you're always pushing a boulder uphill, I want you to know about CISO. CISO helps growing companies get order ready, reduce risk, and stay resilient without drowning in tools, endless checklists, or one time reports that quietly rot the moment the audit ends. This isn't shelfware. It's not drive by consulting.
Speaker 7:With CISO, you don't just get advice. You get hands on support from real security engineers, GRC specialists, and former CISOs who help you build, operate, and continuously improve your security program over time. Whether you're chasing SOC two, ISO 27,001, CMMC, HIPAA, or you're simply trying to get security under control so the business can move faster, CISO meets you where you are. Their managed VGRC model gives you enterprise level expertise without hiring a full internal team or reinventing the wheel. The focus is simple.
Speaker 7:Clear priorities, practical controls, and measurable progress leadership can actually understand. Visit cisodellc.com and start the conversation. Security you can trust, compliance you can prove, and people you can depend on.
Justin Leapline:Alright. Welcome back to Distilled Security Podcast. Before we get into kind of our final topic here, Joe, I believe you had something to
Rick Yocum:Well, it's it's my monthly reminder about b sides Pittsburgh. It's coming up July 10. Tickets are still $20 until April 14. And being that today is the March, by the time this is out there, you're gonna be really close to that deadline. So get your ticket if you haven't already.
Rick Yocum:Then they go up to $35 until June 12, and then I think it's in the 70 range. Mhmm. And right now, call for speakers closes April 17. So if you're listening to this, it's gonna be really close. Get that submitted.
Justin Leapline:We'll have it out from the website. Here. Yeah. You know, type of thing. So you'll have a week.
Justin Leapline:You have a week. At least a week. Yeah. Time.
Rick Yocum:Yep. And and and maybe four days until the
Justin Leapline:prices Get on it. Don't delay. Yeah.
Rick Yocum:And then sponsor tables, they're going fast. We're getting it's gonna be a great event this year. We're getting a lot of sponsors to help us make that happen, but we still have tables left. Yep. So please, if you're a sponsor, get your your request in for sponsorship.
Rick Yocum:And if you're one of our long long time attendees and you know some sponsors that you think will be doing a good job of helping us make this thing happen, encourage them. Yeah.
Justin Leapline:Yeah.
Rick Yocum:So thank you.
Justin Leapline:It was at the end of last year, you almost had, like, somebody competing to get, like, the the platinum one. I remember at the after party, there were
Rick Yocum:Yeah.
Justin Leapline:Vendors are like, no. I'm doing it. No. I'm doing it.
Rick Yocum:Yeah. Yeah. We've we we we've had a couple different new new perks, and I think the highest level out there, it's a little pricey, but it is, you know, well well worth the perks you can get.
Matthew J. Schiavone:You get
Rick Yocum:a And that one's still open. Yeah. And, you know, you really get to show your support to the community. Yeah. Like, all the money for this goes right back into the next b sides, planning this stuff.
Rick Yocum:Everybody, all of us, do the we're volunteering to Wow. To make this happen.
Joe Wynn:And it's really why, like, for such a long time, the tickets are are really good price because if they were at cost, they're a lot closer to that $70
Rick Yocum:Yeah.
Joe Wynn:Or more. Yeah. For the meal and the stuff you get.
Rick Yocum:Yeah. For the quality of presentations you get all day and the quality of education you'll have and just for the atmosphere and and being able to be part of that, it is, you know, you wish to be charging twice as much as the highest amount.
Joe Wynn:Yeah. But for the community.
Rick Yocum:Yeah. Great.
Justin Leapline:Alright. So to kinda put a bow on all this stuff, we've been doing a lot of complaining, I feel like, in the first hour. Now what? Yeah. So, yeah, why don't we talk about some now what are we doing, you know, with this?
Justin Leapline:What like, is SOC two dead? Are we waiting for changes? Is there something better out there? How do we modify our programs to make this resilient, you know, into issues and bad auditors out there? Like, what are what are some things on the table that we could be doing?
Joe Wynn:Well, I mean, I I would say, first and foremost, if if you have Dell as a piece of software or
Justin Leapline:or or even if you think not even them. Like, they they're not the only company.
Joe Wynn:That's true. Absolutely right.
Justin Leapline:So that's
Joe Wynn:what I was gonna say. Or or you know that you're doing a level of checkbox compliance that might come back to haunt you. Right? You probably should think really hard about leveling up to a provider that'll, you know, kinda provide a more complete, you know, SOC review experience or whatever the ad test.
Rick Yocum:Right. Let's break this down a little bit. So you have you know, there there's a few components. You have the auditor Mhmm. And the auditor is gonna come in, and they're either gonna give you a quality or a non quality audit.
Rick Yocum:And I think you can tell the difference based on, you know, just just under get understanding for how they test and what they do. And then you have and you don't have to use compliance automation tooling or GRC tooling. You can still have that same audit problem with you manually collecting all of your evidence Mhmm. And still having somebody not really pay attention to look at it, or you can then have a more quality audit with a more reputable auditor that you can trust. Right.
Rick Yocum:And so kinda break that apart. It's not just the software, and it's not just the audit. It's a lot about what are you doing in the middle Right.
Matthew J. Schiavone:As
Joe Wynn:well. Well, and and, Matt, I mean, if if I was a potential client, right, and I'm coming from a place where I had maybe a not so good provider and I'm looking to move, but maybe I don't have the budget to do, the full on thing, are there certain levers I can pull? Or, like, what would be the the glide path towards moving from maybe a checkbox, you know, compliance place to a place where you're getting a slightly more thorough experience?
Matthew J. Schiavone:That is that's a good question because, you know, we try to keep our fees competitive, but, you know, we produce perform a quality audit.
Joe Wynn:There's gonna be a floor.
Matthew J. Schiavone:So we try to in those instances, we try our our best to meet the customer at a competitive place and hoping they see the value in that. And Does
Justin Leapline:control infrastructure matter at all? Like, if they're doing automated testing and you're getting all those reports and everything, like, instead of them doing all, like, manual controls or something like that. Like
Matthew J. Schiavone:It does, but we've gotten burned in those instances because some clients think it's a set it and forget it. Okay. And they set it and then they forget it and we get in there and it leads to more questions than we have answers and then we're bothering the clients more than they anticipated because we were supposed
Justin Leapline:to It's more work. More work on
Matthew J. Schiavone:those. Yeah.
Justin Leapline:Yeah. Yeah. Exactly. I don't know what you mean.
Matthew J. Schiavone:And a lot of these platforms, auditors advertise a hands free, hands off audit and we can't really operate in that way.
Joe Wynn:Yeah. Nor yeah. That that's kind of the point. Right?
Matthew J. Schiavone:Yeah. Right. We do our best to leverage those platforms as they're intended.
Rick Yocum:Yep. But you still have to talk to the customer.
Justin Leapline:Yeah. Yeah. And understand the infrastructure. Yeah. And it's not even like like the GRC automation tooling is one thing, but it's also like, what if I set up AWS, you know, security center to, like, monitor all my controls and map it back to a a standard and, you know, now I have real time monitoring for the instances.
Justin Leapline:Hopefully, they're ScopeRight are into that. And now all you need to do is, on the technical side, download all the the stuff into there. You still have to look at change management and all that. But Right. I'm just wondering if anything, like, would affect like, if I'm strapped for cash, could I help put those automation controls into place where it makes your life easy from a testing standpoint?
Matthew J. Schiavone:Yeah. Absolutely. Okay. But we you know, like I said, we've gotten burned.
Joe Wynn:So we
Matthew J. Schiavone:we we have our floor. If we can if you can demonstrate to us in the, you know, the presale the sales cycle that, hey. These are our controls. They're operating as intended. We we typically don't get that deep because we're not really presented with this problem too often.
Matthew J. Schiavone:Gotcha. But during that process, we do review the controls. We review the scope to make sure everything's aligned. Yep. We like to hear what went right with past audits, what went wrong, how can we improve.
Matthew J. Schiavone:And if one of their, you know, things they liked is the auditors never talked to them, then then that's a that's a
Justin Leapline:red flag But, for
Matthew J. Schiavone:yeah, automation is helping, but we we just find ourselves in these instances where the clients have that platform and they're just not maintaining it. Right. Not And I get that. Yeah.
Justin Leapline:So, like, maybe, you know, whatever it is in your contracts, like, year one, if you're working with a new client, that's a high risk, you know, because you're unfamiliar with the environment. Maybe year two and three, like, once you get familiar with it and, like, okay, they have a pretty mature environment we can rely on, maybe, you know, type of thing. Right. Give them some discounts or I don't know.
Matthew J. Schiavone:Yeah. No. And and it helps for us when we're view reviewing the prior year report. You know, we understand the marketplace. Yeah.
Matthew J. Schiavone:Yeah. We know who's issuing these reports. We know the price point. We know their tolerance for, you know, evidence
Justin Leapline:Yeah.
Matthew J. Schiavone:And timeliness. So, you know, reviewing prior year reports helps us certainly.
Justin Leapline:Yeah. Gotcha.
Joe Wynn:Well and I would imagine, you know, maybe, maybe not sometimes, but sometimes there might be scope levers you could pull. Right? If maybe you're getting a broad checkbox compliance thing, yeah, maybe you don't get certification of the entire environment or whatever. But, you know, maybe you only need this one thing. Yeah.
Joe Wynn:And and you could do that a little quicker.
Justin Leapline:Just looks at their SaaS app, and that's all they care about.
Joe Wynn:Yeah. Could probably do some I I would guess that. But also, don't think it's a bad answer to say, yeah, no. Sometimes, you know, you just people are gonna if you want humans to spend time reviewing the stuff, it's more of an education thing to say, well, a $5,000 sock report doesn't buy you compliance, it buys you liability. And if you wanna actually, you know, have the thing that you can stand by, well, you're gonna need to spend a little more money.
Joe Wynn:I don't think that's a bad message necessarily, but to your point earlier, it's an education thing upstream.
Matthew J. Schiavone:Sometimes they learn the hard way.
Justin Leapline:Yeah.
Rick Yocum:Yeah. I would imagine that when you start looking at all the effort it takes, all you're doing in some cases is trading effort from one area to another. And so your timing isn't really going down. So if you didn't invest in the, you know, whatever the cost is for your, compliance automation tool and you're doing everything manually, well, now you're spending hours of people's time going and gathering all the evidence, collecting it, putting it into a place so that your evidence repository, and those people could have been doing something else. So what do you do to fix that problem?
Rick Yocum:You go and you get the compliance automation tool. You go and get the GRC tool. Mhmm. You move off of spreadsheets and more docs into a system. And so what you're doing is trading, you know, hours of a person's time doing something for a system.
Rick Yocum:Now, that system needs to have somebody configure the integrations. And when the integration breaks and it stops working, you need to go and work with whoever it was that was authorized. So you interrupt somebody else. They have the the person who has the ability to go and reconnect that service back to the, you know, the platform that it's gonna Mhmm. That's being monitored.
Rick Yocum:And then, you know, so that's a time. So Yeah. Really is like time trade offs. If you look at the whole cycle, you're just moving things to areas, and you try to get it working as efficiently as possible. So in a best case scenario, you know, you do away with spreadsheets, and now you have it on a system.
Rick Yocum:And then you get that system to be able to create some automation so it's repeatable. And then you need to be able to do your own validation that that stuff keeps being reliable. Yeah. And then that might mean that well, now you're gonna go to the auditor. Well, the auditor sure, you are be able to feed the auditor all this stuff more quickly, maybe in a little bit more reliable way.
Rick Yocum:And I'm thinking about it, Matt, you you might have a different way of thinking about this. But as you're auditing, you have a number of hours you have to do. If all of a sudden, you were given access to a system that said, here's, you know, here's a thousand points in time that this control was operating effectively because it was automated. Mhmm. Now, you're gonna look at that and see Yeah.
Rick Yocum:You know, now now you have some testing to do, like, well, you know, how is that good? And we covered set in the earlier half. And so now, your testing is you're still spending time testing something. I don't see where you're shrink where you're shrinking down the testing period. You're just shifting what you're looking at in order to say that's reliable as you go on to do something else.
Rick Yocum:What are you finding?
Matthew J. Schiavone:I think that brings up a good point with comparing ISO audits to SOC two audits where in ISO you have a required number of hours, man days. SOC two, there's no required amount of hours. So these firms, these a corp can just turn out these reports. And I mean, they were doing ISO certifications as well, so probably fabricating those time sheets and everything. But Joe The
Justin Leapline:corp wasn't. Potentially, the other company was. Sorry. Was it the gen whatever? Yeah.
Justin Leapline:Yeah. Gen Whatever. Yeah. Gen. Yeah.
Justin Leapline:Yeah. Allegedly. Well, allegedly. Allegedly.
Matthew J. Schiavone:But, Joe, to your point in a SOC two audit, we have to examine every control. And if it's you know, we're testing operating effectiveness, we have to sample based on the guidance. So sorry. I kinda lost my train of thought there.
Justin Leapline:Do you make up your own sampling methodology?
Matthew J. Schiavone:We have PBC work papers Yeah. Thompson Reuters' checkpoint. But you document your own methodology and it into there. We justify if we deviate from their confidence levels. So 95% confidence level, 90% confidence level based on the population.
Matthew J. Schiavone:Yep. Justify based on automated controls, level of risk. We can justify, but we have to support that, and that is one thing that peer review drills into.
Joe Wynn:Well, when you start to validate, I assume, the accuracy and completeness of the evidence being provided to you.
Matthew J. Schiavone:Of the population.
Justin Leapline:Yeah. Yeah. Yeah. Yeah. Yeah.
Justin Leapline:Yeah. So exact yeah. If control is highly automated, you got a thousand things, maybe you're only testing one, two, three, you know, of it knowing the reliance of it, you know, full Yeah. You have to test the process more. Yeah.
Justin Leapline:Exactly. Yeah. It might get in reliable.
Rick Yocum:And that was my point. You're you went from just examining the evidence that used to be a screenshot that somebody would do and you might say, well, I have to do my walk through. Show me how you did that screenshot. Show me how you log in and and get that. Yeah.
Rick Yocum:And Oh, I use that a common test you would do?
Matthew J. Schiavone:Yes. And that brings up a great point with automated pipelines and software development. You know, legacy SOC two, we were looking at change tickets and making sure the process is tied out. Now we're a little bit more reliant on the processes and the controls throughout that process that might not be as legacy as a change Right. In
Justin Leapline:Yeah. Good. Yeah. You got pipelines with rules around it. So you're testing those rules are enforced.
Justin Leapline:When were they override you know, overridden, all that stuff and everything. Yeah. Yeah.
Joe Wynn:But to your point, like, if it if it is like, oh, yeah. We use Vanta, Andrata or whatever for, like Ordrata. For, like, 80% of all these controls, it's like, okay. Well, now we need to test the access control to that system, and we need to test the change management of that system. So to your point, Joe, yeah, you're I mean, it's another shift left.
Joe Wynn:It might help, but it just depends on how much it scales based on what it's displacing.
Rick Yocum:Yeah. If you look at the total ROI and the economic value of it
Justin Leapline:Yeah.
Rick Yocum:And you look at everybody's time involved and you look at all the costs for all those pieces of time Yeah. I'm not really seeing how one, you know, one level of effort is different than another.
Joe Wynn:It's like
Rick Yocum:it's like in physics. So if you're walking up a flight of stairs or you take a ramp, the energy expended is gonna be the equal amount from going from point a to point b.
Joe Wynn:Right.
Justin Leapline:Right.
Rick Yocum:And I'm thinking that you're not gonna see the overall. It's just are you paying somebody to do it and not hiring more people internally to get things done? Is it gonna cost more to have the more reliable audit? And what are you what are you gaining somewhere else?
Joe Wynn:Right. So now you're messing with factors that are like one rung out from the actual objective. Right? So the actual objective is a do compliance stuff. Okay.
Joe Wynn:Great. But what's one rung out from that? It's what's the overall complexity of the environment? Alright. Am I making it more complex by introducing this thing that's not directly achieving functions but indirectly achieving all these functions at once?
Joe Wynn:That's typically going to be a con of sort. Right? More complex is bad, often. And then on the other side, it's okay, but am I also instituting hygienic practices and a set of diligence and and setting a series of expectations that are going to build virtuous cycles? I have a friend named Stuart who who talks a decent amount about when you do certain things he's he does a ton of stuff very well, but he's, like, really focused on, like, sock stuff and all that.
Joe Wynn:Like, one of the things that he's seen when he's helped organizations like really level that stuff up is other hygienic practices in IT or other compliance functions will get a little better because the watchers are getting smarter.
Rick Yocum:Oh, yeah.
Joe Wynn:And I was like, oh, that's super clever. And so but but again, you're one rung out from the direct compliance function of this thing. And I was actually wondering, like, from a SOC perspective and and because again, if you can't adjust the baseline of price, like, that much. Right? There's only so much you can do.
Joe Wynn:You'll try to help, but quality work is quality work. It costs money to do quality work. That makes sense. Yeah. So, like, how do you if you're on the other side of the equation for the numerator, how much can we spend on this and you're trying to go from what used to be a low price tag to a higher price tag.
Joe Wynn:How do you justify that? And you're okay. And now you need to talk about, I think, things like, well, what are the virtuous cycles this can set up? Or, you know, how how do how how can we start to see other teams perform better from a overall resiliency perspective because the line two stuff is happening in a more automated fashion or a or a more complete fashion?
Rick Yocum:Well, that one one of the things I was think you just maybe think about was we we have some customers who don't even want to get they're not doing the security because they need the certification or the out of station. They're doing it because they can just get better security.
Joe Wynn:They wanna do good stuff.
Rick Yocum:And they and and and I love that. And so I always look at So they're
Justin Leapline:getting a certification because they just wanna be forced to
Rick Yocum:No. Some of them don't even get it. They just they're like they're putting in the compliance automation tools. They're getting better because they now have tools that can scan everything in like seconds and tell them where there's a problem.
Joe Wynn:It's like auditing a class because you wanna learn not because you're getting the credits.
Rick Yocum:Exactly. And and and then at that point, we're like, wow. Yeah. You're you're doing this so well. Now that you're actually are secure and you have good mature process in place, some of their customers understand that because of some of the evidence they can get.
Rick Yocum:Mhmm. And because they have the meaningful conversations, they don't even always need to get that audit report. They pass the third party risk management with flying colors
Joe Wynn:Right.
Rick Yocum:Because of what they're already doing. And they're like, the minute I'm gonna stop getting through this, and I need to get the audit, we'll pay for the audit. Right. But right now, not having the audit isn't preventing me from closing deals. Once it starts to prevent me from closing deals, then, maybe we need to shift from having as many That's cool.
Rick Yocum:Security, GRC people being able to answer all these, you know, questionnaires. And we'll have in the SOC report, we'll have in the ISO certification lower our burden, our friction to closing deals, then we'll then we'll get it.
Joe Wynn:Oh, that's such a cool point. Yeah. So what's the tipping point, you know, from just have a good program to, oh, I need the report to the stamp?
Rick Yocum:Because what's gonna be worse for that customer is they they're developing the the tool. They're developing the software. They deploy that software to their customers. Some some of these in a way that they deploy them, every customer gets their own instance of it Sure. And the customer has full capability to scan this thing for Vault.
Rick Yocum:Mhmm. And the minute that their tool or that their product is known to be deficient is a reputation problem. They're like, what's worse to us isn't any of this other stuff. It's reputation issue we'll have if we if one of our customers find that we didn't do as much as we could have Right. In order to prevent something and they get breached.
Rick Yocum:Mhmm. And so they're like, that'll kill us. That'll be the end. Yeah. So they're doing security for the purpose of doing good security because that's what they need to do.
Justin Leapline:We did at Episode, it was a few months ago, but we put in a zero bug policy, you know, and we count security issues as bugs, you know, into that. Yeah. So any bugs, basically, have a shelf life at most a week, you know, into that more
Rick Yocum:So any known any known bugs
Justin Leapline:Any known bugs.
Rick Yocum:You you resolve them in a week.
Justin Leapline:Yeah. Wow. So anything like that. More critical, the higher visibility, obviously, they are. But if it's like a CSS spacing issue, even that gets fixed within a week.
Justin Leapline:Mhmm. Because, again, you're looking at quality, you know, and security issues are just quality issues.
Matthew J. Schiavone:You know?
Joe Wynn:Well, it's either fix it faster, start maintaining a giant backlog that just gets bigger.
Justin Leapline:Right. And then all of a sudden, look back at the ticket a year later and we're like, why did I create that? Or, you know, or it didn't get flagged and fixed, you know, into that. It's just there's a lot of reasons for it that we did it, but it's it's worked out great. You know?
Justin Leapline:And now anything that comes through, we're fixing it before it gets on production or if it sneaks into production. Actually, just this week, I I just hooked in that the late to the game, other people have done this. But any issues issued from Sentry automatically get triggered, and an AI agent automatically creates the fix, commits the PR, and then whole bunch of, like, things get reviewed. And, basically, the only thing for me to fix it is a hit except, like, to merge the PR.
Matthew J. Schiavone:Yeah.
Justin Leapline:Yeah. Like, that's the only thing.
Rick Yocum:And I So you're the human in the loop. You check it. The
Justin Leapline:argument's good. Human in the loop. Like, yeah. And I'm even at some point, I might even turn that off, you know, because we have a whole bunch of validation. We have end to end tests.
Justin Leapline:We have our, like, our the relational security test.
Joe Wynn:Other automated stuff would fail.
Justin Leapline:Yeah. All of it is tested on HPR going into into that. So we have all those tests, and all those have to be passing before I even have the option to hit merge, you know, into that. But, yeah, at some point, I'm I'm still getting comfortable with it, you know, type of thing. But at some point, I'll be like, now if it's a bug and you're confident, you fix it, yeah, do it, you know, type of thing.
Rick Yocum:Well, that's cool.
Justin Leapline:I see.
Rick Yocum:Matt, I saw you were gonna comment on something earlier.
Matthew J. Schiavone:I was gonna ask in those instances where the organizations just wanna do security, they wanna do it well, they still get through the third party vendor risk management process effortlessly. How what is assisting them through that process so easily? Is it just their ability to answer the questions? What what evidence are they Well,
Rick Yocum:it's it's because they have a a strong second line of defense in place. So if you're familiar with the three lines of defense, the first line is doing the work. The second line is validating that the first line's work is being done properly, and that's all done even before an internal audit comes in. They do have audit, but it's more like an internal audit function. It's not important to them to go and get that external audit, that external validation because everything management says, we're pretty comfortable because we have the three lines of defense operating well.
Rick Yocum:And, you know, just just cheap plug for CISO. That's one of the things we do. We focus on being that second line of defense. So, Matt, you didn't set me up for that intentionally, but thank you. And so that that's what they have.
Rick Yocum:They have basically a team of people who are super knowledgeable on what to look for, how to push issues both to the other teams, how to get things escalated, how to create the right awareness with top leadership so that top leadership understands, well, what is the risk? Is that a business risk? What will be the business impact if that thing doesn't happen properly or if that risk comes to fruition? Then it becomes like, you know, a nonconformity and
Joe Wynn:Right.
Rick Yocum:The whole idea is before that nonconformity becomes an incident or even worse turns into a data breach, top leadership says, get us involved. We will make sure we stop doing the things we're doing, so focus on fixing this so we can get that risk back into tolerance and and and and stop it from becoming a nonconformity. So those are the things that they're doing right.
Joe Wynn:Well, and from, like, a prospect mechanics perspective, do those clients typically have something like a a standard attestation package that they'll use to send to their prospects? Like, so CISO is working with a client, say, and that client has clients then
Justin Leapline:Oh, yeah.
Joe Wynn:Do they have, like, a standard
Rick Yocum:Well
Joe Wynn:set of things often?
Rick Yocum:Some of the trust portals that are out there Yep. Are actually looking at real data in the back end on these systems Mhmm. And giving you actual real data Live or near
Justin Leapline:live data.
Rick Yocum:Yeah. And so when you see that it's working Mhmm. And you trust the system and you know there's a human in the loop
Justin Leapline:Yeah.
Rick Yocum:Then it creates a little bit more reliability. So that's what you're getting.
Joe Wynn:Yeah. The trust portal mechanics, super. Yeah.
Justin Leapline:So you guys like them? Like what? Trust centers, trust portals?
Rick Yocum:Oh, when they're done well.
Justin Leapline:Yeah. Yeah. That was actually a thing in the Delve report in the first one was that when they first set it up, everything was green before they even entered any information into it. I was like,
Joe Wynn:starts green. Starts green.
Justin Leapline:But, you know, I'd like one of the things so well, we're designing a a feature within Piskey to have a trust center Yeah. You know, into that. One of the things that's interesting is because when have you logged into a trust center and someone has been read? Right. Have you ever thought of that, you know, type of thing?
Justin Leapline:Have you ever seen a control failing in their list of assurance controls and being tested? The most likely not, but that doesn't mean, like so most will remove it.
Rick Yocum:Right. Remove it. Yeah. I
Matthew J. Schiavone:see that.
Justin Leapline:Know, into that so that everything will still show green, you know, but So what don't know what's there or not. You don't
Joe Wynn:know what was there yesterday.
Justin Leapline:Right. Exactly.
Rick Yocum:And so what you wanna do in those situations when you're the customer and you're doing your TPR, your third party risk management, If what you're relying on is their trust portal, then how are you logging into it? What are you capturing? What are your snapshots? Are you doing a screenshot of the trust portal? Is this risky enough vendor that you're using that you should need to look at them more frequently.
Rick Yocum:So Right. You know, then have you compared last screenshot to this one? Right. Is it still have the things on it that you thought were important last time? If not, that's not the end of the story.
Rick Yocum:You don't just say, oh, yeah. I don't see the thing. It's now I need to have a conversation.
Joe Wynn:Right.
Rick Yocum:Now I need to call up and talk to their security team, understand what's changed.
Justin Leapline:But it's hard to identify those. Like, they'll have, you know, four or five dozen controls already there in all the various groups and all that stuff. How do you know one disappeared? You know? Ear chat GPT.
Justin Leapline:Tell me
Rick Yocum:Yeah.
Justin Leapline:Tell me what's different in the screenshots. Go to the way machine. Right. Well,
Rick Yocum:no. No. If you're doing this, what's your evidence that you so if you're the GRC analyst reports,
Justin Leapline:you know, that mainly you're going up to the trust portal to download their SOC two or ISO certification or whatever it may
Rick Yocum:be. But you also should look at the controls that are they're saying are operating effectively.
Justin Leapline:Yeah.
Rick Yocum:And if they're on there, and it and if it is indeed the situation where it removes the thing that's not being done anymore
Justin Leapline:Yeah.
Rick Yocum:And you actually cared about that thing, your TPRM process should be saying, oh, these are what I'm relying on. I'm relying on this report, or I'm relying on not only the reports I can download, but I'm relying on the controls that were displayed to me, screenshot it. And then next time,
Speaker 2:what's changed?
Joe Wynn:I smell a new I smell a new application that just evaluates trust portals that are out there for changes over time.
Justin Leapline:Yeah. Or maybe if you have your standard controls, it should go up to the trust portal and actually look at it and say
Joe Wynn:Yeah. If you have
Justin Leapline:any requests to do a pen test once a year, and maybe they're a month late, and so that drops off their Right. Annual pen test requirement. And you go up and, like, well, I require a pen test once a year. Because maybe the control is meaningless to you. Maybe.
Justin Leapline:Yeah. You might not care. Yeah. Exactly. Depends on what it is.
Justin Leapline:But if you had maybe AI going up and saying, here are the controls I care about. Tell me if they're operating effectively in this trust portal here. And then you get kind of a comparison to say, I don't see this here, and it might be they just left it off and never had it, you know, or they're deficient at this. Right.
Rick Yocum:Right? And then that evokes a conversation.
Joe Wynn:Yeah. Exactly. But that conversation part's critical. Right? Because if it's just if you're just always looking for green Right.
Justin Leapline:You don't have time to do something about it when it's missing a red. It's just theater. You're just Yeah. That's often when when we talk about the bit site or security scorecard and stuff of that nature, it's it it might be an alright tool. Like, I really appreciate it.
Justin Leapline:There was one customer I had it in place with, and Log four j came out, and we're able to see all the vendors that we worked with that actually had an active export that actually, you know, version of that that was actually publicly accessible. Obviously, it's not all of them, but we were able to initiate a communication and be like, hey. Go Oh, yeah. For sure. You know, type of thing, which was really handy.
Justin Leapline:But for the most part, the only thing I really use it for is to get kind of a finger in the air. Are you doing patch management well or not? You know? So if I see their public website riddled with old patches, then maybe their SaaS app that's behind the off, you know, block Right. Maybe they're not doing too well on patches either, you know, that I can't really see, you know, from the view of a public website, you know, type of thing.
Justin Leapline:So that, again, initiates a conversation. Be like, you got struts open. You got this open. You got that open. You tell me you're doing patch management.
Justin Leapline:What's going on here? You know?
Joe Wynn:But I I think a lot of GRC teams are are staffed for business as usual. Mhmm. And if business as usual means things are green and you don't actually have enough time in the day to react to when things are red, like, that's an issue. You're need to talk about that.
Rick Yocum:Yeah. So, Matt, got a question for you. You know, what should companies be asking their auditors when they're looking to hire an audit company?
Justin Leapline:What what what Oh, like what what
Joe Wynn:makes a good audit firm?
Rick Yocum:Yeah. Yeah. What would you say that you'd expect customers or potential customers to be asking?
Matthew J. Schiavone:Experience with certain technologies, industries, you know, you wanna make sure they have that background. I think their methodology for just collecting evidence, conducting walkthroughs, unfortunately, you you do want that hands on approach to a degree. There's value there.
Justin Leapline:So the assessment methodology overall?
Matthew J. Schiavone:The assessment methodology. Communication protocols, I think when you have centralized communications, regular communication touch points, I think that just kinda demonstrates a commitment to quality and keeping things on track and moving from an audit firm's perspective. If they're reluctant to do that or they don't offer that kind of regular touch points, I mean, I think it just indicates that they're just maybe, you know, not as committed to the engagement as
Justin Leapline:When you say communication, are you talking about, like, in between year, like, touch points and evidence collecting, or that's something separate?
Matthew J. Schiavone:Both. Okay. I think throughout the engagement and then throughout the year. Yeah. I like to have quarterly between audits, even maybe sometimes monthly just say, hey.
Matthew J. Schiavone:Have there been changes? Have you experienced anything? Do you need help with anything that wouldn't empower independence?
Justin Leapline:Any questions? Any evidence during that time?
Matthew J. Schiavone:We typically don't unless we're engaged to conduct interim testing Okay. In which sometimes we are Yeah. To lessen the burden of field
Justin Leapline:work. Exactly. Yep. And I think that's actually a good I think out of this here, people should actually do more interim testing, you know, into that. I think that's a just a great practice to get into.
Justin Leapline:Not only like, it can't be that much more expensive.
Matthew J. Schiavone:Right? Like Oftentimes, we don't charge more. It's just a matter of timing.
Justin Leapline:And then you're not getting the qualified opinion surprise. Like, if you catch it in first quarter and
Joe Wynn:You can still fix it.
Justin Leapline:Yeah. It doesn't turn into a big issue, you know, into that. So yeah.
Matthew J. Schiavone:I think the problem is people just don't like audits, and they don't wanna be burdened with it. And the other the other value of keep keeping in touch with the auditee throughout the year is there's numerous times throughout the year. We have those ongoing conversations, and they told us about a migration to a new platform or technology. And we have to remind them that they have to retain that legacy evidence before the migration because it's in that window.
Justin Leapline:Right.
Matthew J. Schiavone:Had we not done that or, you know, had they not realized that Right. When audit time comes, that's a scope limitation, and that's automatically a qualified opinion. Mhmm.
Justin Leapline:Unless they cut the scope cut the time frame. Yes.
Matthew J. Schiavone:Yeah. Yes. Unless they cut the time frame.
Justin Leapline:So Which how many people look at it and actually know the time frame into Well,
Matthew J. Schiavone:I don't know. I mean, how
Justin Leapline:You you do. And I know everybody probably at this table, but we've talked about, like, they delete the findings out of it and customers Well look at
Matthew J. Schiavone:Rue, that's a good that's a good third party risk management practice question. Are you comparing last year's time period and making sure it's concurrent Right. Continuous?
Joe Wynn:I was actually wondering that as a potential, like, cost shifting measure potentially is like, well, if you're trying to justify a higher price tag, could you do a bridge letter for six like, do hey. Do the full thing for a year, and then, you know, just have a six month bridge letter, and then you start the next one. Right? So you're more like eighteen months apart as
Justin Leapline:opposed twelve months. Letter for five years? Is that what Yeah. Right. Got one five years ago.
Justin Leapline:Here's our bridge letter.
Joe Wynn:So mark of a bad firm signing a four plus year bridge letter.
Matthew J. Schiavone:It is management of the organization's responsibility to define the period. We have instances where people just do a nine month period every year. Yeah. January to October or January to September One. Every year.
Matthew J. Schiavone:I think they want the break. Just to I mean, honestly, I don't know.
Justin Leapline:So that they don't collect. I
Rick Yocum:don't know.
Joe Wynn:We're getting tired
Justin Leapline:of all this evidence collection. Okay.
Joe Wynn:If you're gonna break the rules, these are
Justin Leapline:the three months. This is busy. These are our three cheap months. Yeah. Now, that being said, we, of course, have
Matthew J. Schiavone:we have internal processes to evaluate the risks.
Joe Wynn:Of course.
Matthew J. Schiavone:We see, you know, a system shift in those nine month that next period. It's an elevated risk. What are they hiding? And we have to dig dig in the controls a little bit further. You know, we see six month periods.
Matthew J. Schiavone:We see type ones every year.
Justin Leapline:Yeah. Right. Yeah. And I can understand the abridged, you know, like, time frame, so especially when you're starting off, you know, or I mentioned that time, like, do we hurry up and get another one because of our first one? Yeah.
Justin Leapline:Get a clean one? I mean, there's wiggle room, you know, into that. But, yeah, I haven't heard the nine month one that's on a ongoing basis. Because once you get settled, it's a annual review, typically.
Speaker 7:You know?
Justin Leapline:Should be.
Matthew J. Schiavone:I mean
Justin Leapline:Typically. That's what customers are expecting for the most part. Yeah. Yeah. I wonder how many of those customers actually know that they're only getting a nine month one.
Justin Leapline:I'd be curious.
Matthew J. Schiavone:I out of I'll say out of 500 reports, round numbers that I've issued, two or three times have I seen, like, a nine month rate.
Justin Leapline:Yeah. Yeah. But I'm saying the the customers they hand that report to, do they even know that it's a nine month report, not a an annual report, you know, type of thing. Yeah. I don't know.
Justin Leapline:Yeah.
Joe Wynn:Yeah. Interesting.
Justin Leapline:So trust centers, alright. If I like them. Programmed sufficiently. I like them as a communication point. Honestly, there's a lot of faults that I'd look at from a people running security programs.
Justin Leapline:And I think one of the biggest things is that they're not espousing what they're doing more often, both internally and externally. You know? Oftentimes, as you know, security is the Red Hat stepchild in the back, like, don't make too much noise. We're required to do this. Just, you know, keep doing what you're doing.
Justin Leapline:Get out, like, all that. But, like, all the good stuff like, we see the the success stories every single day with our clients and customers and our companies that we're working with. That should be communicated out, you know, all the wins that we're getting, all the customers, like, we've we've won clients because we were superior when it came down to between two big contracts, we had better security, and that was a defining factor of why we got the contract. Right. You know, type of thing.
Justin Leapline:And people don't realize that, that that's a selling point. So
Joe Wynn:if The burden of explaining if you're if you're the the ultimate client choosing and you're in a highly regulated industry for for instance, the burden of explaining why you went with the less secure partner can be pretty heavy.
Justin Leapline:Yeah. So you got two final ones. You want got one from Dell's and got one from Matt. You know? Which which one are you gonna go with?
Matthew J. Schiavone:Question on trust center,
Justin Leapline:Yeah.
Matthew J. Schiavone:Yeah. How can a prospect ensure that the trust center scopes appropriately? Because to your point earlier, if you're in a platform, you might not be covering all the AWS and Yeah.
Justin Leapline:And that's a big thing. They'll actually the little notes, you can actually customize and put in what you're covering into that. And, actually, I really like a lot of the things. Vanta actually has a really good subprocessor, third party, however you put it. They'll actually say what the company is, how they're using it, what the region they're using it in, and what for what service and all that stuff, and they update that list.
Justin Leapline:And you can get notifications when they change. Same thing with the controls. You can say, here's the controls of, like, what we're doing and what services they apply to in, like, a an upper statement into that. Because it could be not the entire company. It Right.
Justin Leapline:Know, it's it's a public facing thing. So more than likely, it's gonna be customer. If you can apply it to the entire company, again, if you're a, you know, a a SaaS company, that's a lot easier, you know, done than if you're, like, a multiregional hospital system, you know, type of thing that's a little bit harder. Right. You know?
Justin Leapline:And I probably wouldn't wanna list out all my controls because there'll be a lot of deficiencies, you know, in various areas, you know, type of thing. So yeah. So it all depends. Yeah.
Joe Wynn:Okay. But yeah. I it's funny. You were talking about internally and externally. One of my clients, we were just talking about on putting on the road map for the coming year, an internal trust portal that in two to three years, depending on how it goes, will likely become a portion of it will likely become an externally facing trust portal.
Joe Wynn:But yeah. Gotcha. So a couple
Justin Leapline:of FastFire things and everything. Do you think there's room for, like, a consumer reports type thing for good companies out there or a BBB or a publicly run rating system for companies.
Joe Wynn:Like, under the other one?
Justin Leapline:No. No. No. But, yeah, I guess it would be more for the security realm. Not necessarily the company itself, but, you know, like, you know, I got a, you know, a bad report from this company, you know, and here's all the details.
Justin Leapline:You
Joe Wynn:know? Angie's list for security and compliance things.
Justin Leapline:Is there room for that? Is it to do think
Rick Yocum:I don't know. I I see a lot of companies signing NDAs in order to Mhmm. Get these documents. And even if you look at what you get from, say, Azure or AWS, you're you're you're able to get their SOC two report. But you signed an NDA, and then that it base it it pretty well says that this is for you to evaluate
Justin Leapline:in their term buried in their terms condition.
Rick Yocum:You you're you're clicking through and you're Yeah. You're agreeing. So it's a legally binding agreement that for confidentiality, the stuff you're getting in. When they give you their SOC two and let you download it, you're obligated to protect that because that's what you agreed to in exchange for the service. Oh, that probably wouldn't work.
Joe Wynn:Yeah. What's what's your what's your cease and desist tolerance?
Justin Leapline:Yeah. And so Exactly.
Rick Yocum:And and so, like, I have an instance of a customer who their their customer's third party or their customers was asking them to provide the evidence for, you know, their security controls. And and I'm not sure why they asked for this, but they said, oh, well, if you're using cloud services, and you then you're probably getting their SOC two, just upload those to our portal so we can see that you're doing it. And they're like, no. We can't share AWS's SOC two with you because we signed an agreement or click through an agreement that said that we're the I'm not gonna. It's for us.
Rick Yocum:Yeah. It's not for us to pass on. And so that was explained and
Justin Leapline:It's funny.
Rick Yocum:That the customer
Justin Leapline:You get them passed on quite a bit. You do. You do.
Joe Wynn:That's a it's a good practice.
Justin Leapline:Yeah. Let's talk to yeah. Here it goes. This is a Yeah. AWS made it.
Justin Leapline:Yeah. Yeah.
Joe Wynn:In fact, I've seen a lot of instances where people will just try to rely upon that as opposed to the thing that they built
Justin Leapline:on top of that. The time. Yeah.
Rick Yocum:Oh, yeah. That does happen all the time. And that's just uneducated customers who's not they don't they don't even know what the SOC two is achieving for them. They just know they should ask for
Joe Wynn:it. Yeah.
Justin Leapline:So one of the things I think, you know, I I bring up that consumer reporting. One of the fixes that AICPA could do is more improve their website where they list some of their quality reporting and actually more publicly shame and or control their quality into that. Like, if companies were actually afraid of doing poor quality, you know, and actually being named, you know, more viciously, I think there'd be it would settle out some of these quick one week SOC two, maybe. I don't know. Ecosystems are weird, man.
Joe Wynn:It's like it's like pushing on one part of a balloon and another part, you know, gets bigger because, like, I I don't know. The AICPA also is gonna have its own risks in terms of making its clients who are the AICPA's clients ultimately. Right? So who's paying the fees, all that stuff. And so you make it too hard on them or too aggressive, and now there's room for another one.
Justin Leapline:Yeah. I know. And it's a balance. It's a Yeah. You know, into that.
Justin Leapline:But I think to you know, the pendulum has swung very far this way.
Joe Wynn:They do have to defend the rep. Back. Yeah. To defend the reputation of
Justin Leapline:this Yeah. Exactly. So I think public shaming is probably the least, you know, into that. I mean, it and it would be for you have a whole bunch of serious bad stuff and make it a I mean, they do have a portal and everything, but it should be prominent that, like, when I go up and look for a new CPA, you know, I I should check this portal first Well you know, type of thing and seeing, like, what's their quality like, you know, into this.
Joe Wynn:Well, at the risk of g r's poor beleaguered GRC teams having to review yet another thing, Should it be normalized, or would anyone even do it if someone was like, yeah. Thanks for that SOC report. I want to see the last AICPA peer review of the firm that performed it. Like, should that be a default appendix in stock reports or something like that? Or at least a summarized, you're right, de identified version of something like that.
Joe Wynn:Should there be a report card of the the whoever's Of the auditor. Yeah. Of the auditor. Yeah.
Matthew J. Schiavone:We commonly include that in proposals.
Joe Wynn:Yeah. But
Matthew J. Schiavone:doesn't make its way to the reports and really proposals is all.
Rick Yocum:So that's another thing when I, you know, when I'm asking like what what should you ask your auditor is, you know, is it possible for me to see your last peer review?
Joe Wynn:And and what would we ask for? Would you ask for the peer review or is there a specific letter
Justin Leapline:or a specific statement? Can go get.
Matthew J. Schiavone:There there's the peer review portal and then there's there's a letter that may or may not be uploaded to that portal. Okay.
Justin Leapline:So So Yeah. But I think, yeah, I I think they need to improve that a little bit, you know, type of thing, where it's a lot easier to check the quality of who you're considering or who you're in bed with, you know, into that to, you know, validate it.
Joe Wynn:Yeah. Or at least well known. I mean, there's a lot of people that wouldn't know about that portal. Yep.
Matthew J. Schiavone:I also think one of the AICPA's problems this is not a knock on them, but remember, it's accounting wide. So we're focused here on SOC two and security.
Justin Leapline:That's absolutely true.
Matthew J. Schiavone:They have a much broader scope of services that they're concerned with. They've labeled SOC two as a high risk because they have the so they have the enhanced oversight function. But, you know, we're fixated on probably just this very small segment of what the a v AICP is overall.
Justin Leapline:And that is true. And I've actually
Joe Wynn:Go ahead.
Justin Leapline:Seen some of the especially back in the SaaS seventy where security was kinda intertwined with some of that financial, you know, reliance controls and all that stuff. We actually in another security firm I was with, we got hired I was a security consultant under a CPA to test out the security control.
Joe Wynn:Oh, yeah.
Justin Leapline:Because they're like, I don't know. Right. Because they're tech controls. I know numbers. You know?
Justin Leapline:Like, I'm checking their general ledger and, you know, some of the reconciliation stuff or, what about this passwords and all this stuff? Like, I still feel like it's kinda like that. You know? Like, your group is specialized, you know, into this, but another CPA firm might not have that skill set. You know, a three person office, they're like, we could do SOC two too.
Justin Leapline:You know? Why not?
Matthew J. Schiavone:Sure. And in our planning, we have to well, we have to document that we have the competence, we have the experience, or maybe we're out we're using outside resources
Justin Leapline:Yeah.
Matthew J. Schiavone:Because we don't. So those are all considerations that we have and document during the planning.
Justin Leapline:But that doesn't I mean, you say that's all documented. There are other companies. I'm sure they're out there that they they think they're competent. You know? They saw a YouTube video out, you know, now that that they're competent, you know, type of
Matthew J. Schiavone:thing. Sure. Sure. But that's part of, you know, the peer review process. Do you actually have competent people?
Matthew J. Schiavone:Yeah. Know? But again, to our point, the peer review process once every three years after eight or eighteen months after the first report. So But I love
Joe Wynn:the point you made about AICPA's overall remit being like accounting wide. That's super interesting. And I would guess, although I don't know this for sure, the peer review processes for these SOC reports align to the generalized peer review processes for high risk stuff in general. And I actually wonder if this is an opportunity to say, oh, you know what? This is specialized enough.
Joe Wynn:It needs a slightly different process for tech side peer review stuff. Because I mean, how quick does technology change?
Justin Leapline:That's Like, yeah.
Joe Wynn:Interesting. No. You're you're absolutely right. It's three year process for Anything high risk.
Matthew J. Schiavone:Yeah. Well, not just high risk, any attestation, any assurance, you know. Yeah. It's all included in peer review. They've just labeled SOC two, SOC one as a higher risk area.
Joe Wynn:Gotcha. So that that doesn't even necessarily mean it gets a higher frequency. It just No. Interesting. Yeah.
Matthew J. Schiavone:Yeah. The sample selection rate is probably higher.
Joe Wynn:That makes sense. Yeah.
Matthew J. Schiavone:Of course. But same frequency.
Justin Leapline:So that's what that dictates. Higher risk means we're testing more companies and everything. Got it.
Joe Wynn:Interesting. Oh, that's good. Any So if I'm a security nerd, here's here's a question I
Justin Leapline:have for just the panel. You are. If I'm a security well, yes. True.
Joe Wynn:But if if I am in this situation, what do I do about it? I'm not a GRC guy. I'm just a security guy. Is there anything I should be doing as a CSO who relies on a chief compliance officer over there to to do the compliance work? Like, is there anything
Justin Leapline:for there and as in another company?
Joe Wynn:Well, whatever. It could be virtual. It could be internally. But someone else is taking on taking on the third party risk stuff and all that stuff. I'm just responsible for the perimeter and the crunchy bits, right, of of technology security.
Joe Wynn:Is there anything I should be thinking about or looking at internally? Because I because our audience is kinda made up of security nerds and GRC nerds, I think. So there's probably some security nerds being like, yeah, and and and bully on you if you've listened to two hours of this
Justin Leapline:Yeah. Right. If you're a security guy. And you're talking about the security nerd. Are they getting a SOC two or some type of attestation?
Joe Wynn:I don't know. They they just they're they're they're being asked by their executives, do we need to think about any internal controls, or do we need to do anything in response to this?
Justin Leapline:I I honestly think that's absent in everything we talked about here. That's just a good security program. You know? You should be looking at the what controls you need to put in place to minimize the risk that you're most susceptible to and put those in place. Forget about any ISO, SOC two, anything along that lines.
Justin Leapline:It doesn't matter, you know, at that point.
Joe Wynn:But so does this potentially change the risk ranking of your vendors and who you're paying attention to and stuff like that?
Justin Leapline:I I I don't think the the vendor's risk is, you know, dependent on the vendors and how you're utilizing them.
Joe Wynn:That's probably poorly put. Yes. The risk the general risk would be dependent on how you're using them, but your ability to rely on the reports coming out from them might be put in question. And so
Justin Leapline:Depending on the quality of the auditor, you know, because How do know that? To that. Yeah. And that's where I was like, there should be a validation, you know, into that. I think, you know, I don't think unless AICPA does something, you know, specifically to SOC two, I don't think there'll be a big market shift.
Justin Leapline:There'll be a little bump, you know, right now where companies are like, I don't like, you know Mhmm. This one company and let's shift around. But in six, nine months from now, it's I think we're gonna settle back into the same thing, you know.
Joe Wynn:There'll be a breach next month where we like
Justin Leapline:Well talking about that instead. Yeah. Exactly. You know, type of thing.
Matthew J. Schiavone:I I don't know. I
Justin Leapline:You think it it's here to stay?
Matthew J. Schiavone:Am I I don't know.
Justin Leapline:I mean, it's it's
Matthew J. Schiavone:still thinking.
Justin Leapline:We'll we'll we'll come back on
Joe Wynn:I like optimism. Yeah. Exactly. I like optimism.
Justin Leapline:I don't know. I look at everybody's memory so short, you know Right. Shortchanged right now. So it's like, well, this is a really big concern and then, you know, 2026 comes as like, I just wanted $5. Sock too.
Justin Leapline:You know? Like The buyer of the SOC report
Matthew J. Schiavone:is not gonna change. The auditor's not gonna change. What's gonna have to change I think is the AICPA a little
Justin Leapline:bit That's what I'm talking about.
Matthew J. Schiavone:And mostly the user of the reports.
Rick Yocum:Right. If they start The ultimate customer of the report Ultimate customer. Isn't paying for it. They're getting it handed to them when they ask for their vendors
Matthew J. Schiavone:reports. I wonder rejecting these and it's gonna shift.
Joe Wynn:Yeah. And I wonder if I some of the
Justin Leapline:don't see that. I don't know.
Joe Wynn:I think that the the driver of that would be could see if some big players if the right people get pissed off at big players, and we're talking about like the the Westinghouses or, like, or whoever, like, the the Northrop Grumman's, the, like, the really big, big people that, like, look at tons and tons and tons of these. Right? Really any probably Fortune 100. If they start going, yeah. You know what?
Joe Wynn:No. We're gonna start accept reports. We're gonna start requiring ISOs. No. Just we're gonna start requiring a different we're we're not interested in SOC reports anymore.
Justin Leapline:Yeah. I don't think they're
Rick Yocum:necessarily gonna fix the problem because you're just getting somebody else's report that could also be bad from a different standard.
Joe Wynn:Yeah. It's true. It's it's slightly it's I don't know. It's managed differently though. Right?
Joe Wynn:Like the whole ISO ecosystem like we talked about before
Justin Leapline:It can be. More structured.
Rick Yocum:It can be, but like we saw in the evidence, if, you know, if everything in this report is true, they were getting both ISO and
Joe Wynn:those reports. Yeah.
Matthew J. Schiavone:Yeah. So if not, why else would these customers of Dell be scrubbing evidence that they were audited and supported by Dell? I think because they know that it's not gonna be accepted, there's a risk. They obviously don't want their name associated with
Justin Leapline:Yeah.
Matthew J. Schiavone:With with the incident.
Justin Leapline:I think the negative publicity with getting a cheap report Yeah. Today, you know, type of thing. But do you But if big players stop accepting a SOC too Right.
Joe Wynn:That's an issue.
Justin Leapline:That's a big thing to do. Can you imagine, like and what if you're already in bed with them? Like, it's it's not an easy thing to rip a vendor out depending on how embedded they are.
Joe Wynn:I have seen really big players push around small players quite a bit.
Justin Leapline:Small players
Joe Wynn:In terms of GRC world.
Justin Leapline:It depends on how embedded they are, you know, type of thing. Oh, yeah. Sure. They're like, you told me an idea of SOC two. I was like, well, I have one from Delve.
Justin Leapline:Well, we don't accept that.
Joe Wynn:Yeah. But then I go Well,
Justin Leapline:you didn't tell me that. I I have one. You told me I needed one.
Joe Wynn:Well, it's not gonna be no. We're just letting you know in, like, two years, it's not gonna be accepted anymore.
Justin Leapline:Okay. And then maybe they have if they give it a nice time frame, then, yes, so they they could switch. You know? Yeah. Yeah.
Joe Wynn:I think that's what shifts the market though. When the really big players are like, I don't know that we're gonna rely on this anymore.
Justin Leapline:I don't see that shifting. I don't see that yet, at least.
Joe Wynn:Oh, I haven't seen anything to indicate that. I'm just suggesting like, you know, that that would be to me the turning point. If one or two big players start to shift on that Yeah. It's like Yeah.
Rick Yocum:That's that's what would have to be true
Joe Wynn:Yeah. In order for
Rick Yocum:this to happen.
Justin Leapline:And I think for that to happen, again, I think if AICPA did a little bit more on shaming the companies, you know, a little bit more prominent, then it would be easier for companies to look at the the blacklist or the naughty list, you know, into that and say, okay. Yeah. Anybody AICPA has had, you know, three egregious reports or one in the last 20, you know, you know, twenty months, we're not accepting, you know, or something like that. I don't know how No.
Joe Wynn:You're right.
Justin Leapline:Work. But then you can obviously do some quality to say, we know there's some bad ones out there, and this is where we don't accept it. You know? Yeah.
Joe Wynn:Well, I think you're right. And not to do too much crystal balling here, but for if if a big player was gonna like, we all know, like, it you should be doing good security stuff, but one way or the other, this is like a liability game. Right? Mhmm. And you get third party attestations to shift risk and liability and all that stuff.
Joe Wynn:And so if at some point it becomes, again, too difficult for some of these big, again, call them Fortune 100 players to defend taking SOC reports from a risk reduction perspective because the AICPA hasn't responded appropriately to something like this, then I think they'll just start to shift because they're pay they're spending so much in lawyer fees to defend why it's okay for them to accept SOC. He's like, yeah. Yeah. Never mind. We'll just make everyone change.
Justin Leapline:Yeah. I mean, we see this in every industry, though. Like, PCI was the trust waves before they rebranded, you know, into that.
Joe Wynn:That's true.
Justin Leapline:You know? Was That's absolutely right. A cheap PCI player. They, you know, beat out the market by an easy $8.09, $10 a lot of times, you know, into that.
Joe Wynn:And then what so then did how did PCI respond to that? I mean, they they made their
Justin Leapline:They audited them, but then they paid them a lot of money in sponsorships and a lot of their conferences and all that stuff, and they stuck around, you know, for the longest time till they got bought out by some Japanese company and got shifted around.
Joe Wynn:But Yeah.
Justin Leapline:Yeah. That they're they're they're still around for what's the company? I forget. The the owner shifted to another company. He basically did the same thing.
Justin Leapline:You know?
Joe Wynn:Is PCI different, though? Because there's no other game in town. Right? So like
Justin Leapline:It was mandated. Yeah.
Joe Wynn:That's what I'm saying. Like, yeah, the the the brands are just like We're not. Yeah. Do it or don't take cards. Right?
Joe Wynn:Whereas this, like, this is all about like, oh, can you ultimately trust
Justin Leapline:a
Joe Wynn:vendor while you could do just your own questionnaires and say, yeah. We're not taking SOC two's anymore. Or you could take only ISO, or you could probably do a couple different things.
Justin Leapline:Yeah. But, I mean, this I think the same thing will happen with CMMC once it gets, like, some legs and everything. There'll be a natural degradation of the audit firms. You know? They'll they'll get away with what they can get away with.
Joe Wynn:The incentive alignment structure is built to be a race to the bottom unless someone enforces the quality.
Justin Leapline:Right. Exactly.
Matthew J. Schiavone:I don't know who and I well, I don't recall who, but today, I did see a CMMC in thirty days. So it's it's already started.
Joe Wynn:They're called dive. We're we're gonna we're gonna come out
Justin Leapline:with the CMMC in twenty nine days now. Yeah.
Rick Yocum:You can't do.
Justin Leapline:Can't do it. You can't do it. You
Joe Wynn:it. Audit.
Rick Yocum:Yeah. There you go.
Justin Leapline:Alright, gents. Any last thoughts here? Anything like that?
Rick Yocum:Yeah. Matt, how do people get ahold of you in order to see if you can give them a quality bid for their SOC two or
Joe Wynn:SOC two in the right number of days.
Matthew J. Schiavone:Yeah. I appreciate that.
Justin Leapline:That's just yeah. Yeah.
Matthew J. Schiavone:Sorry. Matt.chevon@sickagecom. I'll probably be linked in a LinkedIn post.
Justin Leapline:Yeah. And we'll have them in the the YouTube show notes and everything else. So we'll have your LinkedIn into that.
Matthew J. Schiavone:Or go to our sickich website, sikich.com, and you'll find me there under third party attestation. So thanks, Joe.
Rick Yocum:Alright. Yeah. Thank you. Thanks for coming.
Justin Leapline:Yeah. And thank you everybody for joining us here. Don't forget to like, comment, and subscribe. This will be coming out in a well, it'll be coming out now because you're viewing it right now. So cheers, guys.
Justin Leapline:Until next month. Cheers.
Rick Yocum:Cheers.
Joe Wynn:I gotta reach.
Creators and Guests
