Podcast Episode 2: We Spent a Church’s Yearly AI Budget in One Weekend. Here’s What We Found | Ep 224

Description

AI got very good at surfacing security vulnerabilities fast. In this episode of Rock Cast, Jon, Emily, and Nicole share the story behind Rockʼs recent security releases.Jon explains how the team built a series of agents to scan Rockʼs code, why Rockʼs flexibility makes every fix a balancing act, and whether Rock will stay open source.The main takeaway here is that security is a two-way street. The core team is working its hardest to roll out fixes, but they only protect your church once you install them. As a Rock admin, hereʼs your 5 step action plan to make sure your church stays protected and secured:Install security releases when they land.Watch for the red banner on your homepage.Work through the field types guide and move your Lava over.Keep cleaning up predictable IDs.Report security issues through Talos.Visit the show notes to download all the resources talked about in this episode. Don't forget to join the Rock Cast Rocket Chat Channel to see what other churches are saying about this episode.

Transcribed Content

Welcome back to Rockcast. If you guys have been following us on social media, you've probably noticed some security posts coming out over the last few days. Today, we're gonna talk about the story behind that, how our team has been working diligently to protect and launch these releases, and then we're gonna share how you can stay on the most secured version of Rock to protect your church against AI threats and vulnerabilities. So John, can you share with us... You guys have been hunkered down for the past few weeks. What's been going on? Yeah. So that's a a great word for it, hunkered down. This is something we've been calling for, I don't know how many months. Right? Quite a while. Eight months that AI would be definitely impacting security. And we've been talking about that I think on this podcast quite a bit. I think it's interesting though in history, how things come about. one minute, the world's fine and the next minute, the world's not. And it's usually a very small spark that does that. So I think if you look at World War one, I think it's an interesting example of how a small spark can launch a major world changing revolution. So basically, in World War one, one driver took a wrong turn and started a world war. Now Ouch. That was the spark. Obviously, underneath it all, there's a lot of other geopolitical things going on that were happening. And I think this use case is very the same here. Right? So AI has been gradually getting better and better and better. But it wasn't until very recently that the models were actually really good at finding vulnerabilities. , we've been trying and prototyping that for quite a while. Right? And it found a couple small things, but nothing big. But sometimes it takes a small spark, and then by then, it launches this big flame. So I would say a couple of months ago, models got really, really good. Not only that, but the techniques for... And the harnesses that use these models became better at finding things. So the small spark was an organization had had a bug bounty out for some other things and someone found a bug within Rock. And so they reported it to us and we fixed it. But then all of a sudden we started seeing, oh, wow, these models are caught up. Look, as we're fixing this specific issue, it's saying, oh, by the way, here's some other things that are similar to it. And so that's what really kind of launched this whole process. So again, it's one minute, the world's fine, and the next minute, all of a sudden, you have all these new capabilities that you can do. And and kind of what I would describe it as the the... We were given... The world was... Has now been given one of the most amazing metal detectors to find needles in haystacks when the needle is a security vulnerability. , in the past decades ago, we didn't have a metal detector, and, , you really had a difficult time trying to find needles. Occasionally, someone would step on one or... ? And all of sudden, ouch. That's a needle. , let's fix this. Or occasionally, we'd find one that, , we were... As we were writing it. But now we have this melody detector and it's finding all of these needles. So is these bugs that have been out there, have they been there for years or are they just now coming out or being discovered? A little bit of both. I mean, of them have definitely been there for a long, long time and some of them are relatively new. And AI is just bringing it up to the surface? Yeah. So we have about 2,000,000 lines of code. And so there's a lot of opportunity. The haystack is big. Right? And so... And for every line of code you write, there's an opportunity for either a bug or security vulnerability. So the more lines you have, the more probably you're going to... More needles are going be in your haystack. I guess that makes sense. But I mean, I thought we do a lot of penetration tests and we do our due diligence. Is it possible this is just negligence on our part? No. I don't think so. When you take the pen test, for example, we don't necessarily do the... We pay for the pen test. So we do the pen test. We don't actually do the work for the pen test. So we hire professional pen testers. Our pen test company comes highly, highly, highly recommended. And these are experts in their field with... Especially over the years, they have the best tools and the best training, and they bring and do their work. And so we've we tried to to do one in a year, and we didn't... Haven't done that since the very beginning. So we've we've had four or five pen tests, and they're quite expensive. I mean, they're $40,000. And a lot of these issues that we're finding have been in the product since then. So someone could ask , hey, are you upset that your pen testers didn't find this? And honestly, no. It would be unfair for me to say that's not... They didn't do a good job when I'm using a tool that they didn't have, that didn't exist back then. So, , it'd be unfair. It'd be hindsight biased. It'd be , in hindsight, they should have found all these. It's , okay. But they didn't have the same tools I'm using to find them. So that's kind of unfair. And and that's just to think about it from the context of pen testing. , should we have found them with... , regardless of pen testers. And the analogy holds true there too. It's , we weren't doing anything negligent to not have these. This is , now that we have these tools and we can find them, , it's much, much different. So really this is very much about timing and things changing. And it's also not just a Rock thing, Right? This is impacting every industry and every potential code base, which I think we'll talk about maybe a little bit further into the podcast. But you mentioned that the framework and the models have changed, and so you're able to focus on finding these vulnerabilities. Is it easy to find them? I still wouldn't say it's necessarily easy. In a sense, it's a magic wand, but it's not a magic wand. It feels magic. But for instance, we have 3,000,000 lines of code. You can't just go into cloud or anything and say, go find security vulnerabilities. Mhmm. You can, but you may not... You're not... The results aren't gonna be very good. So instead, what we've done is we've we've worked really hard to to try to figure out what is the best way to find these with the tools available. So we've created a whole suite of tools. One of them will chop the code up into little pieces so that agents can go in and look at atomic areas of code. So we chop it into pieces, and then we have a skill that basically says, go make sub agents that run across these small pieces. And so not only do you have small pieces, but we have multiple agents running across all the small pieces. And then the harness that we have will go do that, not just with one model, but they'll go do it with three different models from three different makers. So we'll have Claude running on it. We'll have OpenAI running on it and also Grok running on it. And so then there's another agent. It's very difficult. , it's very complex. There's another agent that's just listening and watching it all three people talk about these issues, and then they all get reported up to that agent. That agent then kind of puts them all together and says, hey, did y'all find the same things? Or, oh, no, you didn't? Surprisingly, they they all find very different things. There's some overlap, but it's not nearly what I thought it would be. I thought the overlap would be about 70%, and it's actually the opposite. It's about 30% that they find the same ones. And so they're finding different ones, which is kind of interesting because I think we all know, , especially OPUS 5.5 is amazing. , that's a huge unlock right there. And then for OpenAI, we're using Astra, which is a really great model. And, , a lot of people kind of look down on Grok because, , benchmark scores are not quite, , comparable to to the other two. But actually, it found a a good share of them. So it did a really good job. And so this other agent, assembles that all together, and then you can have false positives. And so that agent will basically scale and rate them as a critical high, medium, or low. Then it'll go through and confirm them. They'll take a deeper dive into that segment of code. And and then at the end, we run another agent across the whole thing and try to say, okay, You found x many vulnerabilities. Can you group them together together for us? So that... I mean, you'd hate to give the developers a a huge list of... A huge number of tasks in Asana to go fix. Right? When lot of them are , well, they're kind of the same thing. If I'm if I'm gonna fix them, it'd be so much more productive to fix them as one Asana task than as maybe a few. So CoSiU does that kind of grouping. So very, very complex in terms of that working. It uses tools I said, agents, sub agents. There's also something called a workflow in Cloud that we use. And so that's another tool that basically manages the agents and the agents report up to other agents. And... So, it sounds there's a whole lot of engineering behind it. It's not just AI, but you guys have thoughtfully broken that down. Yeah. It's not just something you just use the model. You have to think through this and you have You have to assemble the metal detector. Yeah. Did not come analogy. Did not come fully assembled. I think another word you could use to describe it is expensive. Mhmm. , that run took several days and was not cheap. In fact, I would say it's probably what we spent on... I mean, I ran it over the weekend. What we spent that one weekend is probably double what most churches spend for the whole year for everybody and their staff. Oh, wow. The whole time I'm watching the cost go up, right? And it's , well, do I stop? But I'm looking at it, it's finding real stuff. And it's , well, it's still cheaper than a pen test, Not by a lot, but... And it's finding real value. Right? So these are not Exactly. Trivial things. It's not micro UI improvements. No. These are very important security findings. So we had to run the whole thing. Afterwards, now we're going through... Now that we have the results, obviously, we're fixing the things, but at the same time we're improving that agentic harness so that we could probably be more efficient next time. I think we can probably get 30% savings next time because of lessons learned. We also have an incremental way of doing it. So we can say, hey, from last time to this time, only only look at the files that changed. Know, that said, I think there is a lot of good reasons to run it across the whole thing again. Because... Just because they found a good list now doesn't mean they couldn't find more if we ran it again. So our plan is to go through, get all these fixed, then we'll probably do our professional pen test. We kinda have that on a light pause for a second. It was about to start. So we have that on a light pause because there's no use having other people come in Yes. Before we have everything fixed because they're just gonna find and it's just gonna waste their time. So we want to wait till we get ours fixed. Interesting. Yeah. So it's definitely been a wild, wild ride. Yeah. Definitely different costs associated right now with maintaining software that's been built previously for sure. And, , in the new process of finding these security bugs, , what's the part that's... In every process, there's a part that takes longer or is faster. What right now is the piece that takes the longest? Probably the triaging. , it's provided a good list of these. And so we have to go through and verify as a human, is this really a bug? Again, another thing that's very interesting, think it's gonna be very specific to our use case, but actually very general to what other people are going to experience as they bring in more agentic workflows into the organization, is making sure that your agents have knowledge. So we're spending a decent amount of time saying, well, that's not really the way it works in Rock. , you assume something that's not true. And so we're not just saying throw it away. We're going into our docs and saying... Not not the docs that humans read, but docs that our code reads or agents read and saying, hey, this is the way it actually works. And so we're putting more and more knowledge into the code base, which will help these agents in the future. And I think everybody's gonna be doing that with their jobs, is putting more and more of what they know that's only in their head into systems that then agents can help work through and take some of those trivial things away. So the triage takes time because we have to classify them, but then also we have to come up with a solution. , some of these issues are very difficult, and it'd be easy for us if we just wanted to say, what? Fix it. But then we broke, , functionality that people expect, and that's where this balancing act comes into play. The biggest thing that makes Rock... Well, there's two things that make Rock difficult from a security perspective. One is its extensibility, and I'd say that's the first one. There's nothing in Rock that's, , hard coded or or fixed. Everything is very flexible. Most applications, a page is a page and it has specific functionality, and that's all it does. Nothing in Rock does that. And no one in the community wants Rock to be that. That's right. For good reason. When you think about things attributes, blocks, there's just so many things. We've given people not necessarily an application, we've given them a toolbox. And because we can't control how it's used, it makes it very difficult to do these security fixes. The other one is our open sourceness. Yes. It's so much easier for an agent to look through your source code and find a problem than to look through a running app and and where it has no source to find the problem. But again, that's a problem for today, but hopefully once these things get resolved, then it's not. Several people have asked, are you gonna go close source? And it's , well, that's not in our intention. That's not in our heart. That's not where we wanna be. If it makes sense, we might have to. I don't think that's been determined. It's a big question mark to me. We have to also understand that the source is already out there. if we went and closed source tomorrow, the benefits of that don't really come in for another couple years. So I would say for those of you who are worried about that, , trust us, we're we're not trying to do that. We we prefer not to do that. We we actually have a pretty good idea of what we might wanna do, which just keeps it open source. There's other difficulties with the open source, , especially as you move to NGED TIC programming, there's some things we just kinda to do behind the scenes. Mhmm. , if we started doing PRs the way we wanna do PRs, we would be generating 10, 20, 30, maybe even a 100 PRs a day. But we don't want that all in the open. And some of those PRs can be , well, that was stupid. ? Not that a human did something stupid, but a but a bot did something stupid. And and that's laundry you probably don't wanna have out there. But we're trying to find a way that we can have our cake and eat it too. But... Yeah. So it's... It is difficult. Let me give an example too, , of the of the things that we found without giving too many details. So if you think about field types, that's a lot of thing... That's a concept a lot of people don't understand. But if I tell you that field types are what drives attributes, then you probably get it. , all those attribute types are really just powered by field types under the covers. So, , people can put those field types anywhere or those attributes anywhere. You can put them on a workflow entry form. You can put them pretty much anywhere. Right? And if you put that out to the general public, they can type stuff in there that maybe you didn't intend them to do. And a lot of times it doesn't matter because, , you're just getting the the data. But sometimes you put that data in places that'll resurface again someplace else on a web page. So maybe you have an attribute that updates someone's first name, for instance. Well, now that first name is shown on the person profile page, and potentially someone could click on that and potentially be taken someplace that again doesn't in of itself is not a bad thing, but maybe they can spoof, and maybe all a sudden your staff thinks it's a safe place and it's not a safe place. Mhmm. So the easy answer that most people would do is just , good. You can't put you can't put bad stuff in these fields. And that sounds good, except sometimes bad stuff is good stuff, HTML and Lava. And so what do we do? , do we just... Alright. Our our our two options are chop it, and now you can't do the stuff that you want to do, or let it go through, and now you can do stuff that could be that could be dangerous, and now you have to take all the preventative steps. And there are preventative steps today that that the community can do, could have done to protect that, but unlikely to think through or do or They may not know they're making that choice when they make the choice today, what the consequences are. Right. So those are the choices. , can't have powerful, extensible things, or you take on a whole bunch more of the burden. Now there's a third choice, and this is a choice that we decided to do, was that we would give you even more configuration. So we would say, you can't have fun things unless you check this switch and this switch. So on an attribute, you might see after these security fixes that there are enabling buttons to enable certain certain functionality. Now you gotta be careful. , don't enable that unless you think through it, but you don't have to run it through lava filters and do a whole bunch of manual stuff yourself. So then the question becomes , okay, well, do you just turn on that switch and go into default off mode? And break things for people that have other configurations in their system? Or do you leave it to default on, but then you have to go to every single place and enable it? And so we chose another third option, which is to say, there's a master switch. So we default it off at every attribute level. And then there's a master override that says, well, don't turn on the default off switch yet. And then it allows you to turn it on and then go find the exceptions and then turn it back off again and then go fix everything. That's similar to how we changed the Lava engine. Exactly. Yeah. Yeah. And so we have a blog post that kinda goes through all that and helps you determine Here's what I'd say, though. Security is a partnership, always is, always will be. So you have to go read that guide. You have to go take those steps because eventually we're gonna take that other switch away. Because at a certain point, we do have to make sure this is secure. We can't leave it in that state. Now you have the ability to go turn it on today if you want. It's not on necessarily us, but some security researchers might say, well, you can't ship in an unsecured state, which is fair and right, but they don't also have to live with the community who maybe doesn't have the time, the resources. So we're trying to push and play and do the right thing from all angles. And I think sometimes security people can be somewhat difficult because they only see one angle, and they only care about the one angle. And if we played that way, many people would have issues. And so it's kind of a balancing act. Of course, you want perfect security, but that's possibility. Not And even comes down to resourcing. How many resources do you put into security? Well, the easy answer is all that it takes. Okay, well, you do nothing about security then, right? Right. So it's a balancing act. And it's a very difficult balancing act. And so we're trying to walk this tightrope. So here's what I'd say, it's , take that blog post seriously. It's often been the case that people hear us, but they don't listen and do it. And then we're left years down the path with it not being done. This is one that it cannot happen. And it can't just be intent to do. It needs to get a spot on the calendar. Yeah. And it honestly shouldn't take too much time. Nicole, I'm sure we can get a link in the show notes to Oh, I'll drop it there. So, but we're serious about that one. it won't be long before we take away the master switch and enabled. Yeah. Makes So what other steps do our Rock admins need to take to be secure and make sure that they're protected? So stay up to date for sure. when you see these security releases, you have to take them seriously. Other things you can do is realize that security is, I said, a partnership, especially with a toolbox. So we can't make a perfectly secured product that keeps you from doing things that would be insecure. You can't... Not least a toolbox. Right. Now if I wanted to be a simpler solution that everything was fixed and easy that might be possible. But even then, there's always gonna be something bad you can do. Let me give you an extreme example. In any product, we have this this problem. You could put every user in the Rock administrators group. There's nothing I can do to prevent you from doing that. Would that be a bad idea? Yes. Everybody. That's ... That's... Everybody gets that. Right? That's the extreme. Right? So Yeah. Now if you walk that back a bit, that's on the very far extreme. , we had some feedback of, hey, , if you enable... There's there's a setting that enables lava commands everywhere. Someone said, hey, did if you enable SQL that bad things can happen? Mhmm. It's , yeah. And that's not a best practice. ... And I think they knew that. I'm not trying to... They they certainly knew that. They were just wanting to bring it up. And it's , what would you... What what can we do? I mean, we could take it away, but, , that's just not the way those... That feature is engineered. So, , longer term, what we're gonna do is, , move that to another place and say, we can put , Out's a global attribute. So we don't really have a lot of options in terms of warning you about things. So we're gonna pull that back into a system setting and then be able to put warnings over the top of it , hey, don't turn on... Be careful what you turn on because honestly, we don't have control over all those. Some of some of those can be plug ins, and we don't even know what they are or what they do. But for a few of the ones that ship from core, we can give more more specifics of what the danger level would be. So that's another, again, that's another example that's , I don't know how to solve that without taking away everybody's toys. Yeah. And it's about finding the balance act. At a certain point, if you wanna run a toolkit in a in a toolbox... , we've used the analogy too. Rock is kind of handing someone a loaded gun. It's a great tool for defense, and and you can take ground, and you can do a lot of great things, but you can also shoot yourself in the foot if you're not careful. It's just... So it takes a lot of responsibility. And going back to that sequel thing too, I think it's another another good example. Again, the person who was reporting it to us knew these things and, , was just trying to be a good citizen and say, hey, just wanted to raise my hand and say, , about this. Right? And so in the use case, it's , well, why did they flip the switch? Well, they wanted to do something that didn't come out of the box. And it's , okay. Well, that's that's not a good reason to do it. Mhmm. It's saying, well, it was really cold in the house, so I started a fire in the living room. It's sometimes you just... You can't. , there's a boundary you don't go past. We do live in a world where sometimes we... Although we can elect to do it, we just shouldn't. Right. That's all parts of the world. And you can still go into the ideas and say, hey, I want to do this, but I can't. And if I did, it would be insecure. Can you can you can you guys, , change this? And so, yeah, that's fine. But, , I use Asana a lot, and there's a million things I wish Asana did differently, but it doesn't. And I could hack it, I could probably try to inject JavaScript and to do stuff. I'm , I don't. I'm... It's just... Yeah. It does it does a great job. I Asana, but it doesn't do everything I want. And I have to know where where is the line that I just don't cross. And I think a lot of times, we at Spark are trying to give these best practices. We're trying to tell people what not to do to keep their systems secure and running and upgradeable. And many times, many churches listen to us, and they follow those best practices, but sometimes there are churches who don't. And there's sometimes there's even a few rare cases where a church not only won't, but would be very vocal about why they're not going to. A while back, were talking about cloned blocks. Remember that a couple years ago? Yep. This is a great case of why that was, again, a best practice that we still stick to. We're going through fixing security things. Guess who's not gonna get those? Anyone who can't update. Well, also anybody who could clone the block. Yeah. That block is not gonna be updated. You might have a clone block that has security issues in it. There's no way for us to do that. we tell you don't do it. And for those who did, it's gonna be really dicey. And we don't know. Yeah, there's no way for us to know. Yeah. So again, follow the best practices. It is a partnership. And we can't tell you all the things not to do. There's an infinite number of things not to do. We're gonna try to do as many as we can, the example of moving the enabled commands into a system setting where we can provide more warnings. But we're also of limited team too. Right. , we we want to do that. When the idea was suggested, it's , great, I that idea. Have no problems with that idea. How do I get someone to go do that? what isn't done because of that? Right. And the timing of all of this, I mean, we've been talking about it for a long time, But, , we're pre conference, we're trying to push v 20. So there's just a lot going on. Our whole team is is focused on security updates right now. Yeah. The whole dev team, every person is on security. Yeah. So everything we wanted to be doing right now is which on is, , we're not complaining. That is the right answer. That's just not an answer that is great time. Right. And the dev team's been, I mean, building the tools that find the security issues, fixing the security issues. But beyond that, a whole new system was built for reporting. Yeah. So how do we how do we get security issues, , told to us? Before, we're using an inbox, a email inbox, and we felt we needed something better than that. So there is some tools built in the GitHub, but they're they're good. They're not bad. But we wanted something a little bit that we can innovate a little bit better on and and make sure that, , can work with us. Mhmm. And so we built our own. We call it Talos, which is kind of a fun name for it. But it's a way for the community to report the security issues to us. And then we'll be able to build agents now that can go in there, process them, help bring them to us. So on the backside, there's... , it has an MCP and we can go get all the stuff out of it. , at first, we didn't know if that was a good idea or not, if we should just use the GitHub one, but already it's proven to be super helpful. And if there's something we want changed, we just tell our agent to go add a new feature. And, , it's a little harder now. We have to read the specs and we're not... It's definitely not Vibe coded, but it's spec coded. So it makes it so we can go even faster. But I'd also, , put out there too, as we're doing all these things, getting v 20 ready, the conference, now security. Also, just realize the state of development has changed too. We went through that over the last eight months where we went from being artisanal coders to agentic coders, but we're going through it again. Everything changed in those eight months, but now everything's changing again. And so we're taking the next step from basically being AI augmented, , development team to being an AI, , centered development team. And that's... It's really difficult because we're way ahead of the curve on that. There's organizations obviously who are further, right Mhmm. But trying to figure out what they're doing and how they're doing it. There's not a lot of, , off the shelf for this. Factors zero off the shelf for this. They're customizing everything. And so we're we're trying to re learn everything again that we just learned from eight months ago. And then apply it because there's not a guidebook. Right. Yeah. So it's definitely a stressful time to be a developer, I would say, for sure. But on top of that reporting system, we've also put out the first wave of fixes, which was nineteen five, eighteen five, and 19... Seventeen nine. But, , security fixes, sometimes when you put them in, you actually might actually break something. So we actually broke something, and that was within the check scanner. So we made a change to how images work with MIME types. It's one of those security fixes that, , at a certain point, you kinda roll your eyes. It's , really? But it's important, so we made the change. But it had to do with some MIME types, and the check scanner was relying on some fairly loose support for MIME types. And so we broke the check scanner. And that's that's hard because that's code that's not in the same repo. That's code code that that never changes. , we haven't updated the check scanner forever. And so, unfortunately, it broke that. But because it's critical, we put out an immediate release so people didn't have to, , go do a whole bunch of wacky stuff. That release was so fast. Yeah. So we got that out, and we're working on another wave of security fixes. So you probably see one. Don't take this to the bank, but you'll probably see something next week of another one. And there'll be a third wave after that, and we'll just keep doing waves until we get through all of them. We don't wanna... We could sit back and wait. That'd be a lot easier for us and just get them all in one and push it out, but we just don't feel that's the best option. You'll start to see waves keep going until we until we don't need to. We're trying to balance not doing too many waves, but at the same time getting them out in an orderly quick fashion. Yeah. So it's all difficult. None of it's easy. Oh, I I know I was gonna say too. Some of these things this mime type one, again, not going into too many details, but that has been in Rock since the, , pre one o. And what I would say then, wasn't much of an issue back then. Technology changes. Right? Right. So back then, some of the exploits opportunities didn't exist. So when did that become a security vulnerability? Was it when it was written? I would kinda say not quite. But at a certain point across the line where now it's a security vulnerability because the new things have been added to some file types and such. And now all of a sudden, it is a security vulnerability. But it's just a wind in the night. Wind does that all of a sudden become something? And it's not all of a sudden, there's a flag that goes up and we miss the flag. It's just all of a sudden, some new technologies came into play slowly and incorporated into browsers. And then all a sudden, it's a security vulnerability. So that again is kind of shows you kind of what we're up against. It's not quite as black and white as it might appear. Yeah. Some of them are black and white, but many of them aren't. And it's Yeah. , back when IE was the browser, it didn't have that problem. It had a lot of other problems. But that security vulnerability was not one of them. So... But we also, , wanna push out more tools. So one of the tools that we added in the in the first wave of releases, so if you've updated, you already have this, is there's a new block on your home... On your internal homepage. And if you're a Rock administrator and you're not on a secured version after that wave, you will now see a red banner telling you you're you're on an unsecured version you need to update. And you can't dismiss it and you can't get rid of it. That's very helpful. It's sometimes easy to miss in the busyness of day to day, especially in the pace we have now. Yeah. We just wanted to tool this out so that no one could ever say, Well, I didn't know I was not on this unsecured version. So, John, what about for the church database admin that says, well, we've got... , after we get through the holidays, we have a timeline where we can go through and do some updates. And what would you what would you advise in this case? Well, it's just about rolling dice. At the end of the day, though, the way I look at it is everything's about risk. You gotta mitigate risk. You can never be without zero risk. So, , how much insurance do you buy? , everybody has to decide that. Mhmm. And actually every culture kind of decides that. There's some cultures that are very into insurance, and they almost insure everything. almost everybody would have homeowners insurance that would pay for even breakages of hot water heaters and everything. Other cultures have no insurance, either because they can't afford it or they just deal with the risk. And so it's hard for me to say what they should do. I think they should just understand that there is risk. Understand the risk objectively, and then decide how much risk they're willing to put in. Yeah. That makes sense. But I would say the security issues, we try to make them, , they're only security issues. There's not features baked in and we try to make them as safe as possible. Yep. Well, so do you have any kind of an encouraging note to leave our listeners with? I don't know. I'd some encouraging There's a lot of work to do. Mean, it's a partnership. We're going to do our part. But they need to be working on right now the field types. And I would say too, another one that we need to keep talking about is the predictable IDs. That's a feature we rolled in a long time ago. And so they need to be working on the sites and the pages that they've made to make sure that they're not using IDs and the URLs. So to get working on that. And there's there's tools to turn it off. , once you get that all cleaned up, you can turn that off. We still have some on the core side we need to do. So we'll be taking... We'll be expediting that once we get through to some of this other work. But yeah, I guess just as much as we're having to do extra work because of security, they're gonna have to do extra work because of security too, and not just with Rock. I'm sure it's with other things, which kind of brings up something else I discovered through all this, which I thought was very interesting and that the community would find interesting, is most people, I think, in the technical community know that there's a a global security database of vulnerabilities. It's commonly known as the CVE system. It's an international database, so it's it's... Everybody uses it. Here's what I didn't know. It's the most antiquated piece of software I have ever seen. It's in the 90s still at best. The user experience is horrible. It's painfully slow, which has created a whole cottage industry of people who make databases on top of that database just so you can use it. But you gotta go to the source to to get some of these numbers and such. So basically, the way it works is products can submit their vulnerabilities into this database and get a number, or researchers can submit their findings into this database and get a number. So, again, very antiquated technology, and it's very hard to get a number. , we can't just go get a number. We we have to either become a registered, what they call a CNA, for our product to get numbers, which is you gotta go through a whole bunch of training. You have to have a bunch of, , kind of two people in the organization that had to be always on the registry. It's just a bit to it. However, you can... What most people do is they don't become that themselves. They have someone be a proxy for them. And the most common proxy, especially for open source systems, is GitHub. So GitHub can be the CNA for you. But what I found very interesting is typically when you ask GitHub to go get you a number, it takes two to three days. Well, in May, GitHub came out and said, Hey, because of all the AI that's going on right now and all the exploits that are being found, it's actually taking thirty to forty five days to get a number. So that kind of tells you the burden that everybody's under. Yeah. Now think about that, that's May. A lot has changed between May and today. The models and the tools have gotten way better since May. So it's a bit of a lagging indicator. They also said in May that they were getting 10x as many requests for CVE numbers as they had the year before. Wow. And what I would say is... And if you look at the CVE metrics, I've been posting some of the findings on x as I figured this out. But all those findings are not even accurate because they're looking at... It's this rabbit going through the snake. We're looking backwards. Those numbers are based off three months ago. All those numbers are being put in today were requested three months ago. So I think in a few months, we're gonna see that... I mean, it's already... If you look at the chart, it's already going Almost straight up. Up and to the right. Yeah. And we haven't even hit the real part yet. And so I feel there's a very quiet nine one one happening that no one has even recognized yet because the data is is trailing the problem. And this is international. This is not just related to Rock. Every software platform is included in Yeah. Now many platforms don't participate in the CVU system. Some people have asked us if why we don't participate in it and encouraging us that we should. And here's our thoughts is, , one of the things we are adamant about is we don't want and see no benefit in talking about the details of these exploits. And it has nothing to do with anything that we're trying to keep quiet. It's we're trying to protect churches. And, , some security, , researchers would say, well, you give them a certain amount of time, and then it's okay to talk about. And I disagree. Our audience, some of them move a little slower. The average church doesn't, but it only takes a few churches moving slow that now they're a lamb that hasn't... They're not in the herd anymore. And we're not... And what's the value of talking about the detail? Is it fun to talk about? I mean, it's kind of driving by an accident and wanting to look at it. What does it matter? Just keep driving, please. We see there's no benefit to to doing that. Now what I didn't know and I discovered in going through this is most CVEs have little to no details. So Oh, interesting. In doing this, again, I'm researching this. And, , I tried to pick a product that was kinda similar to ours that has more money than we do. And so I picked Salesforce because, I mean, their budget is just a teeny bit bigger than ours. A tidge. Yeah. I mean, have a high rise building in San Francisco. Don't know. Nicole, how's our fundraising going? Not that well. Just now. It's good. We don't get a high rise? No. That's for future years. Okay. Good. Good. Glad you conference. Yeah. Glad you have goals. So I'm looking at some of their CVEs, and they've got some terrible... , you can't see the details, but you can... , it's flagged as complete account takeover with no credentials and only Internet access, which is pretty much the worst of it. Right? The worst thing you can have. They've got they've got those. And then the details are , yeah. Don't do bad things, which I totally agree with. I'm not... , I I actually agree with that. So I did... I was under the impression that you had to give, , way more details than that. I'd also say look at some of the ones at Apple, look at some of the ones with Microsoft. There's some pretty scary things that people are finding with the help of AI tools of zero day exploits that take over your whole machine from from nothing. There's some Linux ones that came out recently that were very, very scary. So it's everywhere. So back to the CV system, are we going to participate or not? Initially, we said we were going to. I'm hesitating now because of of the efforts it takes to even participate. , if it's gonna take two months to get a number, , what... And what's the value? What's the value being added? Now the security researchers would say, well, that's how you communicate to those using your product. And I think that's somewhat true. But if we were an infrastructure product, let's say I was a JavaScript framework that had a... That made a cool input for people, the CVE system is critical that I participate in that because that's majority of people's only way of knowing that there's a problem. I'm way down the stack. People have built way on top of me, and I have no contact with those people. We're pretty much at the top of the stack. , our churches build off of us. They build stuff with us, but there's not five levels above us who have built on Boulder, which is based off of slate, which is built off of Rock. And we have great communication with our community, and we provide more tools even now. So I don't see the value in the CVE system. So I'm kinda neutral to it, except it's it's a big burden to get these CVE numbers. And if you look on Twitter, there's some open source products whose stance is, show me a CVE number and I'll start looking into it. Makes sense. Kind of, but that's basing pound sand for two months. Well, well, the exploit's there. , we're not that. Oh, you mean we won't look into it at all until Yeah. I thought you meant show me a CVE number and I'll start reporting them. I can't get them. Right. But you mean we're not gonna work on it. Yeah. That's their that's their take is Yikes. We're not even gonna look at it until there's a CVE number. Oh. Yeah. It's very impressive how quickly the dev team is turning around all of these security issue fixes. Right. And we're trying to be as responsive as possible. We take this seriously, but... Well, track record is showing that. Now it's the church's turn to to pay attention and and make updates. Yeah. So as it relates to CV system, I don't know. I I see no benefit, and and it's just extra cost and extra resources that we don't have. That said, if someone has another good reason, we're not against it. We don't hate the CV system, especially now that we better understand that you can... Literally, your problem can be, just I described today, a mime type issue with the get file. Right. Okay. That's fine. But I don't see how that helps anybody. So to wrap up all the conversations and for the next steps for our Rock admins, they should... , when a security release lands, they should look for that red block that'll be on their homepage. And then they need to read the blog on field types and start moving their lava over? Yeah. I mean, you might need to enable lava on some on some attributes. And and again, have some ideas of how to make that even easier in the future too, the reporting of that, because right now you have to go into a separate exception log. And so we might eventually come up with another type of log that... But these things take time to build. Sure. And actually, , AI may drastically helps the build time. What it doesn't help with is the conception time. , how does this work? How should it work? Should do this? Oh, we should do this too. We find that takes just as long today, but the development still takes time too. And if they find any security issues, they should report it through Talos? Yes. And start updating. That's pretty much our action plan for the community. Yes. And you can find Talos on the footer of the website. There's a security tab there now. Security? And I believe also from GitHub. Is that right? Yes. So if you go to GitHub, GitHub repo is supposed to have a security file that tells you what to do if you find something. And we've updated ours. So there's multiple ways to do that. You log in using your community credentials. So it's all very easy. Great. Well, we'll make sure to link all that information in the show notes. And before we wrap up, we do have some exciting conference announcements. Of them is that we've surpassed our attendance from last year. So And that's super then we also met with Hortz Scholz for a little bit. That went well. It did. We're excited for everything that he'll be sharing. In his keynote presentation. Yeah. It's gonna be very exciting and very applicable to the community. Yeah. And to leaders and we're looking forward to that. And our last announcement is that our hotel cutoff dates are approaching. So if you want to save some money, make sure you book your stay before Monday. That's October 12 to get your rooms. And we'll see you guys at conference and on the next episode.