RX 24 - Navigating Rock Updates: A Step-by-Step Guide - Tony Visconti

Transcribed Content

Welcome to the second afternoon session on the New to Rock track. Today we have with us Tony Visconti, who's done a ton of church management and team training, that's gonna share his best practices and tips with us. Tony? Hey, thank you, Josh. I heard we have a lot of new to Rock people in the room. Go ahead and raise your hand if you're new to Rock. Okay. How about database administrators? Go ahead and raise your hand if you're a database administrator. Okay. Any executive pastors or ministry related staff? Okay. Over here. Great. Okay. So I got a good mix. Well, I hope there's a little bit of something for everyone today. I'm gonna show you a peek inside what our update process is. So I'm gonna give you a lot of tips and tricks and best practices that you can kinda choose from. So this is kinda like the charcuterie board of updates. Know that you don't have to take everything, take what you like, find out what tastes good to you, implement it in your system, and hopefully you'll walk away feeling a lot better about your next Rock update. A little bit about me. I'm the digital ministry manager at College Church in Wheaton. I've been working in the church IT space for about fourteen years. I do a lot of systems administration both inside Rock as well as in the Microsoft three sixty five space. I wore a couple different hats in that way. And I'm really passionate about helping churches and other small businesses navigate payment processing agreements. That one's from my other slide I did last year, actually. But if you want to talk about payment processing, we can definitely go there. All right. The takeaway. You can grab that, of course, from the normal place. I'll leave that QR code up there for a few seconds if you want to scan that now. The takeaway is simply my PowerPoint presentation. There's a lot of resource links that I have inside of there, so that's why the takeaway is primarily useful after the fact. You can also go to our website, collegechurch.org/rx20four. I think everyone's got it. I'll keep moving here. I can always go back to at the end. So who's this presentation for? If you're new to Rock, it should give you an idea of what the update process can look like and how you can mitigate risk and feel great about it. It's for people who wanna take more ownership of their update process. I know a lot of churches work with a partner when it comes to updating their system. Maybe you're looking to own that a little bit more. That should be a great session for you. It's also for seasoned Iraq Veterans who are just wanna review their process and see if there's anything incrementally they can do to improve it. Alright. So what are some of our goals today? We're going to review what a ROC update looks like and feels like. We're going to spend a lot of time looking at the before process and the after. The during is actually not the biggest process of a rock update. You know, it can be as simple as just clicking a couple buttons, honestly. But there's lots you can do to set yourself up well before and after the update that's gonna mitigate risks and also produce repeatable, consistent update results, help you not to brick your system. Why did I present on this topic? I found that when I entered the rock space and I was new, I wasn't really confident that our updates were gonna go well. I read the documentation, I was like, that is not enough for me. I wanna know all the ins and outs, and I wanna be confident that I can hit the ground running after an update. So I would encourage you to start with the documentation, as in most situations. But I think you'll find that there's more to be researched and invested in beyond that. Alright. The update, if you're looking the documentation around updating, if you're looking to find it, is in the admin hero guide. It's summed up in this one sentence. Once you decide you're ready, simply click the install button and Rock will do the rest. And that's for the most part true in most situations. I'm gonna help it hopefully be true in all situations once you walk away from this. Alright. Where are we going today? Let's start a little course. We're gonna talk about how to decide when to update, why you will want to pursue updates, how you can go about the update process step by step, and how to troubleshoot when you run into issues. Hopefully you won't. All right, the when. When you can update and what you can update to is going to depend on if you're on early access or general access. You know, in the keynote session today, they talked about that $4.1 number. That's the the donation level that Rock churches are asked to reach if they're going be a part of their early access program. And then general access is for all other groups. Early access is really about entering into that community part of Rock, that experience that makes Rock so powerful. We know that Rock powers churches, and churches power Rock. And one of the benefits that you get when being a part of Early Access is access to all those early releases one year before everyone else. And if you're a small or a new ministry and that dollar number is hard for you to reach, just a reminder that you can apply for a grant. So Rock has four core values: accessibility, community, craftsmanship, and innovation. And being a part of early access is one of the ways that you can enter into that community area. General access, we talked about this a little bit, right, so I'll just keep moving along. All right. Let's talk about update planning. We're going to talk about a few different things when it comes to planning for your update. How to select how often to update. How to pick a specific time of the week and time of the day, what you can do to prepare for your update, what are some things you can do to make your update go more smoothly, like investing in a change on, and how you can arrange supports to make sure your update goes smoothly. The whole goal here is to minimize risk and also to prepare your staff for the pending update. So as you're thinking about an update frequency, I want you to be thinking about these these questions and deciding collaboratively with your leadership team when you're gonna update. There's gonna be certain factors to consider in terms of the cost of updating, whether you're gonna do it on your own or with a partner, the quality of the update, how smoothly it's gonna go, and also how much risk you're ready to take on. Either because you're ignoring updates that include security features and putting that off, or you're jumping into an update and maybe being on that bleeding edge and one of the first persons to enter into it. Update frequency is also another decision point. Are you going to wait for each major release to come out, or are you going to jump on those minor releases that come out along the way? With any project, we can adjust the time we're spending it. We can adjust how much cost, how much outside resources we're investing in to help us with the update, and also the scope of the update. Are we simply pushing update and then walking away? Are we configuring and implementing all the new features that come along with it? Rock's recommendation, it would be mine as well, is that you would always ensure that you're running on the latest version as soon as possible, especially when it comes to a patch that has a security in it. Alright. Here's some factors to consider when you're selecting your update window. There's no perfect update window, but you should consider things like, when does my team use Rock the Moats? What's our when does our peak staff usage occur? On the flip side, when is our peak congregate usage? When are most people on our event registrations page, or checking in, or giving, or viewing our website on the content end? When are our system backups updating or running? And when will we have support availability both from people in the community and when our partners that we're already engaged with are available? Some typical windows to consider early in the morning before your staff gets in. That's great because you know that if something goes wrong, you're going to have the whole day ahead of you to work with partners and the community to try and figure out what's going on. End of the workday can be great because you usually get in there before a lot of your congregation is actively using the site. Night shift, almost everyone's out. Your night shift might be a little bit later than mine. I've got young kids. Alright. Let's take a look at the why. Why would why do you want to update? I already talked a little bit about this. Security reasons reason oh, sorry. Security releases are a major reason. You wanna make sure that you're limiting your attack surfaces and and getting rid of those known security issues. Bug fish fixes, especially bug fixes that address features that you're actively using. New features, we saw so many new features coming in version 17. Obviously, you're not going be able to take advantage of those till you get to version 17, so new features is a very compelling reason to update your system. And then also it's this community mindedness. It's this perspective shift from my I have to update to I get to update. I get to update and be a part of alpha testing, or beta testing, or just one of the first churches to experience what a new major release looks like. I have to update to keep up with security moving to I get to update, so I get to participate in community, whether it's in rocket chat and sharing my experience with the update, or with my team being the first one to jump in there and to test out those new features. Alright. Let's talk about how you go about your update. So I usually break it down into three major steps. I'm preparing and I'm researching. I'm getting ready for the update even before I decide my exact deadline or my time frame for that. I'm previewing the update in either my sandbox system or the demo the demo system online depending on what I'm testing and if I if I need my data or if I can just use the data that's out there in the demo system. And then I'm testing the actual update on our dev system before I move it into production later on. Alright. So how do we research? Well, what we want to do there is we want to figure out, looking at their release information, how is the update going to affect the features that my team and my church uses? I'm always asking the question, are there any specific updates that I've requested? Are there any updates that I will need to notify internal or external users about because it's going to directly impact? And are there any features that I saw and looked interesting, maybe I don't quite understand how they fully work, and are going to require some more investigation on my part, some more digging? I'm thinking about, at this time also, are there any specific staff teams that I need to have a meeting with to do some collaboration around a new feature or a feature that's changing significantly, and get their feedback? We can't make all these decisions on our own. Alright, there's a lot of release resources here. You certainly do not have to make use of all of them. Some of them are much more valuable than than others. I usually start with the Rock release notes, and then as a subset of that, looking at the release road map. That's my major starting point because it has the most level of detail about what's coming up in an update, and it goes line by line. It's it's categorical, so I can pay attention to certain features that our team uses more so than others. More and more, the Obsidian block release page is becoming a really important research sorry, resource for me. It's gonna tell me specifically which blocks are being swapped or chopped out, and you won't find that detail inside the release notes. So I'd really recommend going there each release, reading which blocks are changing and how they're changing. That's gonna set you up well and make sure that there's no surprises for you when all of a sudden a block looks brand new. Impact statistics is another kind of lesser known resource that I think is really helpful, especially in my mind when I'm trying to assess risk. I want to know before I update, how many people have done this before me? And I can answer that question. I can answer how many other churches are on 17 before I go there. I want to know, am I on the bleeding edge, or am I just riding on everyone's coattails right now when it comes to going forward with this update? And then as time permits, I'm trying to stay up to date on the blog, the podcast. I'm using GitHub search. I'll show you a little bit more about that. I'm doing global searches on RocketCheck for specific version numbers so I can see all the chatter that's been out there about the update that I'm moving to. And then if I have a support partner, I'm checking in with them on a monthly basis and just saying, hey, tell me what you've heard about Release X. Good, bad, in between. What's been your experience with the 20 to 30 to 40 churches that you support. They have much better visibility to how specific updates are impacting different churches. So let's take a look at a couple of those resources. When you go to the Obsidian Block conversion page, you're gonna get a list of blocks that are changing for each release and how they're changing. Whether they're being chopped, which means that the old blocks are being completely removed, or they're being swapped, which means that the new block is available but the old block also still remains. So look into that a little bit more, reread those definitions and get an understanding of which blocks are changing and try and think about if those blocks are a core to how your team operates. All right, impact statistics. This is just a bar chart. If you hover over a specific section of the sorry, pie chart, it'll show you the version number and then it'll show you the number of churches that are already using that version. So very comforting when I can see that 141 churches are already using 16.5. Rocket Chat, another great resource. I can go into that search area, I can type 16.5 or whatever release that I'm about to move to, I can click on that global search button, and I can look at every single thread that has started where someone mentioned that specific release. That might sound like a lot of work. It usually isn't, especially if you're moving to a newer release. There's probably been a max of 10 to 20 conversations, and you can see what other questions or whatever problems other Rock community members are running into. All right, hotfixes, looking ahead. So the release roadmap, we can use this to kind of get an understanding of where each release is at in the development pipeline. So if I'm moving to 16.5 and I'm on this page, I can also understand that 16.6 is in alpha testing and 16.7 is in early development. So it gives me an idea of what's kind of coming down the pipeline. Are there going to be more hot fixes that are going to release more minor fixes? Or after 16.5 is the next major release, and maybe there'll be a bigger time gap in between those two. This information is kind of interesting. It becomes more helpful once you actually bring that knowledge into GitHub. So if you go to the Spark development network repository in GitHub, you can click on that issues tab, and then you can start to filter down and look at the issues that are associated with each update. So if I'm moving to 16.5 in this scenario, and I know that sixteen point six and sixteen point seven are coming later on, the question that I want to know is, is there anything that 16.5 broke that is shortly after going to be fixed in sixteen point six or sixteen point seven, and should I wait? And so that's what you can do with GitHub is you can filter down the issue log based on version or release number and find out kind of what's going to be fixed later on. It gives you an idea if something major broke in the release that you're about to go to. If you're trying to find out, well, what about the bugs that I've submitted? Are those going to be fixed or have they been fixed? So you can filter the issue log down to author, and you can select yourself in that author dropdown there, and you can see exactly when and what releases your fixes are coming in. So usually, you you might have one or two, and that's really exciting to share with your staff. Hey, we submitted this bug and it's coming in the next release. You can of course combine those author and label searches if you want to find out exactly which of your bugs are being fixed and what releases. Alright, when there is a major release, you get some unique resources. For each major release, there's typically an update video. Very helpful resource, both for yourself as you're trying to understand what's coming in that release, but then also something that you can point your staff to. You're also going to get a documentation change log for each of the, I guess, main manuals. Can go up to the top of that main manual page and you can say, hey, show me all the changes, or show me a summarized list of changes that have taken place between version 15 and version 16. So I'll usually go to the admin hero guide or the engagement guide, those manuals that I often come back to. And when a new major release is coming out, I say, well, show me everything that's changed about this manual. I don't want to have to read through the whole entire thing again, but show me a summarized list of what the core team felt was important to update in the documentation manual. All right, so we're kind of navigating through how. We've talked a little bit about prepare. Let's move on to preview now. So a couple different options depending on where you're at. The easy button for previewing a new release, if it's available, is to go to the demo site. You know, there's no setup. There's nothing that you have to do. You can go to RockRMS and find that demo site. I don't know the exact link off the top of my head, actually. But that's available to you. That's one option. If you're thinking about setting up a sandbox, it's not very costly and it's not very time consuming, and the manuals are great. There's also community members and support partners out there that can walk you through doing that if you're if you're interested. If you don't have Azure set up, you can use your Azure grants, so you can even kinda cover the cost in that way. Lots of different options for finding a cost effective sandbox environment. I've linked some resources here about setting up Azure hosting, and the community is also another great resource if you feel like you're just stuck or just want someone to partner with you when you're doing as you're doing that. When you are doing a preview of an update, one thing you should consider is taking a copy of your production data and pushing it into your dev environment. That way you can be confident that any new features you've implemented or any new data that you've updated, you're seeing the exact same thing. And when you push that production update, it's likely to go just as successful as it was in your sandbox environment. All right. If you want to learn more about what it looks like to push your production data to your sandbox or your dev system, some great recipes and resources out there, go ahead and hit those up in the takeaway later on. We talked about how you can go to rock.rocksolidchurchdemo.com. If the update is already out there, that's an option. Or just ping the community. Say, Hey, is anyone on +1 or +1? Can you show me around? Can you tell me what your experience was updating? Our community is incredibly generous, and I am pretty confident someone would be willing to do that for you. I already talked about preparing and previewing. Let's talk about testing for a moment. Testing really starts with building out a test plan or a list of test cases. This doesn't have to be super complicated. You can do it in Excel or Word. What you really want to do is just document it so that you're running the same set of tests every time that you update. So think about those core features that you want to test every update, the profile page, your online giving, things that if they break, people will notice. You want to make sure that you can get logged back into your site, so make sure your authentication methods are working. Event registration's very public facing. Check-in is an absolute must every time you update to do thorough testing. That's because it's dependent on so many different external inputs. It's dependent on hardware, often various forms of hardware, whether that's tablets or iPads or scanners or printers. You've got all these different hardware dependencies. You've got the people factor, right? You've got your congregants, your end users, your staff, right, who might need to learn how to use use new screens or navigate new configuration options. You've got the software itself, which has been updated. So you've got a lot of different areas which kind of come together in unique ways, sometimes unanticipated ways. And so don't assume just because check-in wasn't specifically updated, that check-in is going to be okay. Always test check-in before the weekend, or before any major initiative is going to take place after an update. We've certainly been bit by that in the past. I would want to spare you of some of that pain as you move forward. Documents, a list of tests that are maybe specific to release. If there's new features or major updates to specific blocks that you use, you'll probably have tests that you don't run every update, but you're going to run this time. And then invite your staff into it. You've got a sandbox environment, maybe you've got the demo environment. You don't have to own all of that testing yourself. It also really helps if something goes sideways later on, because it wasn't all you who gave the thumbs up and said the update was ready to go. You got buy in from your staff who checked out the update and said, Yeah, we've looked at it, we're confident we're ready to move forward. If you want to jump into the community and be a part of the Alpha Beta team, I put a link there. The Alpha Beta team also has a test list, so if you're not really sure where to start with what you might look at testing, you can look at that Alpha Beta test list as just kind of a brainstorm of the types of areas, the types of tests you might want to run on your system after each update. All right. So now let's take a look at our step by step process for going through updates. This is what's worked well for me. It's what I've gleaned from our our partner in this space, Kingdom First Solutions. They helped me design this and kinda spoken to a little bit. It's not gonna be what your update process looks like necessarily because each system is a little bit different, but at least it can give you some ideas of the types of things I'm doing, and the number of steps, and what you might want to consider incorporating into your update checklist. We've already touched on many of these things already. Step one for me is research, and that can take two weeks, it can take three, it can take a week. It just depends on how big the update is, and how much pressure is on me, both in my mind and externally, to get that update pushed. When I've done enough research to feel confident that we're ready to move forward, I'm gonna set a two week out time frame and then kind of work this checklist back from there. So two weeks out, I'm pushing the update in our test environment, and I'm previewing it. I'm inviting our staff into it and say, hey, go to our demo environment, check out the features that you use all the time, give me feedback, let me know what's working well, let me know what's confusing. So that's really where we're we're getting out there early and we're we're spending some time using the update in a test environment. A week out, we're doing certain things from a communication perspective, primarily to get our team and our our congregation ready for the update. Day of, we certainly have some tasks to alert our team again, primarily put some notifications on our external website, let users know that the system's going to be down. We're doing the update production steps, which are fairly limited. And then we're doing that post update testing to make sure, as far as we can tell, that the system is in good shape. Let's take a look at each of those in a little bit more detail. Alright. So we talked about research already. I'm not gonna dig into this again, but these are the resources that I'm looking at to try and get a sense of if we're ready and if how quickly we need to move in order to take advantage of the update. Two weeks out in our sandbox environment, I'm doing these types of things. I'm looking at all our plug ins and seeing if they need any updates and if I need to connect with any of our plug in providers to better understand what's going on with their team. I'm potentially and likely refreshing our sandbox environment from our production environment. That's gonna give me the greatest level of confidence that our eventual production update is gonna go well because it has a near copy of everything that currently exists within production. This isn't let's see, not restricted to specific time window. That's just a note there to say that, hey, when we do our update in dev, everything's much easier, right? I can do it at any time during the day. I can do it on any day. I can spread it out. I'm under much less of a time crunch, so it's a lot less stressful doing that first update inside of our sandbox environment. I'm running those every update tests as well as those version specific tests in our dev environments. I'm looking at how the features interact with our data. The Spark team can never anticipate every single potential outcome that's going to occur in every system because all of our data is unique and all the ways we set up our systems are a little bit unique. So I'm looking at how our our data affects that update and vice versa. And then I'm using the public demo site and connecting with the community as necessary. All right, one week out, I'm notifying our staff via email or Teams or whatever communication method works the best. And I'm preparing our future maintenance page that's going to let online users know that our site's temporarily offline, but should hopefully be back up and running within a half an hour or hour, depending on the update size. Day of, we're using a Lava short code that's tied to some global attributes to easily post a pop up notification. That pop up notification is only gonna be shown once. After they click the dismiss notification, they won't see it the rest of the day unless we change it. It's just letting them know and reminding them, hey, Rock is gonna be out of commission at this time today. If you have any problems, reach out here. It helps them understand what's going to be down and what isn't. And we're sending this to all of our staff, and then we're sending a reminder to users who are still in the system ten minutes prior to the update, or maybe reaching out to their phone or Teams saying, hey, I see you're still in the system, it's still active. Just a reminder, we're about to push update and take down our system. This is just a takeaway resource. Here's the reminder email that I sent out. I won't dig into that now. When I'm ready to actually start the update, here's the list of steps, the actual checklist that I'm gonna print off. First, I'm gonna verify that my backups are in place, both my database and my web server, make sure those are in a good state. We use the Azure management plugin, which can sometimes scale or scale scale up or scale down the database depending on the time of the day. So I want to make sure that's off. I don't want that scaling our database any time during the update. I'm remote connecting into our production environment or test environment depending on where we're at with that. We're going to take our site completely offline by simply renaming one file. And IIS, you just simply rename the app underscore offline file. Just, well, you're renaming the app off underline file dot HTML to make sure that that file is in place and that will take down the site. I'm gonna wait ten minutes and that will ensure that my transaction log in SQL has run and the backup is in a good state. I'm stopping the IIS web server. I'm about to take a backup of the www root file. So I'm going to delete our avatar cache file because that takes up a lot of extra space we don't need. You can see many of these steps are gonna be specific to how we use the environment and might not be something that you need to include. But this is exactly the list of steps that we do. The the key goal is to make sure your backups are in place and to take the server offline temporarily. All right, after that we're going to bring the site back up. We're going to resize our SQL Server instance to make sure we can push the update as quickly as possible, make sure nothing times out during our updates. We use Cloudflare, so there's a couple steps there for disabling caching and purging the cache. We're gonna bring IIS back up. And this is really ready where we're ready to click that Update button. I like to document the number of service jobs that are in place prior to the update. Then I can look at that number afterwards and see if if any new jobs were added or any new jobs are running. I'm going to go into admin tools, general settings, rock update, find the oldest update. I might have multiple updates pending. I like to push them one by one. You can jump to the latest version. I never recommended. I always recommend doing them one by one. It helps you pinpoint where a problem area is if you run into an issue. So I find the oldest update. I click on Install. I wait for that Eureka screen to pop back up. And then I just want to make sure everything is settled down before I move any further. So on my web server, I'm looking at the CPU and making sure that's dropping DECT down to baseline, which is typically pretty close to zero. I'm looking at that I I think I'm an IIS worker process and making sure that that's not taking up a considerable amount of the CPU. I wanna make sure everything's calmed down, everything's ready to go before I start clicking any buttons in the system. There's a lot of post update jobs that can be running. I just want to let those complete before I move any farther forward. Then I'm going click the restart button as recommended on that Eureka screen. After this, I'm gonna jump in and do some basic testing, make sure my profile page is still loading, make sure I can log out and log back into the system. Anything else that is just really simple. I wanna make sure that the site's up and running, that sort of thing. I can get to admin screens. If I have more dot releases to push, maybe I'm multiple versions behind, this is where I'm going to go back to step a sorry, step b here and push another update. Okay? Push another update until I get to the point that I want to be. After I'm done with all the dot releases or all the major releases, then I'm going in and pushing all of my plug in updates. I'm going to stop IIS at this point. I'm going to back up the w w root folder. And this is the the place that I can come back to if anything else goes sideways after this. Of course, I've got my w w root folder also from before the update as well. I always like to have that clean copy of the system immediately after an update. So if I need to look at what happened after this update later on, can kind of go back and compare to that clean point in time. When I restart IIS, I'm re enabling the Azure Management plug in. I'm ready for that to take back over on the DTU side, scaling it up and down. At this point, I'm ready to let external users back into the system and our staff back in the system. The system is stable enough. That doesn't mean I'm completely done with my testing. It just means that I'm Okay with other people to start coming into the system again. Disable my Cloudflare cache. I wanted to start caching again. And very important, this one's like double bolded on my checklist. Resize my database back down to the normal level. I don't want to be charged for 400 or 800 DTUs for any longer than is necessary. All right, here's where we test, test, test. So we get out that test plan, we're going to go through those every update test scenarios, we're going to go through those release specific test scenarios, And we want to prioritize critical ROC functionality first. So it can be tempting to go test those new features, but you really want to go test check-in, work through your in my mind, my 10 to 15 tests for my giving page. It's a little bit painful, but consistency drives consistent results. And then invite staff feedback. Hey, the update's out there. Get in there. Start clicking around. If you run into any issues, I'm here for you, right? All right, so maybe you're at a place where you have offloaded your updates to your support company, or maybe someone else on your staff team runs updates, but you want to learn how to do it. You can kind of follow the classic five steps of training to get there. Have them do the work, and you watch, and then talk about the experience. Move into an environment where they're still doing, but you're helping, and you're doing a little bit, and they're helping, and you're doing everything on your own, but they're just there kind of in the wings in case things get hairy. So it's pretty simple from a training perspective, and I'm confident that you can take anyone from knowing nothing about updates to being the person who's primarily responsible for them. All right, let's just talk a little bit about troubleshooting, then we'll make room for a couple minutes of questions. If something goes sideways, and it'll probably happen to you at some point in your rock career somewhere, first is start by taking a deep breath. It can be very stressful when your system is down and your staff is coming in in fifteen minutes, or you really want to go to bed but you can't log into your site. It can be really stressful and discouraging. So take a deep breath, pray, invite Jesus into that space, and know that He says, Do not worry. Don't go it alone. If you're at home, grab your spouse, ask them to pray for you. If you're at work, grab a coworker, maybe your boss, say, hey, things are, you know, not in good shape. Can you help me figure this out? Here's what I'm seeing. Talk talk it out loud. If you have availability to a support partner, give them a ring, ask them for help, and you can always also reach out to the Rock community. Try and be detailed as possible. I know you're you're stressed out, but take time to explain what you've done so far, what you tried that's not working, and any error messages that are coming up. There's some great resources that the Rock community provides. One's by Triumph Tech. It's called Triumphant Problem Solving. This is a good one to look at before you're in the midst of things. And then there's another great resource that can help you kind of in the heat of the moment, what to do when things go wrong, can point you where to look, whether it's in an exception log or in the browser. Know that with Obsidian coming out, you're going to start seeing more and more exceptions directly in the browser console. So if you've never seen that area before, you don't know what that is, and you're responsible for troubleshooting on a day to day basis, grab a community member, grab me after this, and I'd be happy to show you where those exception messages might show up.