We're in September, and this month is known as the preparedness month. In fact, the U.S. Government designed it as such at the national level to underscore the importance of being prepared. From the website itself:
National Preparedness Month is an observance each September to raise awareness about the importance of preparing for disasters and emergencies that could happen at any time.This may seem to apply more to things like forest fires, economic collapses or war, and if you're an IT professional you may not see the link between the things you do and this definition, but in a world that's increasingly dependent on complex technologies, going over your IT preparedness is an incredibly important exercise. In this post, I'll go over a couple of items you may want to think about and give you some tips that will hopefully help you to improve your organization's posture when it comes to disaster preparedness.
There are a number of potential issues that can impact your IT infrastructure with varying levels of impact. Right now as I write this post, there are thousands of technicians that used to maintain IT systems in a peaceful country, that now have to do the same job in a war torn country, under extreme conditions, while being bombed by Russian invaders. It takes tremendous courage for someone to keep critical infrastructure in working order in Ukraine, knowing that they could be facing armed invaders at any point. They do it because millions of people depend on electricity, the internet, the banking system, government systems, payment, and so on to stay operational. It's extremely hard to plan for a war, and I won't claim to know the fool proof method of making sure you survive an event like that, but the more prepared you are, the better your chances. Everything from redundant servers in different physical locations, disaster recovery plans, proper backups and proper documentation for business countinuity are key.
Common events like ransomware are easier to plan for because it's been discussed so much. Bad actors rely on poor security to gain a foothold in your network. Why not take this opportunity to review your admin user accounts, service accounts and other types of accesses to make sure everything is as expected? Most organizations have old accounts that aren't used anymore, or credentials that were never rotated. Any API key or token should be changed routinely, any admin account should be validated by the proper managers to make sure the employees are still with the organization, and service accounts that are no longer in use should be removed.
If you can't avoid a bad event, backups are obviously very important to have, but it's very common for an organization to believe they have sufficient backups, only to find out after an event that their backups aren't complete or even available. Go over your backup systems and ensure that they are properly being done. Can a bad actor who gains access to your servers pivot to the backup system and erase them? Test your backups by restoring them on a temporary system. Review your restoration processes and ensure they meet your business requirements. Backups may be a set and forget type of thing, but you should review the entire setup at least once a year.
Finally, make sure you stay up to date on recent developments and that your systems and processes adapt to the changes. Most organizations have systems that were set up for one purpose or another and then left running without too much attention paid to them. You may be doing regular software updates, but is the server actually running an OS that went out of support by the vendor? Maybe a recent vulnerability went unaddressed and it might be worth changing how your workflow is designed. Even local news can be of use, like a prolonged drought affecting your capacity to properly cool your datacenter.
Problems usually come out of nowhere and don't happen the way you expect. It could be that the person who needs to address a crashed router is on vacation and no one else knows where they left the documentation. Or you need to physically reroute some networking cables but no one thought to leave a headlight on site, forcing you to hold your phone's flashlight in one hand while doing your job with the other, or simple things like making sure you have extra water bottle at the office, and that the needed keys and tokens are kept with multiple responsible people.
Preparedness involves every field in the industry and is everyone's responsibility. While you may have a team assigned to backups or DR, everything done in IT should take risks into account, from a security standpoint and otherwise, to make sure the organization is in a good state of preparedness.