There are a few times I have been taught the value of testing your backups. That is why even disaster recovery needs to be validated. The first story I share is probably a worst-case scenario. However, it can happen, so we need to be aware of failure points. We must validate our backups regularly to ensure the process we are counting on is there when we need it.
Testing Your Backups Before They Are NeededThere are many ways to test backups and our disaster recovery plan. We can do the extreme version, cut power to the required systems, and attempt to replace them via the procedure. However, that may not be needed. We may be able to do partial testing and ensure we can restore or copy the required files. Even simple things like an automated update process can cause issues with our plan. Therefore, we need to test regularly and ensure we keep the component parts updated and ready for use.
Outliers and False ComfortIt is easy to fall into a false sense of comfort about your strategy. That is where the second story comes in. Do not assume that your process working last month means it is still working this month. Likewise, ensure outliers such as systems, applications, or files that are rarely (if ever) used are included in the backups. I know of several situations where all was well until it was determined that some long-ago created file was lost or corrupted. Then they found out that the file or system was not covered in the DR plan. Even a simple act of rebooting a system has caused loss, and when that system was not backed up properly, it was a painful loss.
If you like this season, you will probably like Scott Adams' book, "How to Fail at Almost Everything and Still Win Big: Kind of the Story of My Life."