War Story
I’m not sure how universal the term “War Story” may be, but this term is used to describe a painful event experienced in the life of an IT professional. It may be related to infrastructure, processes, software development, product management, software development lifecycles, etc.
The value of a "war story" is that it provides real life experiences substantiating risk associated with not following good practices. It is always tempting to cut corners and cost in information technology for many good reasons. We need the War Story, sometimes, in order to increase awareness of decision makers the risk behind certain decisions.
Today, I am sharing a war story submitted by one of our readers regarding the need to test your disaster recovery plan. This is a topic we come back to from time to time because it is extremely important. For those of you with decision makers who don’t understand the value exercising a disaster recovery plan, here’s your war story.
Mike Writes:
Thought I share a recent experience that your readers might appreciate.
Q: How long will it take to restore a database server after failure?
A: Longer than you might think if you’ve not tried it before.
We recently had a database server fail. It had been running Windows 2000 and SQL Server 2000.
Not a problem I thought. We had database maintenance plans in place that backed up the databases to disk each evening and the server was backed up to tape every night.
The restore process should simply be a case of reinstalling the OS on a new machine, installing the backup agent, restoring the files, start SQL server and restore the databases.
It was not that easy.
After restoring the files and rebooting, the SQL server failed to start. I tracked this down to two issues.
- When SQL Server had been installed, it added an entry to services section of the registry containing the path C:PROGRA~1MICROS~1MSQLBINNSQLSERVR.EXE
Subsequently other Microsoft products had been installed.
After the restore the path C:PROGRA~1MICROS~1 no longer pointed to the SQL Server folder, but to a folder containing FrontPage.
- Having sorted out the first issue, SQL Server still would not start, as it could not find the Master.mdf file (the file is always open, so not picked up by the backup software). This is now a catch 22 situation, I have a SQL server backup of Master but until I can start SQL Server I can’t restore it. I ended up having to reinstall SQL Server from CD and then restoring all the databases.
Luckily in our case this was a development server, so the downtime was not a major issue, but if this had been a customer facing production server it would have been very different.
The moral of this story is ‘Carry out a test of your restore process’, it may not be as simple as you are hoping it might be.
If you find this helpful, drop me a note at btaylor@sswug.org. Thanks for the reminder Mike!
Cheers,
Ben
$$SWYNK$$
Featured Article(s)
The Secret behind Accurate Estimating (Part 2)
This series covers three methods of charting and logging your work product estimations: Iterative charting to improve accuracy, charting for effective communications and charting for a streamlining process example.
Featured White Paper(s)
Structuring the Unstructured: How to Dimensionalize Semi-Structured Business Data
Written by Interactive Edge
The Business Intelligence industry … (read more)