Did you ever wish you could un-send, or un-do? I had one of those experiences years ago that is still memorable.
I was working on a data warehouse merging data from two difference systems.
Their natural keys were supposed to be unique across systems; there should be no problem merging the data into one database. The problem was, I did not have any hardware with enough space to verify that assertion except on the production system.
I tested for weeks on a development and QA box, with no issues detected. So, we determined it was time to go live. After beginning the import process, and running for 4 hours, we determined that there were indeed duplicate keys from the two different systems, and we had to back out the changes. We stopped the import process and began the long rollback process. After a week passed we gave up on the rollback, and decided to revert to our backups. But we couldn’t stop the rollback process.
We literally had to turn off the power to the database server in order to stop the rollback process.
Restoring the last backup, and importing the data occurring since the backup was taken, took another 2 days. We were offline for almost 2 weeks. Our customers were not happy at all. Worse yet, we still had to figure out a way to merge the two databases, and still did not have adequate resources to test thoroughly before impacting a production system.
I know a guy who spent two whole weeks recovering data because the entire company database was dropped accidentally. He really wished he had an Un-Do.
The point is that there are a number of things in a professional carrier for which there is no un-do. You can’t un-send an Email, or un-say something to someone. So be careful on a personal basis. When it comes to systems, always try to have a rollback plan. If one isn’t possible, make the decision makers aware of the risk THEY are taking, because you would not do it that way.
Enough reminiscing…feel free to make your own application, share your thoughts here online, or drop me a note at btaylor@sswug.org.
Cheers,
Ben