Can you make it work? Those are words I love to hear, or make me cringe depending on the circumstances. What makes the difference? The point in the lifecycle of the product that doesn’t work. When a project is early in the coding or implementation, change is easier to accommodate. When you are in production, change is much more difficult to implement. Usually, when something doesn’t work, it requires change.
How do we get to those times where we need to ask to “Make it Work”? The main reason I have experienced or observed is that we don’t develop and test our system under adequate load. We have environments not capable of a production load, or simply don’t have the time to create an adequate load to test against.
When your system is running at a lower load, it may have a rather perky performance. As the load increases, the system slows down. When peak loads occur, your system may simple refuse to function. At this point, all the little short cuts you took, which have worked fine in the other environments, now add up to disaster.
You need to start asking questions:
- Is my normalization correct for this system?
- Do I have adequate hardware resources?
- Do I have the right indexes?
- Do I have the right statistics?
- Is my hardware configured correctly for this system?
Some of the things you may discover are not so difficult to fix. The problem you are now facing is that your system is already under stress. The tools you use to diagnose what remedies need to be applied add to the system stress, and reduce performance further.
So, test before stress. Don’t test the simple scenarios alone. Attempt to run peak load simulations prior to pushing to your production environment. Don’t take the short cuts; they add up to failure at the worst times. Evaluate your statistics and indexes prior to your production release.
Cheers,
Ben