Testing in Production

This is a topic that always draws some great responses when discussed where I work. Do you Test on your production systems?

I always come to the same conclusion on this one. Why wouldn’t you want to test in production? I know, I know. Your system is too “special” or “secure” or “regulated” or whatever to be able to test in production. So what are you going to do? Let your customers test it for you? Throw the code over the wall to the people that matter most and hope that it works for them? Take the chance that your customer will just understand when the house of cards comes crashing down in a burning heap of lame?

To those that say it just can’t be done, I say that maybe your system is just lacking testability – you haven’t built it right.┬áTo me a testable system is one that has a great handle on control and is inherently observable. If you can’t control and observe the software, you are dead out of the gate. Often, if you solve the control and observation issue, you will find a system that you can test in production – because you engineered it to be easy to do so in any environment.

So take a look at your systems and ask yourself if there are any measures you can take to affect the testability of your system in a way that would lead you to be able to test your system in production. Face it, no matter how you try, your QA systems will never be the same as your production systems. The data, traffic, configurations, scale, timing, etc. will just never match well enough that the tests you run in those environments will catch everything. Change control, make it observable and make sure your system works in production before your customer does it for you!

Centralized vs. Federated Integration Test in the Enterprise

I have been working on the question of federated vs. centralized integration test practices in the enterprise lately. As I have done some research into the topic, I have found that few resources are around on the topic. While some white papers exist, it appears that most companies are in the federated camp: relying on individual divisions to create their own integration test strategies – even when there are many ties among their applications that could benefit from a centralized approach.

Some companies like Google have extremely large tests that involve many applications, and even automate them to some extent. Most though, including the ones that I have worked for, spend time testing software from within their respective silos in an effort to protect their own. Each of these groups tend to create and maintain redundant sets of tests that cover their application needs.

The problem is that many of these needs are the needs of many of the other groups and a great deal of redundant and poorly performing tests are written. Every group creates a test to “create a user and password” for instance. Each is created in their silo and when the functionality changes, each breaks in their own way. Tests that perform things as trivial as this, and of course much more elaborate are created all of the time that could be shared.

Creating a centralized integration test group may be able to fix this redundancy issue and help protect production quality as you do so. Sharing resources, test data management, and testing know how might be a way to create a group that solves the issue of poor communication across your organization when it comes to system integration test. This one set of testers will help build the “moat” that protects your production castle from impending doom.