“To successfully implement continuous delivery, you need to change the culture of how an entire organization views software development efforts.” – Tommy Tynjä

Gartner, in a report on the future of DevOps tool sets, makes the claim that in 2016 “DevOps will evolve from a niche strategy employed by large cloud providers to a mainstream strategy employed by 25 percent of Global 2000 organizations.” With the vote of over 500 of the biggest organizations of the world for this development approach, is it’s fair to say this is a no longer a niche movement, but the norm. As a point of fact, other reports suggest the adoption of DevOps is even wider than that – the RightScale “State of the Cloud” survey last year found DevOps adoption had reached 66% – that’s 2 out of 3 organizations moving to DevOps if these results are to be believed.

So, it’s clear that much of cloud-based product development is now DevOps driven. The reasons for the move are not hard to find – much has been written about how products can be released faster and deployed faster. That apart, though, there is also the claim that the product quality improves. Among the most referred-to reports for DevOps is the Puppet Labs, “State of DevOps” Report. Last year this report found that organizations that turn to DevOps suffered 60 times fewer failures and 38% reported a higher quality of code production as compared to those that used traditional development approaches. That seems unambiguous then – product quality is better with the DevOps approach. If this must be the case, though, what does this mean for the software testing approach?

First, it makes automation central to the testing strategy and execution. As has been said, DevOps is like Agile on steroids. Releases are much more frequent and development cycles are much shorter – so many more iterations and much less time in which to test each of them. This is the perfect storm as far as the need for Test Automation goes. The Automation strategy must be put into place very early in the test cycle, given that there is not much time to build the automating framework once the actual product development gets underway. This implies that the testing team must be involved much earlier in the lifecycle of the product – as early as when the business requirements perhaps. The automation strategy also must be quite comprehensive. It’s necessary to automate unit testing as the implementation gets going. Given the central role of the operational infrastructure, there is Continuous Integration and hence a need for automating this continuous integration testing. Essentially, the objective is to bring automation to bear on each of the important steps, to be ready for continuous deployment.

Then the testing approach, as with many cloud products, must also look beyond the product and at the way it will be deployed and accessed. This means an even greater focus on stress testing, performance testing, load testing and even testing for security. All potential issues that are likely to crop up as users and usage scales must be identified and isolated before hitting the production environment. The same holds true of testing the performance of the product – potential issues that could choke the performance of the product, when live, should be identified early. These are critically important for Cloud products considering how closely the products are tied to the infrastructure they are deployed on. The focus of the testing is on finding weaknesses early – in that sense, the fundamental objective of testing can be said to be preventing issues, rather than in finding and fixing them once released.

One more, quite fundamental, shift in the test approach also becomes apparent. Testing must become tightly integrated into the overall effort. Testing should be continuous, reporting timely and fixes swiftly. For this DevOps teams, tend to comprise people from Development, Operations, and Testing. In many ways, Testing becomes a key enabler of the process of releasing products.

Jez Miller of Heartland Payment Systems has been quoted in the Puppet Labs report mentioned earlier, as saying “The number of issues we had from production emergencies that were triggered by an ops change essentially went to zero. Because we were able to roll changes out in an automated fashion, and then test those changes in the various environments, by the time code got to production, it had been through three other environments—dev, integration, customer test—before it got to production.” This would seem to be the change in the culture that Tommy Tynjä was demanding – have you made that change?