catch-22:(noun):a dilemma or difficult circumstance from which there is no escape because of mutually conflicting or dependent conditions.
I started in the field of software testing back in 2003. I am not hands on anymore, but I continue to engage with organizations in providing consulting and advisory services on software testing. My role involves listening to software testing leaders on their challenges, spend time with their teams to understand what they are doing and how they are doing it, observe and identify any gaps in the process, present the same to the leaders and their teams and how it impacts software quality, come up with approaches and measures to address them and help them get better over time.
After all these years, if I were to call one area as the “catch-22” for organizations when it comes to improving their software quality, what do you think would it be? Badly written tests? Choice of wrong automation tools? test automation framework? Sure, getting these wrong has its impact but I wouldn’t cite them as “blockers”. In my opinion, it is “Test Maintenance”. Why do I call this as “catch-22”? Let me explain.
Test maintenance is the process of keeping your tests updated all the time so that they reflect the features that your teams test on the application every day. In other words, as code changes happen every day, it is important to update your tests to ensure that they stay up to date with code changes. Both manual and automated tests need to be updated and maintained regularly and I will talk about manual tests first here, since they are the source for automated tests.
Often times, when sitting with testing teams in organizations I see that manual tests were built over time by different people and many have left the company. The current team continues to keep adding more and more tests to the suite but and it is not surprising to hear teams holding5000-10000 testcases in their test management tool and say that they need to run each of them for every release. And imagine if you were run these tests for 3 different browser/operating system/device combinations. Most teams do not know how to organize these tests and which tests to run for a release. Hence, when asked if all 5000/10000 needs to be executed for every release, they say “Yes” and claim that’s how it has happened in the organization for years now.
I am not getting into the challenges that exist in the scenario above, but readers can comprehend them. And the place where it becomes a “catch 22” is, when you ask teams to revisit the test suite and organize them better. They tell you that “not a single test can be removed from the test suite because no one knows its impact because they have not been updated and maintained over time.” And when leadership wants to start measuring “Test Effectiveness”, “Defect Finding Efficiency”, “Test Traceability to Requirements, Defects” they are in a quandary as it is impossible to collect any of these metrics because of one problem and that is “test suite not being updated and maintained regularly”.
It is important to update the manual test suite frequently. Ideally it would be good to update the test suite at the end of every release but teams often complain that they don’t have time for the same as releases keep happening bac to back. From my own experience, I have seen testing leaders asking for a week or so after every release to just update the test suite but that practice seems to be absent now. Many companies prefer the yearend holidays as a good time to update the test suite, but I think it has its own challenges, since all changes that happened over the year needs to be addressed at once.
The practice of building automated tests/scripts from manual tests seems to be slowly eroding and most companies have started embracing test automation from day one. As far as it effectively works for the team and the organization, it is fine to take that route though I personally won’t recommend that. Like manual tests maintenance is an inevitable task when you have an automated test suite. Given continuous changes to the codebase, we need to ensure the tests continue to work well as the application evolves. Developers adding new features or fixing newly discovered bugs, will affect your existing tests somehow. The team need to create new tests as needed or adjust existing ones due to these modifications.
Issues such as false positives, flakiness happen when automated tests are not kept up to date. Like the scenario mentioned with manual tests, as team members keep changing, I have seen that no one on the team spends enough time updating the tests and it’s nearly impossible to know how to solve the issues, so they continue to happen. This leads to multiple problems. The suite continues to grow, execution time increases, and there is lack of confidence in the test suite as it doesn’t help in finding regression issues and management starts blaming the organization’s test leadership citing no “Return on Investment(ROI)” from Testing.
A well-maintained test suite helps all stakeholders in the organization gain confidence on application quality. Constantly updating the test suite results in identifying issues quickly, track down the elements causing problems and helps eliminate any bottlenecks identified.
