I’ve been working lately in a couple of projects where the Make vs. Buy dilemma around testing tools has been popping up from multiple fronts.
In order to correctly solve this predicament we need to understand the reasons and the situation where it usually rises within most development and testing Organizations.
From my experience, the flow of events is as follows:
1. In the beginning the Organization uses Word and Excel to manage their process. This includes separate but linked documents or spreadsheets to manage the tests, bugs, risks, etc.
At one time, usually after a medium or large product crisis, someone reaches the conclusion that the Testing Team has outgrown these tools and it now requires a more professional platform to manage their assets and processes.
2. A basic search of the commercially available tools and platforms reveals that the alternatives are to either spend a couple of hundred of thousands of dollars on an Enterprise Platform that will need to be deployed, customized and maintained to match the Organization’s needs; or to buy a number of cheaper products that will require the team to compromise on the functionality or to internally develop their own integrations and customizations (on top of these tools) to supply the features they need.
3. It is usually at this point that the alternative to develop an internal platform comes forward. The rationalization is the following:
If the Organization will need to invest a large sum of money to buy and customize the testing platform;
If regardless of the application or platform chosen, there is no single product that will meet the needs 100%;
If the Organization has the internal knowledge to develop good software products in-house
Why not develop the testing platform in-house, creating a product that meets all the Organization’s needs at a price that is close or lower to what the Commercial Product would charge?
The assumptions above sound pretty logical, and that is why we see many inexperienced and overexcited teams jumping into these projects…
Here is where experience comes into play. There is no magic answer to whether a company should make or buy their testing platform; although after many years in this business I have seen a lot of companies that developed their own platform (and invested more than a couple of development man-years in the project) and then decided to acquire a commercial platform and to migrate their data and process.
The main reasons they tend to do this are the following:
– The total cost to develop and maintain the platform ends up being close or higher than the cost of buying and deploying a commercial platform.
– The commercial platforms have more readily available features and integrations than most in-house projects are able to develop.
– The externally developed tools keep improving their features and actualizing their technology, methodology and processes; while the in-house tools only maintain the functionality based on what the team is currently doing.
Overall and contrary to the initial assumptions and estimations, creating the tool in-house will most times be more expensive and provide less functionality than acquiring an external tool.
I don’t claim that there are no successfully developed in-house testing platforms, I only point at the fact that the hidden costs of creating and maintaining the system may be bigger and make the process less cost-attractive than buying a good system up-front.
Take into account that there are cases where there is no alternative and the Organization needs to opt for an in-house developed platform.
Classic cases are the following:
– When the system under test is extremely complex and technologically advanced requiring a specialized set of tools that is not available commercially (as is the case with many embedded and real-time systems).
– When regulations or other external factors do not allow the Organization to compromise on the functionality required from the testing platform, as is the case with extremely regulated industries such as the aeronautics and pharmaceutical industries.
– When the system uses legacy technology that is no longer supported by any vendor.
To conclude the subject; there are no magic answers but on this issue most of the time you will want to go for a commercial tool and compromise on the price and functionality you will receive. There is no optimal product (at least not for now), but the alternative of developing and maintaining one in-house is more complicated and less feasible than what many of us imagine based on our raw assumptions and lack of experience in the matter.
Practitest is an end-to-end test management tool, that gives you control of the entire testing process - from manual testing to automated testing and CI.
Designed for testers by testers, PractiTest can be customized to your team's ever-changing needs.
With fast professional and methodological support, you can make the most of your time and release products quickly and successfully to meet your user’s needs.