How to approach the Make vs. Buy decision around Testing Tools

I’ve been working lately in a couple of projects where the Make vs. Buy dilemma around testing tools has been popping up from multiple fronts.

In order to correctly solve this predicament we need to understand the reasons and the situation where it usually rises within most development and testing Organizations.

From my experience, the flow of events is as follows:

1. In the beginning the Organization uses Word and Excel to manage their process. This includes separate but linked documents or spreadsheets to manage the tests, bugs, risks, etc.
At one time, usually after a medium or large product crisis, someone reaches the conclusion that the Testing Team has outgrown these tools and it now requires a more professional platform to manage their assets and processes.

2. A basic search of the commercially available tools and platforms reveals that the alternatives are to either spend a couple of hundred of thousands of dollars on an Enterprise Platform that will need to be deployed, customized and maintained to match the Organization’s needs; or to buy a number of cheaper products that will require the team to compromise on the functionality or to internally develop their own integrations and customizations (on top of these tools) to supply the features they need.

3. It is usually at this point that the alternative to develop an internal platform comes forward. The rationalization is the following:

If the Organization will need to invest a large sum of money to buy and customize the testing platform;
AND
If regardless of the application or platform chosen, there is no single product that will meet the needs 100%;
AND
If the Organization has the internal knowledge to develop good software products in-house
THEN:
Why not develop the testing platform in-house, creating a product that meets all the Organization’s needs at a price that is close or lower to what the Commercial Product would charge?

The assumptions above sound pretty logical, and that is why we see many inexperienced and overexcited teams jumping into these projects…

Here is where experience comes into play. There is no magic answer to whether a company should make or buy their testing platform; although after many years in this business I have seen a lot of companies that developed their own platform (and invested more than a couple of development man-years in the project) and then decided to acquire a commercial platform and to migrate their data and process.

The main reasons they tend to do this are the following:
– The total cost to develop and maintain the platform ends up being close or higher than the cost of buying and deploying a commercial platform.
– The commercial platforms have more readily available features and integrations than most in-house projects are able to develop.
– The externally developed tools keep improving their features and actualizing their technology, methodology and processes; while the in-house tools only maintain the functionality based on what the team is currently doing.

Overall and contrary to the initial assumptions and estimations, creating the tool in-house will most times be more expensive and provide less functionality than acquiring an external tool.

I don’t claim that there are no successfully developed in-house testing platforms, I only point at the fact that the hidden costs of creating and maintaining the system may be bigger and make the process less cost-attractive than buying a good system up-front.

Take into account that there are cases where there is no alternative and the Organization needs to opt for an in-house developed platform.
Classic cases are the following:
– When the system under test is extremely complex and technologically advanced requiring a specialized set of tools that is not available commercially (as is the case with many embedded and real-time systems).
– When regulations or other external factors do not allow the Organization to compromise on the functionality required from the testing platform, as is the case with extremely regulated industries such as the aeronautics and pharmaceutical industries.
– When the system uses legacy technology that is no longer supported by any vendor.

To conclude the subject; there are no magic answers but on this issue most of the time you will want to go for a commercial tool and compromise on the price and functionality you will receive. There is no optimal product (at least not for now), but the alternative of developing and maintaining one in-house is more complicated and less feasible than what many of us imagine based on our raw assumptions and lack of experience in the matter.

About PractiTest

Practitest is an end-to-end test management tool, that gives you control of the entire testing process - from manual testing to automated testing and CI.

Designed for testers by testers, PractiTest can be customized to your team's ever-changing needs.

With fast professional and methodological support, you can make the most of your time and release products quickly and successfully to meet your user’s needs.

2 Responses to How to approach the Make vs. Buy decision around Testing Tools

  1. Anonymous July 3, 2008 at 1:18 pm #

    Unfortunately, many Telecom companies are forced to make their own Test Automation harnest, as there are hardly any Off-The-Shelf tools for that matter.
    When it comes to automating Test Equipment usage, one can build it’s own using home made drivers, and if lucky enough might find some basic functionality readily available in tools such as Lab-View.
    These often do not include connection to Test Management platforms, and good Automation logging options.
    Now a new player seem to target this market – called QualiSystems, but their system seem to be extreemly heavy.
    Some companies such as Aqua tailor applications per company need – but that’s a wasteful solution.

    Kobi Halperin

  2. Joel Montvelisky July 6, 2008 at 9:30 am #

    Hi Kobi,

    Your point is correct, there are situations when you will need to create your own tool out of necessity; and then the only thing to do is to make sure the organization is always on the right side of the ROI curve.

    I had the chance to review QualiSystem’s product during a conference a couple of weeks ago and it seemed promising; I didn’t have enough time to go deep into its functionality but they may be on to something.
    I also asked about their integration to other tools and they claimed to be working on something with QC, which sounds logical since emerging tool vendors like to make sure they can plug into the existing systems.

Leave a Reply