Criteria to consider when selecting a commercial off-the-shelf (COTS) software package

First the somewhat straight-forward things…

  • Current requirements: What do you need right now? How much configuration/customisation would be required?
  • Future requirements: What do you anticipate you will definitely need later? What is the vendor’s roadmap, including frequency of release? How expensive is it to move away from this technology? How much influence do you have over the roadmap?
  • Implementability: What platform does it require? How difficult is it to integrate with existing systems? Does it have hardware scaling issues? Are there single points of failure?
  • Supportability: What is available in terms of training? Consulting? Documentation? Community? How stable is the vendor? If the vendor is also intended to be an integration partner, how aligned are you culturally?
  • Cost: How much will it cost to implement (license, hosting, customisation), maintain, upgrade, modify?
  • Deliverability: (If customisation or extensive configuration is required) Will customisation be done through APIs (good) or will it need to be done by modifying internals (bad)? How difficult is it to test the package (especially in an automated fashion)? How difficult is it to automate installation, configuration setup, and builds (wizards are bad, scripted APIs are good)? How difficult is it to setup version control especially integrated with your existing configuration management system?

Key non-obvious points

Reduce customisation as much as possible… otherwise you will be overwhelmed by the cost and effort of upgrades.

This suggests that you should modify the business process to match the package rather than vice versa… which in turn suggests that you should generally not be considering packages for strategic business processes / capabilities. This also suggests that you want to be very clear about and enforce boundaries to prevent package features creeping into strategic areas.

Customisation or extensive configuration highlights the need for deliverability.

If the package is like an appliance (e.g., Microsoft Word), it should just work. Initial selection and upgrades might be more about manual, exploratory testing. However, once we start introducing customisation, then the importance of being able to setup automated testing (as well as other development features) grows.

Especially with the typical COTS packages in terms of CRM, ERP, financials, the main options tend to have feature parity which means you should generally focus on other aspects.

This could be vendor alignment, testability, modifiability, etc.

It is better to reduce commitment than attempt to commit to a perfectly correct decision.

If the package does not require as much commitment (e.g., hosted service), we’ve maintained options to change our mind later and don’t need to worry as much about making an optimal decision up front.

Scripted customisation scenarios may be even more important than scripted demos.

In a customisation situation you want to assess the modifiability of a package in a consistent way by providing several customisation scenarios and seeing how vendors are able to respond.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Jason Yip

Jason Yip


Staff Agile Coach at Spotify, ex-ThoughtWorks, ex-CruiseControl