Tool Evaluation
Part of Evaluating marketing software
Building a vendor shortlist without relying on feature counts
Shortlist marketing software suppliers by essential tasks, proposed edition limits and unresolved questions rather than advertised feature totals.
Build a marketing software shortlist by asking which suppliers could plausibly complete your essential tasks under the editions and conditions they propose. Record for each candidate a reason for inclusion and the question that might rule it out. Counting advertised features answers neither question.
Set a narrow entry test
Start with one or two tasks that matter most and the conditions they must meet. A team seeking to pass campaign enquiries to sales might require a visible source label, a controlled correction and a reliable hand-off. Another team may care most about content approval or reporting. Use your team's tasks, not a generic category checklist.
For each essential condition, decide what published material can be checked and what needs a supplier response. An advertised integration may justify asking a question, but its name alone does not show which fields move, in which direction or under which edition. Mark those details as unconfirmed.
Keep the first screen proportionate: its job is to find credible candidates for deeper evaluation.
Search through more than one route
Check relevant tools the organisation already has access to, supplier documentation and a wider market search. Ask informed colleagues about products they have used, but separate their experience from your requirements. A familiar supplier may deserve consideration, but familiarity does not establish that the proposed use case works.
Send candidates the same short description of the task. Ask each for the proposed product and edition, a route through the task, material dependencies and known limitations. Consistent questions make replies easier to compare and expose answers that describe a different problem.
| Candidate | Reason to consider | Essential question | Current evidence |
|---|---|---|---|
| Existing tool | Already available to the team | Can its current setup complete the task? | Internal owner to confirm |
| New candidate A | Describes a relevant workflow | Is that workflow in the offered edition? | Supplier response needed |
| New candidate B | Proposes another route to the same result | What extra system or work does it require? | Demonstration needed |
These rows show how to keep a discovery log; they are not findings about actual suppliers.
Remove clear mismatches; retain honest unknowns
Exclude a candidate when reliable evidence shows it cannot meet an essential condition, or when the supplier cannot propose a workable route. Record the reason and the evidence. If the answer depends on configuration, another product or an unconfirmed edition, keep it conditional until the necessary detail is available. Never silently treat “possible” as “included”.
Avoid a points total based on the number of features mentioned. An extra template library does not compensate for an essential hand-off that fails.
A candidate with fewer advertised functions may still be appropriate if it handles the core task with less operational effort. These are decision principles, not a ranking of products.
Check whether the evidence is comparable. A supplier's product page, a written answer and a demonstration show different things. Keep an unanswered question visible rather than scoring it as success or failure.
Evaluating Supplier Responses: Evidence Types and Reliability
- Product page
- Shows advertised features; may not reflect actual capabilities in the offered edition.
- Supplier written response
- Provides clarity on workflows, dependencies, and known limitations; requires verification.
- Live demonstration
- Reveals real-world performance, configuration effort, and edge cases; highest reliability.
Prioritising Vendors Based on Task Fit, Not Features
- Vendor A: Existing toolAlready available; needs confirmation from internal owner whether current setup meets task requirements.
- Vendor B: New candidate AProposes a workflow in the offered edition; requires supplier response to confirm feasibility.
- Vendor C: New candidate BOffers alternative route; requires demonstration to assess additional system or manual work involved.
Hand over a shortlist with a purpose
For each shortlisted supplier, write a one-line inclusion reason, the task it should show, the offered edition if known, the main unresolved condition and the next evidence request. If candidates solve the task in different ways, let them show those routes while holding the required outcome constant.
The shortlist is ready when the team can explain why each candidate deserves demonstration time and what observation would change that view. If every candidate remains only because it appears in a category list, sharpen the task before inviting sales presentations.
