Tool Evaluation
Testing whether a reporting tool preserves metric definitions
Use a written metric definition and sample cases to check formulas, filters, aggregation, reuse and exports in a reporting tool.
Test a reporting tool with a metric defined before the demonstration. Follow its source fields through filters, calculations, charts and exports. The question is whether the agreed counting rule remains visible and produces the expected result when reports are copied, filtered or edited.
Write a checkable definition
Take a hypothetical “qualified enquiries” measure. Specify which record counts as an enquiry, what qualifies it, which statuses are excluded, whether a person can count twice and which event date places it in a week. Name the source field for each rule and the person authorised to change the definition. Your organisation must supply its own meaning.
For a rate, define numerator and denominator. “Enquiry rate” could use clicks, sessions or submitted forms as its denominator; each answers a different question. State the unit being counted too: person, submission, session or campaign row.
Definition item / Acceptance question
- Population
- Which records may enter the calculation?
- Rule
- Which status, event or formula counts?
- Grain
- What makes two rows the same or different?
- Time
- Which event date, time zone and period apply?
- Presentation
- Which filters, rounding and label will readers see?
- Ownership
- Who approves a change and records its effective date?
Key components of a checkable metric definition
- Population
- Which records may enter the calculation?
- Rule
- Which status, event or formula counts?
- Grain
- What makes two rows the same or different?
- Time
- Which event date, time zone and period apply?
- Presentation
- Which filters, rounding and label will readers see?
- Ownership
- Who approves a change and records its effective date?
Find where the calculation lives
In Data Studio, a calculated field in a data source is available to reports using that source, while a chart-specific field exists only in its chart. A data-source calculated field cannot be used with blended data.
Report editors can override some default aggregations in charts, though an Auto aggregation supplied by a source or formula cannot be changed that way. Ask which calculation is governed and how another chart or report will use it.
Power BI semantic models can use a star-schema design: dimension tables support filtering and grouping, while fact tables store observations or events at a defined granularity. Inspect the table relationships and granularity behind the measure.
Give the tool a definition challenge
Prepare safe sample rows with an eligible enquiry, a duplicate, an excluded status, a late correction and a record outside the selected week. Calculate the expected result first. Ask the supplier or implementation team to show the measure in a table, scorecard, filtered view and export.
Have another authorised editor create a second report from the agreed calculation. Record differences and where they entered the path.
A correct result in one chart establishes little about later use. Keep the approved formula, source fields, filters, owner, effective date and dependent reports together. When they change, repeat the sample cases and record whether earlier periods remain comparable.
