What happens when you can't trust your data?

How to avoid a death spiral by building confidence in your companies product analytics.
Patrick Thompson
Patrick Thompson

"People spend more time on analyzing what tool to use than they do instrumenting and updating their data."

— Brian Balfour How You Battle the "Data Wheel of Death" in Growth

There is a massive trend towards companies investing in improving how they make data-informed decisions. Spending hundreds of thousands of dollars on tools to enable their organization to self-service, yet the number one problem we hear from companies is that the data that flows into these tools is not trustworthy. Similar to an aircraft these tools provide you gauges to course correct your business, but if they are showing you the wrong information you'll end up in a death spiral. This is a massive problem for companies who are reliant on understanding customer telemetry for their long-term success.

Every time consumers of this data encounter an integrity issue it erodes their trust and makes them less likely to use data to make decisions in the future. After a while, they eventually give up altogether and rely only on their intuition, which more often than not is wrong. Worse is when they use the data to make a business decision only to find out in retrospect that the data was inaccurate.

Companies try to solve this today by spending analyst time cleaning up their data and normalizing it instead of empowering the analysts to do what they were hired for, which is helping generate business insights that lead to growth. Retroactively cleaning up your data only works when you know that you have a specific data integrity issue, your analysts can't fix a problem if they don't know about it. It's better to clean up the data at the source and avoid unclean data flowing into your data warehouse altogether.

The reason this problem exists is that the people who are dependent upon this data and the ones responsible for capturing it live in their own worlds. For the majority of product teams, analytics is an afterthought, it's something that they know they should be doing but don't commit the time required to making it part of their DNA. This is primarily because most organizations reward shipping over measuring what is shipped. High performing organizations don't hide behind output but instead focus on the outcomes that they are striving to achieve. The only way to do this is for teams to determine what metrics they want to improve, identify the events that are needed to measure that metric, and align their business to improving those metrics. For your organization to really embrace data, product analytics requires dedicated resources and needs to be thought of as a feature of your product, not something that is one and done.

Today the workflow for determining what events to capture, instrumenting them, and verifying that they are correct is fraught with human error. For product analytics to be a P1 feature there should be a defined process that removes the human error and enables teams to define, track and verify their product analytics as part of the software development life cycle. Today there is no single source of truth for this information, it's often spread across Confluence pages or Google Sheets and quickly becomes out of date. Worse developers have to copy and paste this information or interpret what should be captured from a Jira ticket.

We're actively working on solving this, but there are a few things you can do today to help build confidence in your companies product analytics:

1. Tie incentives to hard metrics

  • Assign metrics to teams and reward them for hitting them
  • Give teams ownership on how to achieve results
  • Make the metric visible to the organization

2. Change the definition of done

  • Don't ship new features without a clear tracking plan
  • Verify that the events are being tracked correctly
  • Measure the outcome of work that is shipped

3. More data ≠ better data

  • Data quality is more important than data volume
  • Structure your events to answer business questions
  • Establish a standard naming convention & company-wide taxonomy

I'm keen to hear any other tips you have to help teams build confidence in their product analytics. If you're actively working on improving your product analytics, we'd love to chat.