# Separating Signal from Spin in the Age of AI
# Source: https://quadsci.ai/blog/signal-from-spin
# Format: RAG-optimized full article text with YAML frontmatter

---
content_type: blog_post
title: "Separating Signal from Spin in the Age of AI"
url: https://quadsci.ai/blog/signal-from-spin
date_published: 2026-01-06
category: Research & Product
author: Bruno Velloso
---


How To Properly Evaluate Revenue and Churn Predictions Without Falling for the Hype

You, like me, are probably getting an unending onslaught of emails and articles about too-good-to-be-true AI solutions to a wide variety of business problems, and are likely struggling with how to separate things that are actually useful from things that are mostly hype. I often see claims about near-perfect accuracy for predicting churn, citing high percentages and vague metrics that are, frankly, difficult to understand.

## Make Sure You Understand Exactly What Any Evaluation Metric is Measuring

The most important thing is that you should take any number with a big grain of salt. Context matters a lot. If you are hearing vague grumblings of 99% accuracy, without a clear sense of what that actually means, it's likely too good to be true.

Important questions to ask:
- What does "accuracy" actually mean? (There are A LOT of definitions in machine learning).
- What data are you using to test the model? (It should be on a never-touched sample, completely separate from training data).
- What exact outcome is the model targeting? (Often the targets are misleading or unclear).
- How can I verify this claim or that this model is useful?
- What does "good performance" mean in this business context?

At QuadSci, we provide a variety of metrics and clearly define them in advance of training. For our multi-class GrowthAI model, which predicts 5 distinct outcomes (from high growth to contraction to churn), the metric that seems to be the most commonly accepted measure of "accuracy" is recall.

## It's Much Easier to Evaluate the Model's Insightfulness and Usefulness than its Predictiveness

I have a strong bias for models that are explainable and transparent. Neural Networks can be incredibly predictive and useful, but often they can over-memorize training data and you can't easily verify WHY it is predicting something will happen.

Understanding why a prediction is made is critical: if you do not know why something will churn, there's not much you can do about it.

We painstakingly convert billions of telemetry data into understandable signals via our proprietary data processors, and we neatly summarize the most important signals driving our model's predictions.

## Ultimately, Just Test it In the Field

We've found that people don't fully buy in until they can "feel" the product. After a trial, once we've trained our model and finished our dashboard, we do something we call "dealer's choice."

We simply ask our client (on the spot) if there are any accounts that they know well that they would like to profile in our dashboard. With no way to prepare in advance. Often, they pick a recent churn that took them by surprise. Nine out of ten times, we correctly identify the risk in the account, up to a year before the event.

Conclusion: If you come across an applied AI or ML solution, make sure you understand the evaluation metric well, and that you can verify and evaluate the model's insights and its predictions directly in the field. Otherwise, give it a pass.