6 Common Debunkings of Ad Performance Claims and Case Studies
There are often crucial missing pieces of data in performance claims and case studies. Data can take credit for conversions that were driven by other channels or factors.There is a misunderstanding of how platforms or data work. Correlation does not always equal causation. The true cause is sometimes oversimplified. Simple mistakes and lack of quality assurance are common problems.
I've been in marketing for nearly 20 years, and I'm skeptical of 97% of performance claims and case studies I see.
Why? I've debunked too many.
The ones that seem too good to be true are always too good to be true.
I’m not saying that all marketers are frauds or liars. I am saying that marketers make mistakes. They also can cherry pick data to make themselves look better.
Here are 6 common debunkings and how you can spot/avoid them. . .
1. There is a crucial missing piece of data that tells the larger story.
Example: Mentioning ROAS without mentioning the scale = 🚩🚩🚩
That 5x ROAS from $100 in spend is a lot easier than from $100,000.
2. Data takes credit for conversions driven by other channels/factors.
This is either through intentional obfuscation or willful ignorance.
"Cold audiences" often have tons of warm users (or existing customers!) in them, but marketers don't understand or like to admit that.
"I didn't know the system would target my existing customers so much" 👀
3. Misunderstanding of how the platform or data works.
Example: FB's Geo targeting is different from Google Analytics (or other reporting tools) location data.
If a FB user recently visited NYC from Paris, he might still get NYC targeted ads, but his device is in Paris.
Another example: Post-iOS14, FB conversions credit the day of the conversion rather than the day of the impression.
If you lower your budget (or pause), you'll *see* improved performance. If you increase the budget, performance will look worse because of lag. It's an optical illusion.
4. Coincidence: Correlation, but not causation.
We don't advertise in a vacuum.
There are external factors like holidays and stock market swings.
Internal factors like landing page changes or offer/price changes.
These can (and often do) impact performance.
5. Oversimplification of the true cause.
"____ is working for us!"
Neglecting to mention all of the people, work, resources, and failures that went into getting ___ to work.
Sure, maybe you got TikTok Spark ads to work, but also maybe you spent $100k to get a top-tier creator or maybe you're friends with a celebrity that did it for free.
Maybe that thing doesn't "work", but rather you got it to work by forcing it to work or luck.
6. Simple mistakes/lack of QA
Ad sets that were labeled as one thing, but they're actually targeting something else completely by mistake.
I've seen this waaay too many times.
7. (Bonus) It's often a smattering of the previous issues combined.
Marketers aren't scientists.
We make mistakes.
Sometimes those mistakes layer on top of other errors and make everything harder to understand.
It happens more than you think.
Be skeptical of what you read on Twitter (or anywhere!).
It’s ok to be critical and ask questions!
