Good Morning, It’s Wednesday, March 19.
Topic: A/B Testing Alternative | T-Test | Excel Tutorial
For: B2B and B2C Managers.
Subject: Statistics → Practical Application
Concept: Hypothesis testing
Application: Using a t-test in Excel to compare business strategies
Don’t keep us a secret. Share this newsletter with friends (copy URL here).
Introduction
There’s a difference between feeling right and being right—and in business, that difference can cost you.
We often rely on experience or intuition to make big decisions. We assume we’re making the best choice and keep moving forward.
But what feels right isn’t always right, and we don’t realize it until we’ve wasted time and money.
We often use tools like A/B testing, but they aren’t always available—sometimes, we have to figure things out without built-in support.
That’s where a simple method like a t-test helps us compare two choices, validate results, and backup our assumptions.
Real-World Example
Imagine you want to test a new advertising channel. You’re already running ads, but you need to decide where to invest more—Google Ads or LinkedIn Ads?
You don’t want to waste money on the wrong platform. This is how the t-test helps make this decision.
Step 1: Define the goal.
You need to find out which ad platform converts better—Google or LinkedIn. The goal is simple: Compare the number of leads each one generates.
Step 2: Collect data.
You run ads on both platforms for the same amount of time, with the same budget, targeting a similar audience.
At the end of the test, you count the number of leads from each platform:
Google Ads: 120 leads
LinkedIn Ads: 150 leads
LinkedIn Ads seem better. But is that difference big enough to matter, or could it just be random?
Step 3: Perform a t-test.
A t-test checks whether the difference between the two numbers (120 vs. 150) is real or just due to chance.
A t-test is a simple statistical method that helps compare two options and tells you if one truly performs better or if the difference is just random.
Step 4: Interpret the results.
The t-test gives you a number called a p-value—if it’s small enough (usually below 0.05), it means LinkedIn Ads really perform better, and it’s not just luck.
If the p-value is below 0.05, you shift more of the budget to LinkedIn Ads because the data proves that it converts better.
If the p-value is above 0.05, the difference isn’t statistically significant, meaning LinkedIn Ads might not actually be better. In that case, you test again or look at other factors.
With only a short test, you can confidently make a data-driven decision and invest where it matters, avoiding losing time on tests and money on the wrong platform.
How to Run a T-test in Excel
A retail chain recently launched a new product and wants to boost sales. It has $5 million to invest in marketing.
They tested two strategies across similar regions with 100 customers each.
Region A: 20% discount coupon campaign. This averages $38 in sales per customer (with a $12 standard deviation).
Region B: A social media influencer campaign. This averages $45 per customer (with a $15 standard deviation).
The social media influencer campaign seems better. But before committing $5 million, we need to be sure this difference ($7 per customer) isn’t just a random chance.
Step 1: Goal. Is the $7 sales difference real or random?
If the influencer campaign truly performs better, this could mean millions in extra revenue. If not, we risk wasting the budget. A t-test will help us decide.
Step 2: Data. In Excel, enter sales data for Region A and Region B in two columns.
Step 3: t-test.
Go to Data > Data Analysis > t-Test: Two-Sample Assuming Unequal Variances.
Select Region A sales as Variable 1 and Region B sales as Variable 2.
Set Alpha = 0.05 (this means we’re testing at a 95% confidence level).
Step 4: Interpret the results. This is a two-tailed test.
The p-value = 0.0003 → Since this is less than 0.05, the $7 sales difference is statistically significant.
This means that, on average, the influencer campaign generates more revenue per customer than the 20% discount coupon campaign.
Pro Tip: To be even more precise, we calculate the Confidence Interval (CI).
Standard Error (SE): 1.92
Margin of Error (ME) = SE × t Critical (1.972) → 3.79
Final CI = $7 ± $3.79 → [$4, $10]
This means that, no matter what sample we take, the influencer campaign generates between $4 and $10 more per customer compared to the 20% discount coupon campaign.
Conclusion:
With this data, the company can confidently invest in the influencer campaign, expecting at least $4 million in additional revenue—and possibly up to $10 million.
Limitations
Needs enough data – Small sample sizes can give unreliable results. More data = more accurate insights.
Only compares two options – If you have more than two choices, other statistical tests (like ANOVA) may be better.
Assumes normal distribution – Works best when data follows a normal pattern; if not, results might be less reliable.
Ignores external factors – A t-test shows if a difference is real, but not why it happens. Other factors (seasonality, competition) could play a role.
Where Else Can You Use This?
Pricing Strategy – Test two price points to determine which drives more sales.
Hiring & Training – Compare performance between employees trained with different methods.
Customer Retention – Compare two loyalty programs to see which keeps customers longer.
Operational Efficiency – Test two workflow processes to see which improves productivity.
Top Links to Deep Dive
Want to go beyond today’s breakdown? Here are the best resources to master this topic:
Harvard Business School Online – A Beginner’s Guide to Hypothesis Testing in Business. Link here.
Kellogg Insight – Is Your Digital-Advertising Campaign Working? Link here.
Stanford GSB Insights – A/B Testing Gets an Upgrade for the Digital Age. Link here.
OpenStax – Introductory Business Statistics. Link here.
Harvard Business School Online – T-tests: Theory and Practice. Link here.