The Noble Manager
Posts
When "Good Enough" Is Actually Great: The One-Sample T-Test

When "Good Enough" Is Actually Great: The One-Sample T-Test

Make confident decisions with imperfect data—and know exactly when to move forward.

Filippo Esposito
April 02, 2025

Good Morning, It’s Wednesday, April 2.

Topic: Digital Campaign Validation | One-Sample T-Test | Excel Tutorial
For: B2B and B2C Managers.
Subject: Statistics → Practical Application
- Concept: Single-sample hypothesis testing
- Application: Using one-sample t-test in Excel to validate marketing investments

Don’t keep us a secret. Share this newsletter with friends (copy URL here).

Introduction

In any business, we set clear targets and expect to hit them precisely.

However, early results often fall slightly short, leading companies to abandon promising campaigns prematurely, costing millions in missed opportunities and wasted resources.

We need a way to determine when "below target" is actually "close enough" to proceed confidently.

The one-sample t-test is this decisive tool—helping you distinguish between a genuine shortfall and normal statistical variation.

This simple statistical method can mean the difference between capturing your peak season or missing it entirely in time-sensitive markets like seasonal product launches.

Real-World Example

Imagine you're the Marketing Director for a national automotive brand.

Your company is preparing to launch a new SUV lineup, and timing is critical—you need to generate leads quickly as you approach the summer buying season.

Last year's META campaign generated poor-quality leads.

This year, you've developed a new digital campaign and must decide quickly whether to roll it out nationally.

Step 1: Define the goal.

Your financial team determined that the campaign needs a 12% lead-to-purchase conversion rate to be profitable at scale. This is your benchmark.

Step 2: Collect data.

You run a four-week test in selected markets, generating 200 qualified leads. Of these, 21 convert to purchases—a 10.5% conversion rate.

At first glance, this looks disappointing—you're missing your 12% target.

The campaign appears to be underperforming, and your initial reaction might be to delay the national rollout and test new creative approaches.

But before making that costly decision, is that difference big enough to matter, or could it just be random?

Step 3: Perform a one-sample t-test.

A one-sample t-test compares your observed value (10.5%) against your target value (12%) and tells you if the difference could be due to random chance.

Step 4: Interpret the results.

The t-test gives you a p-value of 0.514—well above the standard significance threshold of 0.05.

What this actually means: You cannot conclude with statistical confidence that the campaign is performing below your target.

The observed difference is very likely due to random variation in a limited sample.

The confidence interval for your conversion rate is [6.25%, 14.75%].

Since your 12% target falls within this range, there's a good chance your campaign could actually meet or exceed the target when scaled.

Now, you can analyze the financial implications.

Scenario Analysis:

Scenario	Conversion Rate	Sales from 50k leads	Profit ($3200/car)
Lower Bound	6.25%	3,125	$10,0 million
Target	12.00%	6,000	$19.2 million
Upper Bound	14.75%	7,375	$23.6 million

Lower bound (6.25%): If the true conversion rate is at the lower end of our confidence interval, a national campaign would generate approximately 3,125 vehicle sales from 50,000 leads. At an average profit of $3,200 per vehicle, that's $10 million in profit.
Upper bound (14.75%): If the true conversion rate is at the upper end, the same campaign would generate about 7,375 vehicle sales, resulting in $23.6 million in profit.
Target (12%): At exactly your target conversion, you'd generate 6,000 sales and $19.2 million in profit.

The critical insight:

Even at the lower bound of 6.25%, the campaign generates $10 million in profit—enough to be considered successful by your finance team (they set up a threshold for $8 million).

And there's a good chance you'll perform even better. The wide confidence interval reflects the uncertainty with only 200 leads, but the inclusion of 12% supports a case for moving forward.

The cost of delaying the campaign by 4 weeks to refine it further?

Missing approximately 25% of the summer buying season, potentially sacrificing $2.5-5 million in profit.

Given these factors:

A p-value indicating no significant underperformance,
A confidence interval that includes your target,
The seasonal urgency,

Proceeding with the national rollout is clearly justified.

Pro Tip: If you want greater precision in your estimate, you can calculate the sample size needed to narrow your confidence interval.

Required sample size = (Z^2 × p × (1-p)) / E^2

Where:

Z = 1.96 (for 95% confidence)
p = your observed proportion (0.105)
E = desired margin of error (let's say 0.01 or 1%)

This gives us a required sample size of 3611 leads to achieve a margin of error of ±1% and reduce uncertainty.

At your current pace (50 leads/week), that’s over 70 weeks—impractical for the season.

For a more realistic E=3%, you’d need 401 leads total, or 201 more leads (about 4 more weeks).

Conclusion:

The one-sample t-test reveals that despite appearing to fall short of your 12% target, the statistical evidence doesn't support abandoning the campaign.

By proceeding with the national rollout immediately rather than delaying for potentially unnecessary revisions, you capture critical summer sales revenue while maintaining acceptable profitability even in the worst-case scenario.

Limitations

Doesn't prove success – Failing to reject the null hypothesis doesn't prove your campaign meets the target; it just means you can't conclude it doesn't meet it.
Sample size matters – Small samples increase the width of your confidence interval and reduce the test's ability to detect real differences.
Assumes normal distribution – For proportion data like conversion rates, this assumption may be violated with very small or very large proportions.
Context dependent – Statistical significance should be considered alongside practical significance and business constraints.

Where Else Can You Use This?

Product Launches – Determine if pre-launch performance metrics meet requirements before committing to full production.
Price Testing – Assess if a new pricing strategy's performance is consistent with revenue targets.
Customer Satisfaction – Evaluate if satisfaction scores after changes meet predetermined benchmarks.
Website Optimization – Verify if UX improvements achieve performance standards before wider implementation.
Supply Chain – Test if new logistics processes meet efficiency requirements before scaling.