In today’s data-driven marketing landscape, accurately measuring the impact of your campaigns is crucial. Enter incrementality testing — a powerful method to determine the true causal effect of your marketing efforts. In this comprehensive blog post, we’ll explore three approaches to incrementality testing: frequentist, simple Bayesian, and MCMC (Markov Chain Monte Carlo). We’ll use an email campaign as our working example and include visualizations to illustrate key concepts and results.

What is Incrementality Testing?

Incrementality testing aims to answer a fundamental question: “What additional value did our marketing campaign generate?” It helps marketers differentiate between customers who would have converted anyway and those who converted because of the campaign.

This visualization illustrates the concept of incrementality. Out of the total conversions, some would have happened organically, while others are directly attributable to our marketing campaign.

Why is this important for businesses?

Understanding incrementality allows businesses to:

1. Optimize marketing spend: Efficiently allocate marketing resources by understanding what truly drives conversions.

2. Identify the most effective channels: Pinpoint which marketing channels deliver the most value.

3. Avoid cannibalization: Ensure that new campaigns add genuine value rather than merely shifting user behavior from one channel to another.

The Email Campaign Scenario

Imagine an e-commerce company running an email campaign to boost sales. They want to measure the incrementality of this campaign. Here’s the setup:

– Total customers: 100,000

– Treatment group (received email): 50,000

– Control group (didn’t receive email): 50,000

– Conversion rate in treatment group: 5% (2,500 conversions)

– Conversion rate in control group: 4% (2,000 conversions)

This chart shows the difference in conversion rates between the control and treatment groups. Now, let’s explore how we can analyze this data using different approaches.

1. Frequentist Approach

Theoretical Basis

The Frequentist approach to statistics is based on the concept of long-run frequencies. In this framework, parameters are considered fixed but unknown quantities. The approach does not incorporate prior beliefs about parameters and relies solely on the data at hand.

Statistical significance is determined through hypothesis testing, often using p-values to decide whether observed effects are likely due to chance. This method utilizes sample data to compute the probability of observing the given or more extreme outcomes under the assumption that the null hypothesis is true.

Implementation

Step 1: Calculate the lift

Lift = (Treatment conversion rate — Control conversion rate) / Control conversion rate = (5% — 4%) / 4% = 25%

Step 2: Conduct a statistical test

We use a two-proportion z-test to determine if this difference is statistically significant. This test compares the conversion rates of the treatment and control groups to ascertain if the observed difference could occur by chance, and if there were actually no effect from the campaign.

Inference

• Z-statistic: 7.6271
• P-value: 2.3981e-14
• Inference:
• The observed z-statistic being so far from zero indicates that the difference between the treatment and control group conversion rates is highly significant.
• The extremely low p-value (close to zero) strongly rejects the null hypothesis, suggesting that the marketing campaign had a significant incremental effect on conversions.

What does this mean for businesses? The frequentist approach provides a clear, binary decision: either the campaign had a significant effect or it didn’t. In this case, with a very small p-value (< 0.05), we conclude that the email campaign had a significant incremental effect. The visualization shows the observed z-statistic far in the tails of the standard normal distribution, corresponding to a very small p-value, indicating strong evidence against the null hypothesis.

2. Simple Bayesian Approach

Theoretical Basis

The Bayesian approach to statistics involves updating the probability estimate for a hypothesis as more evidence or information becomes available. It combines prior beliefs with the likelihood of the observed data to form a posterior distribution.

Bayesian methods treat parameters as random variables and use probability distributions to describe uncertainties around them. This approach is inherently subjective as it requires specifying a prior distribution based on expert knowledge or past data.

Implementation

Using a simple Bayesian model, we update our beliefs about the effectiveness of the campaign based on the observed data. We start with a prior distribution reflecting our initial understanding or past experiences. As we incorporate the data from the current campaign, the Bayesian updating process yields a posterior distribution that provides a new, refined estimate of the campaign’s effect.

Inference

Understanding the 95% Credible Interval:

• Range: The 95% credible interval for the lift (0.1803 to 0.3230) represents the interval within which the true lift of the marketing campaign is likely to lie with 95% confidence, based on the Bayesian model and the data observed.
• Interpretation: This range tells us that there is a 95% probability that the true lift due to the marketing campaign increases conversion rates by between 18.03% to 32.30% over the control group. It quantifies the uncertainty in our estimate of lift, taking into account both the data and our prior beliefs.

What does this mean for businesses?

The Bayesian approach provides a distribution of plausible values for the campaign’s effect, allowing businesses to make more nuanced decisions. The top graph shows the posterior distributions of conversion rates for both groups, while the bottom graph shows the distribution of the lift.

• Decision Making: This credible interval can be used to make informed decisions about continuing, adjusting, or halting the marketing campaign. If the lower bound of the interval (18.03%) exceeds a predetermined threshold of acceptable ROI, the campaign can be considered effective enough to continue or expand.
• Resource Allocation: Understanding the range of plausible lifts helps in allocating marketing budgets more effectively. For instance, if the upper limit is significantly high, it may justify increased investment in the campaign or similar campaigns in the future.
• Risk Assessment: The width of the interval provides insight into the risk associated with the campaign’s effectiveness. A narrower interval indicates more certainty about the lift’s estimate, whereas a wider interval suggests higher uncertainty and potential risks in banking heavily on the campaign’s outcomes.
• Scenario Planning: The interval allows for scenario planning under different assumptions of lift. Businesses can plan for various scenarios based on different points within the interval, from conservative estimates using the lower bound to more optimistic estimates using the upper bound.

3. MCMC Approach

Theoretical Basis

MCMC (Markov Chain Monte Carlo) is a class of algorithms used in Bayesian statistics to generate samples from a probability distribution where direct sampling is difficult. These samples are used to approximate the posterior distribution of the model’s parameters.

MCMC methods allow for the exploration of posterior distributions that are not analytically tractable. By constructing a Markov chain with an equilibrium distribution equal to the target posterior, MCMC methods facilitate the drawing of dependent samples used to approximate expectations with respect to the posterior.

Implementation

For our email campaign example, we’ve used an MCMC algorithm to simulate the effects of the campaign over time, considering additional variables such as the number of days since the last purchase. This method allows us to explore complex models that incorporate interactions between multiple variables, providing a detailed and nuanced understanding of the effects.

Inference:

• The lift is highest shortly after a previous purchase and decreases as the time since the last purchase increases, indicating a higher responsiveness to the campaign among recently active customers.
• The uncertainty band around the lift estimates suggests variability in the lift, which could be due to different customer behaviors or external factors not captured in the model.
• This approach provides insights into how the campaign’s effectiveness varies over different customer segments based on their recent engagement, allowing for more targeted marketing strategies and resource allocation.

What does this mean for businesses?

The MCMC approach allows for more complex models that can incorporate additional variables to provide a more nuanced understanding of incrementality. Here’s what businesses can learn from this analysis:

1. Personalized Incrementality: The model accounts for individual customer characteristics (days since last purchase), allowing businesses to understand how the campaign’s effectiveness varies across different customer segments.

2. Nuanced Insights: The “Lift by Days Since Last Purchase” graph shows how the effectiveness of the email campaign varies based on how recently a customer has made a purchase.

This reveals that:

a. The campaign has a higher lift (greater incremental effect) for customers who have made a purchase more recently.

b. The lift gradually decreases for customers who haven’t made a purchase in a longer time.

c. There’s a point of diminishing returns, after which the lift starts to plateau for customers who haven’t purchased in a very long time.

3. Targeted Strategies: Based on these insights, businesses can develop more targeted marketing strategies:

a. Prioritize sending promotional emails to customers who have made recent purchases, as they are more likely to be influenced by the campaign.

b. For customers who haven’t purchased in a while, consider different types of campaigns or incentives to re-engage them.

c. Optimize email frequency based on the recency of customer purchases to maximize impact and minimize fatigue.

4. Resource Allocation: The lift variation across customer segments allows for more efficient allocation of marketing resources:

a. Invest more in campaigns targeting recent purchasers where the ROI is likely to be higher.

b. Reallocate resources from low-lift segments to higher-lift segments or to testing alternative strategies for those segments.

5. Continuous Improvement: This model provides a framework for ongoing optimization:

a. As new data is collected, the model can be updated to refine estimates and track changes in campaign effectiveness over time.

b. Different variables (e.g., customer lifetime value, product categories) can be easily incorporated to gain deeper insights.

Conclusion

Incrementality testing is a powerful tool for marketers to measure the true impact of their campaigns. Each method we’ve explored offers unique insights:

– The Frequentist approach provides a straightforward, binary decision on the effect of the campaign, useful for quick go/no-go decisions.

– The Simple Bayesian approach offers a probabilistic view that incorporates uncertainty and prior knowledge, allowing for more nuanced decision-making.

– The MCMC approach handles complex models and provides detailed insights into parameter uncertainties, enabling highly targeted and personalized marketing strategies.

The choice of method depends on your specific needs, data complexity, and comfort with statistical techniques. However, the MCMC approach, as demonstrated, offers the most comprehensive insights by allowing the incorporation of multiple variables and their interactions.

By leveraging these advanced incrementality testing methods, businesses can:

1. Make data-driven decisions about campaign effectiveness

2. Optimize marketing spend across different customer segments

3. Develop highly targeted marketing strategies

4. Continuously improve campaign performance over time

Remember, the goal is not just to measure incrementality, but to use these insights to optimize your marketing strategy, allocate resources effectively, and ultimately drive business growth through more effective, personalized marketing campaigns.

As you implement these methods, keep in mind that the quality of your insights depends on the quality of your data and the relevance of the variables you include in your model. Regularly revisit your incrementality testing approach, incorporate new data sources, and stay open to refining your models as you gain more insights into your customers and campaigns.