Incrementality in Mobile Marketing: A Developer's Guide to Measuring True Campaign Lift

Three paid channels are all claiming credit for the same install. Your MMP dashboard shows healthy numbers, but CAC keeps climbing and you can't explain why. The real problem isn't attribution overlap — it's that none of your current data can answer the one question that actually matters: would those users have installed your app without ever seeing the ad?

That's what incrementality measures. As Adjust describes it, incrementality is “a way of measuring a marketing action's effect on a specific target KPI to determine whether the campaign is providing incremental lift, having a negative impact, or making no impact at all.” (Source: Adjust, A guide to marketing incrementality analysis) It doesn't replace attribution — it answers a fundamentally different question. The sections below cover how to define it, how to test it, and how to use the results to stop funding channels that are taking credit without driving growth.

Incrementality Definition: What It Actually Means

Incrementality Definition

The incrementality definition: incrementality is the measurable causal effect of a marketing action — the volume of conversions, installs, or revenue that would not have occurred without that specific campaign or channel. If 1,000 users installed your app after seeing an ad and 600 of them would have installed regardless, your incremental installs are 400. The other 600 represent wasted spend on users who were already going to convert organically.

This definition matters because it shifts the question from “how many installs did this campaign touch?” to “how many installs did this campaign cause?”

What Is a Synonym for Incrementality?

The closest synonym for incrementality is additionality — both terms describe the volume of outcomes that exist only because of a specific intervention. Marketers also use causal lift and marginal impact interchangeably with incrementality, particularly in academic and data science contexts. All three terms point to the same underlying concept: isolating the effect of a marketing action from what would have happened in its absence.

Incrementality Meaning in a Marketing Context

In mobile marketing, incrementality meaning is most relevant when evaluating paid user acquisition campaigns where organic install rates are non-trivial. If your app has strong organic search presence or word-of-mouth, a significant share of users attributed to paid channels may have arrived anyway.

Understanding incrementality meaning helps you distinguish between campaigns that manufacture demand and those that simply intercept it. Intercepting demand is cheaper — but it's not growth.

Incrementality vs. Attribution: Two Different Questions

What Attribution Modelling Tells You

Attribution modelling answers the question: which channel or touchpoint gets credit for this install? It maps a conversion event back to one or more marketing touchpoints — a click, an impression, a search — using rules or statistical models to divide credit.

What attribution modelling does not tell you is whether the install would have happened without any of those touchpoints. A last-click model attributes the install to the final ad clicked. A data-driven model distributes fractional credit. Neither model tells you whether the user would have found the app organically.

What Incrementality Tells You

Incrementality testing answers a different question entirely: did the ad cause the install? It does this by comparing a group of users who were exposed to an ad against a statistically equivalent group who were not — the holdout group. The difference in conversion rate between the two groups is the incremental lift.

This is causal measurement, not credit allocation. The output is not a percentage split of attribution credit — it's a number representing installs that would not have occurred without the campaign.

When to Use Each (and When to Use Both)

An MMP — a mobile measurement partner — is the infrastructure layer that makes attribution possible at scale. What is an MMP? It's a platform that ingests click and install data from ad networks, applies attribution logic, and reports which channels drove which conversions. MMPs answer “who gets credit?” Incrementality testing answers “did the ad actually cause the install?” — and you typically need an MMP's data as the input for designing and measuring an incrementality test.

Use attribution data for day-to-day campaign management and budget pacing. Run incrementality tests on a quarterly cadence — or when a channel's attributed performance no longer aligns with business outcomes.

Three Methods for Running Incrementality Tests on Mobile

Conversion Lift Tests (Exposed vs. Holdout Groups)

Conversion lift tests split your target audience into two groups before campaign launch: an exposed group that sees your ads, and a holdout group that is withheld from the campaign entirely. The incremental lift is calculated as the difference in install rate between the two groups over the test period.

Randomisation happens at the user level. Each user is assigned to a group based on their device id — typically by hashing the device identifier and applying a modulo split. This ensures the groups are statistically equivalent at the start of the test and that the same user cannot appear in both groups.

Here's a minimal example of how a developer might tag holdout cohort membership via an MMP SDK event parameter:

// At app initialisation, assign holdout group based on device id hash
val cohortGroup = if (deviceId.hashCode() % 10 < 2) "holdout" else "exposed"

// Pass cohort group as a custom event parameter to your MMP
mmp.setEventParameter("cohort_group", cohortGroup)
mmp.logEvent("session_start")

The holdout group should be suppressed at the ad network level — not just excluded from reporting. If holdout users still see ads through other placements, the test is contaminated.

Geo-Based Holdout Experiments

Geo holdout experiments match pairs of geographic markets — cities, DMAs, or countries — that are statistically similar in baseline install rates. One market runs the campaign; the matched market is held out. Lift is measured as the difference in install volume between the two markets.

This approach is the recommended method when user-level device id data is restricted by privacy frameworks like ATT or GDPR. Because randomisation happens at the market level rather than the user level, no individual-level identifiers are required. The tradeoff is that market-level variation introduces noise, and finding genuinely matched markets requires historical data and careful statistical pairing.

Media Mix Modelling (MMM)

Media mix modelling is a statistical regression approach that estimates the contribution of each marketing channel to a top-level business outcome — installs, revenue, or subscriptions — using aggregated spend and conversion data over time. It requires no user-level data, which makes it well-suited to cookieless or privacy-constrained environments.

For mobile app measurement, MMM's main limitation is granularity. It operates on weekly or monthly aggregated data, which makes it difficult to evaluate short-burst campaigns or isolate channel-level effects in small markets. It's better suited to upper-funnel brand spend than to performance UA optimisation.

The space is evolving: Google recently rolled out major improvements to incrementality testing, making it easier for advertisers to measure causal lift more accurately within their ad platforms. (Source: Google Makes Incrementality Testing More Accessible and Accurate, PPC News Feed) For most mobile app teams, conversion lift tests remain the most actionable method — but MMM is a useful complement when device-level data is unavailable.

How to Set Up an Incrementality Test: Step-by-Step

Step 1 — Define Your KPI and Hypothesis

Choose a single primary KPI before designing the test — installs, day-7 retention, or first purchase. Your hypothesis should be falsifiable: “Paid social activity on Meta drives installs that would not occur organically” is testable. “Meta is contributing to growth” is not.

Resist the temptation to measure multiple KPIs in the same test. Secondary metrics can be observed, but only one metric should determine whether the test result is positive or negative.

Step 2 — Calculate Required Sample Size and Test Duration

Statistical power determines whether your test can detect a real effect. The minimum detectable effect (MDE) is the smallest lift percentage you care about finding — if a channel is only 3% incremental, you may decide that's below the threshold of relevance. If it's 20%, that's worth detecting.

As a practical rule of thumb: for most mobile app campaigns, you need at least 10,000 users per group and a minimum of two weeks to get a reliable signal. Running shorter tests or under-powering the sample leads to wide confidence intervals and results that can't be trusted. (Source: Common mistakes in incrementality testing: 7 pitfalls to avoid) Use a sample size calculator before you start — not after the test fails to reach significance.

Step 3 — Configure Your Holdout Group in Your MMP

Most MMPs allow you to define holdout cohorts through custom audience segments or SDK event parameters. The holdout group should be defined before the campaign launches and suppressed at the ad network level — not filtered out in post-processing.

Step 4 — Run the Test Without Touching It

Do not adjust bids, creatives, or targeting during the test. Any mid-test change invalidates the comparison between your exposed and holdout groups. Do not pause spend on the holdout channel because it appears to underperform — that underperformance is the signal you're measuring.

Warning: Short tests and skipped cooldown periods are the two most common reasons incrementality results are unusable. (Source: Common mistakes in incrementality testing: 7 pitfalls to avoid) A cooldown period of at least three to five days after the test ends allows lagging conversions to register before you read results. Reading the dashboard on day one of the cooldown period will produce an artificially low lift estimate.

Step 5 — Read the Results: Lift, Confidence Intervals, and iROAS

Incremental lift is expressed as a percentage: (exposed conversion rate − holdout conversion rate) / holdout conversion rate. A 35% lift means 35% of installs in the exposed group would not have occurred without the campaign.

iROAS — incremental return on ad spend — divides incremental revenue by total campaign spend. A confidence interval that includes zero means the result is not statistically significant: you cannot conclude the campaign had a positive causal effect. Do not act on results with wide confidence intervals. Run a longer or larger test.

Incrementality Testing by Channel: Where It Works and Where It Breaks

Paid Social (Meta, TikTok)

Meta offers native conversion lift tests within Ads Manager, making it the most accessible entry point for teams new to incrementality testing. The limitation is data export — raw holdout-level data is not easily accessible outside Meta's own reporting, so third-party validation is difficult.

TikTok has introduced lift measurement tools, but they are less mature. For high-spend TikTok campaigns, geo holdout experiments often produce cleaner results than platform-native tools.

Google UAC and Apple Search Ads

Google's recent improvements to incrementality testing have made it more accurate and accessible for app advertisers. (Source: Google Makes Incrementality Testing More Accessible and Accurate, PPC News Feed) Google UAC's scale means lift tests can reach statistical significance faster than on smaller networks.

Apple Search Ads present a specific challenge: users searching for apps by category or competitor name have high organic intent. Incrementality lift on Apple Search Ads is often lower than reported attributed installs suggest, because a significant share of those users would have found the app through organic search anyway.

Email and Push Notifications

Email and push are well-suited to holdout testing because you control the send list. Withhold a random 10–20% of your audience from a campaign, measure the difference in conversion rate, and you have a clean lift estimate. The difficulty is that send-list holdouts only measure the value of the message — not the channel's contribution to awareness or intent.

Organic and Influencer Channels

Organic is the hardest channel to test incrementally because there is no control lever — you cannot selectively suppress your app's presence in the App Store. Geo holdouts are the most practical method, comparing install rates in markets with and without influencer or ASO campaigns.

Common Incrementality Testing Mistakes to Avoid

Running Tests That Are Too Short or Too Small

Mistake: Ending a test after one week with 2,000 users per group and treating the result as conclusive.

Fix: Use a sample size calculator before launch. If your install volume can't support 10,000+ users per group over two weeks, consider a geo holdout instead of a user-level split.

Not Accounting for Seasonality and External Events

Mistake: Running a test across a promotional event, a holiday period, or an app store feature — all of which shift baseline install rates in ways that contaminate your holdout comparison.

Fix: Schedule tests during stable, representative periods. If an external event occurs mid-test, document it and consider rerunning rather than interpreting a compromised result.

Contamination Between Test and Control Groups

Mistake: Holdout users see ads through retargeting, lookalike audiences, or cross-channel placements outside the suppression window.

Fix: Define holdout suppression at the ad network level across all active campaigns, not just the one being tested. Contamination consistently inflates holdout conversion rates and deflates measured lift.

Misreading Confidence Intervals as Certainty

Mistake: A test returns 18% lift and the team reallocates budget — without checking that the 95% confidence interval runs from 2% to 34%.

Fix: Wide confidence intervals signal an underpowered test. (Source: Common mistakes in incrementality testing: 7 pitfalls to avoid) Only act on results where the entire confidence interval sits above zero.

Ignoring the Cooldown Period

Mistake: Reading results the day the test ends and missing conversions that register two to four days after the final impression. (Source: Common mistakes in incrementality testing: 7 pitfalls to avoid)

Fix: Build a three-to-five day cooldown into your test timeline before pulling final numbers. The holdout group's lagging conversions are part of the measurement.

Using Incrementality Results to Allocate Budget

Calculate Incremental Cost Per Install (iCPI)

Incremental cost per install is calculated as: total campaign spend divided by incremental installs only. If you spent $50,000 and your MMP reports 5,000 attributed installs, but your lift test shows only 40% were incremental, your iCPI is $50,000 / 2,000 = $25 — not the $10 your dashboard shows.

That gap is the number that should drive budget decisions.

Cut, Shift, or Scale: A Decision Framework

Consider a hypothetical app running $25,000/month across two channels with similar reported install volumes in the MMP:

Channel A is driving real demand at a defensible cost. Channel B is largely intercepting users who would have installed anyway — at more than five times the incremental cost. The correct decision is to cut Channel B's budget and shift it toward Channel A until Channel A's marginal iCPI begins rising.

Combining Incrementality with Attribution Data for Full-Funnel Decisions

Attribution data tells you where users came from. Incrementality data tells you what was actually caused. Used together, they give you a complete picture: which channels are reporting installs accurately, which are over-claiming, and where scaling spend will produce real growth rather than just more dashboard numbers.

Incrementality findings should update budget allocation on a rolling cadence — quarterly at minimum. Install rates, organic baselines, and competitive dynamics shift. A channel that showed 40% lift six months ago may be down to 15% today.

Frequently Asked Questions