Skip to main content
Attribution tells you which email a buyer saw before buying. It cannot tell you whether they would have bought anyway. A holdout can: a random share of your contacts gets no marketing email, and InstantCampaign compares their conversions with everyone who was mailed. The difference is the lift, what the email actually caused.

Turn on a holdout

1

Agree it with your client first

Contacts in the holdout get none of your marketing email while it is on. Transactional mail (receipts, password resets) always goes out.
2

Set the percentage

Settings → Send policies → Holdout group: up to 20% of contacts, for every campaign and automation. An automation can also have its own holdout of up to 50% in its settings panel.
3

Record conversions

Lift counts goals (purchases, sign-ups and other events you mark as goals) from the website snippet, the API or Shopify, and the conversion attribution report lists the goals you already record.
Who is held out is decided by a fixed hash of each contact, so the same people stay in the control group on every send, and nobody picks them. Raising the percentage only adds people.

Read the lift

The Holdout group card shows the last 30 days, an automation’s settings panel the last 90, and a campaign’s page and analytics page show that campaign against the control group it skipped. Each readout shows:
  • the conversion rate of contacts who received marketing email and of the control group;
  • the difference in percentage points with its 95% confidence interval, and whether it is statistically significant (the interval excludes zero);
  • revenue per contact, per currency (currencies are never added together).
Until each group has at least 100 contacts, the readout says there is not enough data instead of printing a number. Spark and the MCP server answer the same question with the get_lift tool, which returns totals only.

Share a lift certificate

A lift certificate is a branded page you can send to a client or a manager. It shows both groups’ conversion rates, the absolute and relative lift, the confidence interval and p-value, the group sizes and how the lift was measured, with your logo and colours. It shows totals only, never contacts.
1

Open Share lift certificate

On a campaign’s page or its analytics page (Proven lift card), or on the Holdout group card for a whole period.
2

Pick the dates, expiry and language

Up to 365 days of results. The link expires after 30, 90, 180 or 365 days. The page can be in any of the eight dashboard languages.
3

Copy the link

Anyone with the link can open it until it expires. The numbers are frozen when you create it, so the page never changes.
Revoke a certificate from the list under the button and its link stops working at once. Only owners and administrators can create or revoke certificates. If a group has fewer than 100 contacts, no certificate is created.

Lift in scheduled reports

Tick Proven lift (control group) in a scheduled report to add the same comparison for the report period, in the email, the online copy and the PDF. It covers the whole workspace; the report’s email filters do not apply.

Engaged reach instead of open rate

Apple Mail Privacy Protection loads the tracking pixel for every message it delivers, read or not, so a raw open rate counts mail nobody looked at. Engaged reach counts the people who opened as a person, clicked, or converted after a visit from the email, out of everyone the email reached. Campaign analytics, the comparison page and scheduled reports lead with it, and the raw open rate carries an explanation. To hide the raw open rate everywhere it is shown next to engaged reach, turn on Settings → General → Lead with engaged reach. Opens are still tracked and still count towards engaged reach. The getting-started guide recommends it. An automation never adds a holdout contact to an ad audience: the Sync to ad audience and Advertising campaign steps skip them and record why, so ads do not reach the control group either.