Skip to main content
You set up A/B tests and send timing in the last two steps of the Send campaign wizard: A/B test and Review & send.

​
A/B tests

An A/B test sends two to five variations to a sample of your audience. Once the decision window ends, the best-performing variation goes to everyone else in the audience.

​
What you can test

The content, sender and subject of the campaign itself are edited on the campaign page. This step only sets up the variations.

​
Set up a test

1

Turn on the test

In the A/B test step, switch on Enable A/B test.
2

Choose the test type and winner metric

Pick the Test type. Under Winner by, choose Open rate or Click rate.
3

Set the sample and decision window

Test sample (%) is the share of the audience that receives the variations. The default is 20%. Decide after is how long to wait before picking a winner: 1, 2, 4, 8, 12 or 24 hours.
4

Fill in the variations

Variation A starts with the campaign’s current subject or template. Fill in B, and click Add variation for up to five in total. For a content test, choose a template for each variation from My Templates.
5

Start the test

Continue to Review & send and click Start A/B test. To start the test later, turn on Schedule for later first.
The wizard warns you if the sample is too small to put at least one contact in each variation. If the sample covers the whole audience, there’s no one left to receive the winner.

​
How the winner is picked

When the decision window ends, InstantCampaign compares the variations on your chosen metric using a statistical significance test.
  • If one variation is clearly ahead, it’s declared the winner and sent to the rest of the audience.
  • If there’s no significant difference, the variation that’s ahead is still sent to the rest. The results say so.
Each variation appears as its own row in the campaign’s Send History (Variation A, Variation B and so on). The winning send is labelled Variation A · winner, for example.

​
Read the results

Open the campaign’s Analytics and scroll to A/B test results. You’ll see:
  • which variation won, or No significant difference — the leader was sent to the rest
  • the confidence, the lift over the runner-up, the p-value and the sample size
  • a table with delivered, open rate, click rate and bounce rate for each variation, plus revenue where you track goals
The AI strategist also reads your most recent A/B results when it plans new campaigns. See AI campaign strategist.
Every variation, and the winner sent to the rest of the audience, uses the Sender Profile and Send policy saved in Sender & Subject on the campaign page. In a Sender name test, each variation replaces only the “from” name.

​
Scheduling

In Review & send, choose one of these options. If you turn none of them on, the campaign sends as soon as you confirm.

​
Schedule for later

Switch on Schedule for later and pick a date and time. The time is in your browser’s time zone. The hint below the field also shows your workspace time zone for reference.

​
Send at each contact’s local time

Switch on Send at each contact’s local time and pick an hour, for example 9:00 AM. The campaign is split by contact time zone, so each contact receives it at that hour where they live. Contacts in New York and Tokyo both get it at 9:00 AM local time. Contacts without a time zone on their profile are sent at the scheduled time, or right away if you didn’t schedule the campaign.

​
Send-time optimisation

Switch on Send-time optimisation to send each contact the email at the hour they usually open, based on their past opens and clicks. Contacts with no history are sent at the scheduled time, or right away.
Send at each contact’s local time and Send-time optimisation can’t be combined: turning one on turns the other off. Neither is available for an A/B test.

​
Tips

  • Test one thing at a time. A content test with different subjects changes two things at once, so you can’t tell which one made the difference.
  • With small audiences, raise the sample percentage or use two variations only, so each variation gets enough recipients to give a meaningful result.
  • A split send (by local time or send-time optimisation) shows up as several rows in Send History, one for each delivery time.