Test two versions of a campaign

An A/B test sends two versions of a campaign to a small part of your audience first. After a waiting time, the version that did better goes to everyone who is left.

#Set up a test

1. Open a draft campaign and go to the Message step.

2. Write your Subject line. This becomes Version A.

3. Press Test two subject lines. If the subject is empty, a note says "Write a subject line first."

4. A Version B box appears. Type Version B subject line. You can also type a Version B preview text. Both have the same character counters as Version A.

5. Under Decide by, choose Opens, Clicks or Orders.

6. Move the Test on slider to set how much of your audience is in the test. It goes from 10% to 50%.

7. Under Decide after, pick a wait from 1 hour to 23 hours.

8. Press Continue and finish the campaign as usual.

To remove the test, press Don't test in the Version B box.

#What the screen tells you

Under the slider, a sentence turns the percentage into people: "10% get A, 10% get B, the winner goes to the other 80% — about 3,200 people". Check that the numbers make sense.

If the groups are too small, a warning shows: "At this size a test won't tell you anything. Send one version." Each version needs at least 50 people in its slice. With an audience of 400 and a 20% slice, each version would get 40 people, so the warning appears.

The test saves only when both versions are different. If they match, a note says "Both subject lines are the same, so the test cannot tell them apart. Change one and it saves."

#Pick what decides the winner

• Clicks is the safest. A click is a person doing something.
• Orders picks the version whose email earned more revenue per person.
• Opens counts a lot of Apple Mail opens that nobody read. The screen says "Apple Mail opens email automatically for a lot of people, so this counts some who never read it. Clicks are the safer measure." The report repeats that warning when Opens decided the test.

We compare the rate per person sent, so two slices of slightly different size are fair to each other.

#What happens when you send

1. Version A goes to half the test slice and Version B to the other half. Each person is placed in a version by a fixed draw, so it is random but stable.

2. Nobody else gets anything until the waiting time ends.

3. When the time is up, we pick the winner and send it to the rest of your audience.

The Spread it out option on the Review step is switched off during a test. The note says "Not while you are testing two versions — the test needs everybody in the sample to get it at once."

#Read the result

Open the campaign. A box titled The test sits above the numbers.

• While it runs: "Still running. We'll pick a winner when the waiting time is up, and the other 3,200 people get it then."
• After: a sentence such as "B won: 4.1% against 3.2% (400 and 400 people)."
• If it is a tie, or one version reached nobody, it says so and asks you to choose.

When people are still waiting, you get a button. After a decision it reads Send version B to the other 3,200. If the test could not decide, there are two buttons: Send version A to the rest and Send version B to the rest.

#Whole-design tests

The Version B box also offers Whole design under What's different. When you pick it, the screen shows a note: "Version B needs its own design. Build it on the Design step, then come back and pick it here." That screen has no place to pick Version B's design. Until it does, test subject lines.

#If it didn't work

• The test won't save. Both subject lines are the same, or Version B is empty. Fix the one the note names.
• The slider stops at 50%. That is the highest slice allowed.
• No winner was sent. Look at The test box. A tie or an empty version asks you to choose.
• Spread it out is greyed out. It is switched off while a test is on.