The split test (or also known as A/B test) is a test method for evaluating two or more variants of a system or a newsletter, in which the original version is tested against a slightly modified version.
The split test is available from a Pro account.

Create split test

To perform a split test, please proceed as follows: Go to your contact overview and, if necessary, filter for the contacts you want to test with.

Please note that you must select at least 500 confirmed contacts for a split test.

Click on "Newsletter Split Test" and on "Apply". 
Select when the split test should be performed: Right now or at a specific time.

ausführung.png

Create variants

In this area you can now determine the distribution and other settings of the test as well as create the different variants.

Please note that you can create a maximum of 5 different variants.

Only variant 1 is preset - but of course you need at least two to start the split test. With a click on the green button "New variant", you create another one.

You can also clone the current variant by clicking the blue "Clone Variant" button.

Assign a title for your variant by clicking on the tab small blue pencil in the tag. This title will not be displayed anywhere for the recipients and only serves you for a better overview. Especially if you want to fall back on an automatically saved draft, it is a good idea to give its variant a title - so you know immediately what kind of draft it is.

title variant.png

Now define at which time this variant should be sent, if you want to test different sending times.

You will be shown when the newsletter will be sent after the start of the test! If you have defined that the split test is to start at 10 a.m. the next day, for example, but you want to send variant 1 at 9 a.m., variant 1 will not be sent until 9 a.m. the following day.

If you do not want to test different times, leave it at "Immediately".

Now, as usual, you can define the e-mail sender and enter a subject for your newsletter.

Now define under "Email settings" which unsubscribe text you would like to use (you or you variant). Attention: This setting can always be found under the editor for the "normal" e-mails, but here at the top for the split test.

unsubscribe message.png
Now you can design the newsletter in the drag & drop editor (switching to the classic editor is not possible). Testing the newsletter is also available to you, as usual.

Once you have finished your first variant, click on the "Variant 2" tab and create your second alternative.

Automatic caching of your variants

Please note that automatic saving takes place every 30 seconds - quickly clicking back and forth between variants with minimal changes to each variant is not recommended, as the changes may then not be reflected in the automatic save.

You can find the temporarily saved drafts under "Load template" and the "Drafts" tab.

Split test settings

Now that you have created your variants, you still need to determine the success criteria and the distribution of the variants.

To do this, click on the Settings tab.

Give the split test a title so that you still know what you are testing or have tested at a later time.

Distribution (Ratio)

Now you can drag the slider and thus determine the distribution. Only up to 60% can be tested! The variant that runs better is then ultimately played out to the remaining 40%.

Note: If you use the slider on the far right, you can easily set an even distribution. If you use the other sliders, you can set the distribution individually (e.g. for 3 variants 10%, 30% and 20%).

Select winner

Under "How should the winner be chosen?" you have two options available:

After a certain time.

If you select this option, you still need to define a time period after the last split test email is sent. How long should you wait until the winner variant is sent to the rest?

Success criteria

Now you have to select which criterion is decisive for you in order to choose a variant as the winner.

You have the following options:

E-mail link clicked
Email opened *
Web tracking event
Tag changed

*This should be taken with a grain of salt: opens can never be tracked accurately!

The requirement for a system to detect email opening is that the recipient also loads all images. Only when the tracking pixel is loaded can the opening be detected.

Also, with Apple, emails are opened by a proxy server before they even reach the recipient. The emails are all counted as opened, even if the recipient has deleted them unseen. For these reasons, this metric should only be used as a guide.

Web tracking event

If you click this criterion, you must define the exact event to respond to on the right.

Tag(s) changed

Define whether you want to react to an assignment or withdrawal of one or more tags and, of course, to which tags.

If several tags have been selected, it is enough to change one of them to be counted as a success.

Edit/delete split test

In case you want to edit, pause or delete the split test, go to Contacts and Scheduled Runs in the menu. Here you will see the current split tests listed. By clicking on "Actions" you have the following options

Edit
Delete
Pause and
Execute now

are available.

Should you delete the test, it will no longer be listed under Scheduled Runs. However, you can still find it under Emails -> Split Tests.

View split tests

After you have started your split test(s), you can of course view them at any time and see the current status or edit the variants.

To do this, click on the "Emails" menu item and the "Split Test" tab. Here you will now see all the A/B tests that you are running and have run.

In the table you can see the numbering, the owner, the title of the split test, type, status and the creation date.

You can also choose whether you really want to view all split tests or only planned, running or completed ones.

Details

By clicking on the blue button "Details" you can view them.

Overview

In this tab you can see all the facts about the split test in tabular form: owner, creation date, and much more.

In addition, the statistics show graphically which variant ran better.

Below you can see the distribution and the complete evaluation. In the table you can see:

Hashtag: Variants
Percent: Percentage playout
Pin: title
Calendar: the date of the last dispatch
Paper flyer: the total emails sent for the variant
Eye: the openings
Chain: link clicks
Cup: the number of contacts who met the success criterion
Scale: here the contacts are displayed according to the weighting

Variants

By clicking on the respective tab (Variant 1, Variant 2 etc.) you will see the email statistics, which should look familiar to you from the "normal" emails.

As usual, you can view the contacts who received, opened and/or clicked on the email by clicking on the blue button.

In the lower area you will also see the variant.

Contact filter used

Under this tab you can view in detail the contact filter you selected for the split test. You can use this filter again by clicking the blue button.

Action

By clicking the Action button you can view all the contacts that were involved in the split test.