Strategy

Email A/B testing in Klaviyo: what to test and how.

Most tests we see in audits change three things at once and get called after a few hours. Here is how we run them instead.

One element at a time, and at least 1,000 impressions per variation before you call it. That is the rule we use for pop-up tests, and email A/B testing works the same way. Change two things at once and you won't know which one made the difference.

The rules for email A/B testing

  1. Test one element. Subject line or button color, never both. If you want to test a whole new layout, treat it as one test with one question: does the new layout win?
  2. Write down the question first. "Does putting the code in the preview text get more orders?" gives you a clear yes or no. A vague "let's try something new" never does.
  3. Wait for enough data. For pop-ups we want 1,000 impressions per variation. For emails there is no magic number. Each version needs enough recipients that a handful of opens or one big order can't swing the result.
  4. Pick the metric that matches the change. More on that below.
  5. Keep the winner and test the next thing. A test only pays off when the result goes into the next send.

What to test in subject lines

This is where we start on most accounts, because every campaign is a fresh chance to test. These are the variables we rotate through:

  • Longer vs. shorter subject lines
  • Caps vs. lowercase
  • Emoji vs. no emoji
  • Preview text vs. no preview text
  • Sale code shown in the preview vs. hidden inside
  • First-name personalization vs. none
  • Plain transactional copy vs. a cultural reference

Our subject line guide has the rules we write by, plus examples of pairs that work.

What to test in flows

Flows are where a test pays off longest. A winning campaign email helps on that one send, while a winning flow email keeps running every day until someone changes it. And one of the mistakes we flag most in audits is flows nobody has touched in six months or more.

Good flow tests, from our own work:

  • Designed vs. plain text for the last email of a sequence. Plain text often wins for personal, founder-style sends. See our plain text guide.
  • When the discount comes in. Email 2 vs. email 3 of an abandonment flow, for example.
  • The offer itself, percentage off vs. a free gift.
  • Subject lines on the first email of each flow, since that one gets the most sends.

When click rates fall under 1%, testing is part of our fix list, together with clearer buttons, fewer links and a plain text version.

What to test in pop-ups

Pop-ups have a clear test order: offers first, then design, then timing. The offer is the biggest reason people sign up, so it gets tested before anything cosmetic.

OrderTestExample variations
1Offer% off vs. $ off vs. free gift
2Design3 variations
3TimingShown after 6, 9 or 12 seconds

The full setup is in our pop-up guide.

Opens, clicks or orders

The metric has to match what you changed. A subject line can only affect whether someone opens, so judge it on opens. But anything inside the email, from the header to the button text, gets judged on clicks or placed orders.

And be careful with opens in general. Apple Mail's privacy protection registers opens that never really happened, so open rates run high and noisy. They still work for comparing two subject lines sent at the same time to the same audience. For anything else, use clicks.

Test segments, tooNot testing segment performance is one of the segmentation mistakes we see most. Send the same campaign to your 30-day and 90-day engaged segments now and then, and compare clicks per recipient. It tells you how far you can widen your sends without hurting results.

How to set it up in Klaviyo

In a campaign, you create an A/B test, build the variations, choose how much of the audience gets the test and pick the winning metric. Klaviyo can send the winner to the rest automatically after the test window, or you pick it yourself.

In a flow, you add an A/B test to a single email in the flow builder. Klaviyo splits the profiles reaching that step between the versions. Let it run until both versions have enough sends, then end the test and keep the winner.

For SMS, one test earns its keep: MMS with a product image vs. plain SMS. MMS costs 2 to 3 times as much, so it has to bring in clearly more revenue to stay.

Watch Leon walk through it

Free on YouTube, in German.

Questions we get a lot

What should I A/B test first in email?

Subject lines. Every campaign gives you a new chance to test one, and the open rate shows a difference fast. On pop-ups, test the offer first.

How many people do I need for an email A/B test?

For pop-ups we wait for at least 1,000 impressions per variation before deciding. For emails, make sure each version reaches enough people that one lucky open or order can't decide the result.

Can you A/B test a flow in Klaviyo?

Yes. You add an A/B test to a single email in the flow, and Klaviyo splits the people reaching that step between the versions. Flows are worth testing because the winner keeps running for months.

Should I judge an A/B test by opens or clicks?

Judge subject line tests by opens and content tests by clicks or placed orders. Opens are inflated by Apple Mail's privacy features, so don't use them for anything below the subject line.

Want to see where your extra revenue is?

Book a free 30 minute audit. We go through your Klaviyo together and show you what we would change first.

Get a free audit