Email marketing can often feel robotic: You type a message on a machine, and an email marketing services sends it over to another machine. But in order to send better emails, you have to remember that there’s a human on the other side — and the humans on your list are different than those on other nonprofit newsletter lists.
You can always look into benchmarks and demographic data to get an idea of what kinds of messaging your audience will respond to and what the ideal send frequency would be, but you can never really know unless you test.
A/B testing, also known as split testing, is when you send 2 versions of an email to a segment of your audience and track which one gets the most opens or clicks. Whichever one wins is sent to the rest of your list. If the process sounds tedious, many email marketing services like Mailchimp have automated it to make it easy — even for email beginners.
Whether you are just dipping your toes into this tool or looking to dive deeper, here is a Mailchimp A/B testing guide to answer all your questions.
When to test
Mailchimp recommends A/B testing when you have a list size of 5,000 to ensure statistical significance. For organizations with smaller list sizes, don’t let this discourage you! You can still run A/B tests, you will just have to wait longer to get statistically significant results, and/or keep in mind that the results may just be suggestions rather than data-driven proof that one type of messaging works better than another.
When we say “always be testing” we mean it. Try to run an A/B test with every newsletter, changing out just one element each time. Once you send the test, we recommend waiting a few hours to get the results before sending to the rest of your list. Be sure to factor this timing into your schedule so you can maintain consistency there.
What to test
As mentioned above, we recommend focuses on just one element per test. Some ideas for A/B testing in Mailchimp:
Subject lines and preheaders
Check out our guide to writing excellent subject lines. Some of our favorite kinds of subject lines include those that are
- Time sensitive: “See you tomorrow”
- Exclusive: “Just for you!”
- Inserting a subscriber’s first name: “How do you want to change the world today, George?”
- Punny: “Don’t stop A/B-elievin’”
- Using emojis: “Don’t break our 💘”
- Topical: “Look what you made us do”
Also, don’t forget to try preheaders or preview text. Think of it as a continuation of your subject line and add narrative to further humanize your digital voice. This gives you more real estate to “sell” your email to the subscriber by informing them of what’s inside.
“From” name
Studies show that including changing the “from” name in your emails to a person’s name can increase open rates by 20%. If your CEO is well-known, or if you have celebrities attached to your organization, ask to use their names! Even if you don’t have a Yoko Ono at your organization, subscribers will be intrigued by the novelty of seeing your name in their inbox. From there, they will recognize your name over time. You can also use both a person’s name and your organization’s name (i.e.,: Meredith from Whole Whale) as a compromise. Test it out and see what works!
Content: Visuals and Calls-to-Action
Test different messaging or formatting within the email. Change up the copy, color, or position of the call-to-action. Try switching out the header image, inserting a gif, or embedding a video if applicable. Test inserting a subscriber’s name, location, or organization into the copy with merge tags. The possibilities are endless! But remember to choose only one per test.
Send time
This test will take a little longer as you’ll have to factor in time, but if you’re looking for the ideal time to send your emails, consider running a test to see when your subscribers are more likely to open emails.
You can look at external data on “industry” standards for your cause to get an idea of where to start. You can also look at your Google Analytics data to see when people are visiting your site and when you have higher goal completions. Try sending at the times when you know people are interested in your content. Or, consider sending on lower-traffic times to reach a different audience. Use this data to narrow down the number of day and time options, and then start sending A/B tests to find the right option for you.
Guide
The Nonprofit Organization’s Guide to Email Marketing
From lead generation to email segmentation to A/B testing, this guide is a one-stop-shop for bringing your email marketing strategy to the next level.
How to test
Now that you’ve chosen what you want to test, it’s time to get to it! Here’s a step-by-step guide to A/B testing with Mailchimp:
1. Go to “Create Campaign.” In the pop-up window, select “Create an Email.”
2. Choose “A/B Test” at the top of the window and name your campaign. Make sure your campaign name is clear and consistent with your other campaign names.
3. Choose the list, group, and/or segment you would like to send to. If you want to test content across different groups or segments, you’ll need to send separate emails rather than conduct a formal A/B test.
4. Choose the variable you want to test and the number of combinations (we recommend sticking to 2 to keep it simple
5. Decide what percentage of your recipients should receive your test combinations:
- We recommend sending the 2 options to 50% of your list, so 25% gets version A, and 25% gets version B. Then the remaining 50% of your list will get the winner. (Consider adjusting depending on the size of your list.)
- To test send times, you must test the 2 variations with your whole list. 50% gets the email at time A, and 50% gets the email at time B.
6. Choose how Mailchimp will determine the winner:
- If you are testing Subject Line, From Name, or Send Time, determine by Open Rate
- If you are testing Content, determine by Click Rate or Revenue (for donations)
- Give subscribers at least 2 hours to open or click the email before determining the winner
7. Set up the email as you would with any other email — you’re just writing two subject lines, from names, or preheaders. If you are testing send times, you will determine those here.
8. Design the email as you would any other — the only difference here is that you’re writing two content options if you are testing content
9. Confirm and schedule as you would with any other email. If you are testing send times, those will already be set in step 7.
Now that you know when to test, what to test, and how to test, it’s time to… actually test! To dive deeper with A/B testing, watch our recorded webinar, The Beginner’s Guide to A/B Testing with Salsa Labs.