💀Our Media Buyer Ruined Our Store

I got a “911” Slack message from one of our developers this morning regarding a client that we have…

We regularly run A/B tests on their site to improve it’s conversion and profitability, but the developers message said…

“WTF!!! [BRANDNAME] has 2 carts showing when you add a product to cart. WHO did this???”

When we looked a little deeper, it was in fact confirmed that there were TWO carts that would be created if you clicked “add to cart” on a product.

We knew one of our team members would never create this kind of change on the site so it had to be one of the client’s employees.

As it turns out, their Media Buyer made the change because he was trying to improve conversion with some A/B Testing of his own, but had made a mistake and messed up the cart.

If this had been a malicious change, it could have been catastrophic to the store.

Even though it was an innocent mistake from a well-intentioned employee, it still created pain for the developers, lost sales, and wasted money spent on ad traffic.

There are two very important takeaways from this little story…

Takeaway #1: ONLY HAVE ONE PERSON that is accountable for website changes.

If you have more than one, it is impossible to track improvements and/or errors.

Everyone on the team can be responsible for making sure the site looks good and is operating well, but only one person should be accountable.

Takeaway #2: Make sure your team knows how A/B Testing is done CORRECTLY.

Correctly performing A/B tests is essential to creating a successful e-commerce brand.

Making mistakes on this can make you feel like you’re swimming in circles in the middle of the ocean.

Why A/B Testing Matters

In the world of e-commerce, small changes can lead to big results. A/B testing allows you to make data-driven decisions rather than relying on guesswork.

We don’t have to have great intuition to be successful. We just have to let the data tell us what to do next.

This means each change you implement has the potential to increase sales, engagement, or any other goal you're aiming for.

It’s about finding out what resonates best with your audience.

What The Heck Is It?

We get it…

We’ve been saying “A/B Testing” a lot so far, and you might not know what that is… so lets clarify:

“A/B testing”, also known as “split testing",” is like conducting a science experiment on your website to find out what changes create the best results.

Imagine you have two versions of a webpage (A and B):

Version A might be your current design, while Version B has one key element changed, like a different headline or call-to-action button.

By splitting your traffic to the page and having 50% seeing Version A and 50% seeing Version B, you can see which one performs better in terms of converting visitors into customers. It's a direct comparison that provides clear results.

If you can remember high school science experiments, this might ring a bell for you. We have a “control” and a “variable,” but more on that in a little

The key is to only change one thing about your test so that you know what makes the difference.

If I am cooking a dish and I add three spices all at once, I don’t know which one of the three made it taste the way it does. It would make more sense to add one spice at a time and taste the difference the single spice makes to create the best possible dish.

How To Do A/B Testing

One of the best things about A/B testing is there are plenty of inexpensive apps like Gem Pages (Not Affiliated) that can do all this for you without the need for a developer or tech wiz.

The 6 Steps:

#1. Identify Your Goal

What’s your main objective?

It could be increasing newsletter sign-ups, boosting product sales, or improving user engagement.

Having a clear goal helps you measure the success of your test accurately.

It’s best to start the “closest to the money” which is Checkout.

Depending on what Shopify plan you are on you may or may not have access to test the Checkout page but you can use this simple Checkout Page Hack to start.

If you cannot A/B test your checkout, start with your cart, then go to your product pages, then collection page, then finally your home page. Working backwards “from the money.”

#2. Choose The Element to Test

Select one variable to test at a time!

Never test more than one thing at a time because you do not know what really made the difference.

This is the biggest mistake people make with A/B testing.

You can test completely different product pages though, for example a short form page vs. a long form page.

Pick a winner then start testing one thing at a time on the winner, like the product name for example.

Tests can be anything from the color of your “Buy Now” button, the images on your homepage, or even the phrasing of your product descriptions

Changing one element at a time ensures you know exactly what impacts your conversion rates.

#3. Create Two Versions (“A” and “B”)

Version A (the control) and Version B (the variation). Make sure they are identical except for the one element you are testing.

#4. Split Your Traffic

Use tools available for Shopify store owners like Gem Pages  that can help you divide your site traffic evenly between the two versions.

This ensures that your test results are reliable.

We recommend an even split (50/50) to get the most accurate results.

#5. Analyze the Results

After running the test long enough to gather enough data, analyze the data by looking at your results in your testing app.

You can look at things like Conversion Rate, Add to Cart Rate, and other metrics.

To us, the goal is always what makes the most money, but sometimes you are testing things like Newsletter signups. You can use OptiMonk for this.

What is “ long enough to gather enough data to make a decision?”

The answer is in the question: “how many clicks or visits do you have to each version?”

Depending on the size of your store, the minimum you want is 200 clicks or visits to the test (or 100 to each version).

The more the merrier though.

2000 clicks will give you a more clear winner than 200.

This test needs to happen in a relatively short time frame. 200 clicks over a year is not going to tell you anything. 

200 clicks in a week is ok. The longer you run your test the better too.

For example if you are getting 200 clicks a day and let the test run 30 days, it takes unknow variables like weekends and holidays that can lift or reduce sales out of the equation.

So which version achieved more of your set goal? If version B increases conversions by a significant margin, consider implementing the change.

#6 Keep Testing

Even if you find a winning version, the world of e-commerce is always evolving.

Regular testing helps keep your site optimized and ahead of competition.

Tools for A/B Testing

Fortunately, Shopify has several apps that make A/B testing straightforward, such as Optimizely, Google Optimize, and VWO.

These tools can help you set up tests, split your traffic, and provide detailed analytics on the performance of your tests.

If you are a small team and not super techy, use something like Gem Pages. We have no affiliation with Gem Pages and we do not get paid to promote them.

It is just a simple app that builds great pages and lets EVERYONE do A/B testing regardless of their skill sets.

Wrapping It Up

A/B testing is a powerful strategy to enhance your Shopify store’s performance. By continually testing and optimizing your website, you can understand your customers better and provide them with experiences that convert more effectively.

Start with a simple test today, and you’ll be on your way to maximizing your store’s potential.

This knowledge can help you and your team from creating the same problems we dealt with this morning.

Remember, every change is an opportunity to learn more about what drives your customers and how to fulfill their needs effectively.

Feel free to reach out with any questions or if you need further insights into getting started with your first A/B test.

To YOUR Success,

-Brent and Rush