Fossil Group

In the ever-changing world, we are in, adaptability is key. But how can we adapt to an environment that is constantly changing? How quickly can we learn and shift strategies? Who could we ask for guidance if this is new to everyone?

We have experienced the outdated method of making decisions based on gut. Data-driven decision-making is the new norm. It is also a popular phrase, sure to win you points in any business meeting.

Being an analyst, I respect the desire to make data-driven decisions. After all, facts are facts. They are the guiding light and the framework that grounds us. No matter how you feel, what you think or believe in, or what you really want, facts remain facts. Thoughts and emotions are irrelevant. This is a powerful thought and what makes A/B testing worthy.

A/B testing enables us to make changes in a controlled environment so that we can confidently understand the impact a change has on customer behavior, conversion, and engagement.

Think about how many times we wish to implement a change on the website or create an experience or add beautiful content or phrase marketing a particular way. We might be confident that what we are putting out there is easy to understand, witty, or a seamless experience. However, we lack different points of view. We see things through our own lenses, through our own experiences. This is where testing comes in. It allows us to make a change, to track the impact, and to learn whether we were right or wrong in our thinking. When we are right, we move forward. When we are wrong, we avoid mistakes and learn big lessons about our customers.

Beginning Testing

Start with a problem you wish to address or start with an idea to improve something that already exists. Let us follow one example all the way through. We believe that the “Add to Bag” button is not visible enough so we wish to move it up in the product pages within our website.

Prioritizing Ideas

Create a system that allows the best ideas to surface, one that does not consider who submitted the idea. The goal is to remove the emotion from the prioritization process. I could write a whole article on this, but in summary, come up with a process that gives ideas points, either zero or one, no gray areas, no in-between. For example, will the test be seen by over 50 percent of traffic to the website? If the answer is yes, the idea gets one point. If the answer is no, the idea gets zero points. In our example, our idea would score one point since more than 50 percent of people view product pages.

“A/B testing enables us to make changes in a controlled environment so that we can confidently understand the impact a change has on customer behavior, conversion, and engagement.”

After you have come up with many criteria (around 10 to 12), test the prioritization. Rate 10 ideas through the system and review the results. If an idea did not get proper points, why? What is missing? This will highlight criteria that need to be added, such as “Does it lessen manual workload?”

Writing a Hypothesis

Formulate a hypothesis by stating “If … then … because …” Be bold and specific in the hypothesis. A good question to help you determine your hypothesis is, what will you do if there is no change? Which version would you implement if the results are flat?

In our example, the hypothesis could be: If we move the “Add to Bag” button higher up in the product pages then the add to cart rate will increase and thus conversion rate would increase because customers would clearly know the next step to follow and it would eliminate friction.

Executing a Test

You will have many details to iron out, such as who you will target, how long the test will run (many online calculators can help you determine the length based on your sample size), and when is someone a part of the experiment (when they view a specific page).

Tip: Don’t test something you would not implement.

In our example, we could target all visitors for ~4 weeks. A person would be a part of the test if/when they viewed a product page.

Reading Results

Pull the results and determine statistical significance. Track beyond the simple metrics. Look at interdependencies and search beyond one metric. You should always analyze with the story in mind, and not just read numbers.

Tip: Don’t base your results on raw numbers (such as revenue), but focus on ratios to determine significant results, especially if your groups are unevenly distributed.

In our example: let’s pretend that both the add-to-cart rate and conversion rate decreased. We also noticed that interactions with other components on the page decreased. This teaches us that the other components contribute to the evaluation phase and need to be more prominent on the page in order for customers to feel comfortable purchasing the product.

Implement Changes

If the test variation wins (the contender), implement the change. If the control wins, either formulate a different idea with your newly acquired knowledge or move on to the next test idea.

In our example, we would leave the control and test rearranging the other components or instead of moving the “Add to Bag” button, we could change the aesthetics of the button to make it stand out more.

Now, what does A/B testing have to do with being wrong? Just about everything. We are wrong so many times. Testing allows us to understand the reason why we are wrong thus avoiding mistakes. It is in these scenarios that we learn the most. Every single time I am proven wrong, I learn something that I didn’t know before.

So, embrace all the times you are wrong and happy learning.