Fossil Group

In the ever-changing world we are in, adaptability is key. But how can we adapt to an environment that is constantly changing? How quickly can we learn and shift strategies? Who could we ask for guidance if this is new to everyone? We’ve experienced the outdated method of making decisions based on gut. Data driven decision making is the new norm. It’s also a popular phrase, sure to win you points in any business meeting.

Being an analyst, I respect the desire to make data driven decisions. After all, facts are facts. They are the guiding light and the framework that grounds us.No matter how you feel, or what you think or believe in, or what you really want - facts remain facts. Thoughts and emotions are irrelevant. This is a powerful thought and what makes A/B testing beautiful.

A/B testing enables us to make changes in a controlled environment so that we can confidently understand the impact a change has to customer behavior, conversion, engagement, etc.

Think about how many times we wish to implement a change on the website or create an experience or add beautiful content or phrase marketing a particular way. We might be confident that what we are putting out there is easy to understand, or witty, or a seamless experience. However, we lack different points of view. We see things through our own lenses, through our own experiences. This is where testing comes in. It allows us to make a change, to track the impact, and to learn whether we were right or wrong in our thinking. When we are right, we move forward. When we are wrong, we avoid mistakes and learn big lessons about our customers.

Start with a problem you wish to address or start with an idea to improve something that already exists. Let’s follow one example all the way through: We believe that the “Add to Bag” button is not visible enough so we wish to move it up in the product pages within our website.

How to prioritize ideas:

Create a system that allows the best ideas to surface, one that does not consider who submitted the idea. The goal is to remove the emotion out of the prioritization process. I could write a whole article on this, but in summary, come up with a process that gives ideas points (either 0 or 1). No grey areas, no in between. For example: Will the test be seen by over 50% of traffic to the website? If the answer is yes, the idea gets 1 point. If the answer is no, the idea gets 0 points. In our example, our idea would score 1 point since more than 50% of people view product pages.

After you have come up with several criteria (around 10- 12), test the prioritization. Rate 10 ideas through the system and review the results. If an idea did not get proper points, why? What is missing? This will highlight criteria that need to be added, such as: “Does it lessen manual workload?”

How to write a hypothesis:

Formulate a hypothesis by stating: “IF … THEN … BECAUSE …” Be bold and specific in the hypothesis. A good question to help you determine your hypothesis is: what will you do if there is no change? Which version would you implement if the results are flat?

" Data driven decision making is the new norm. It’s also a popular phrase, sure to win you points in any business meeting "

In our example, the hypothesis could be: IF we move the “Add to Bag” button higher up in the product pages THEN the add to cart rate will increase and thus conversion rate would increase BECAUSE customers would clearly know the next step to follow and it would eliminate friction.

How to execute a test:

You will have many details to iron out, such as who you will target, how long the test will run (many online calculators can help you determine the length based on your sample size), when is someone a part of the experiment (when they view a specific page), etc.

Tip: Don’t test something you would not implement. In our example, we could target all visitors for ~4 weeks. A person would be a part of the test if/when they viewed a product page.

How to read results:

Pull the results and determine statistical significance. Track beyond the simple metrics. Look at interdependencies and search beyond one metric. You should always analyze with the story in mind, and not just reading numbers. Tip: Don’t base your results on raw numbers (such as Revenue), but focus on ratios to determine significant results, especially if your groups are unevenly distributed.

In our example: let’s pretend that both the add to cart rate and conversion rate decreased. We also noticed that interactions with other components on the page decreased. This teaches us that the other components contribute in the evaluation phase and need to be more prominent in the page in order for customers to feel comfortable purchasing the product.

How to implement changes:

If the test variation wins (the contender), implement the change. If the control wins, either formulate a different idea with your newly acquired knowledge or move on to the next test idea.

In our example, we would leave the control and test rearranging the other components or instead of moving the “Add to Bag” button, we could change the aesthetics of the button to make it stand out more.

Now, what does A/B testing have to do with being wrong? Just about everything. We are wrong so many times.Testing allows us to understand the reason why we are wrong thus avoiding mistakes. It is in these scenarios that we learn the most. Every single time I am proven wrong, I learn something that I didn’t know before.

So embrace all the times you are wrong and happy learning!