Enable Foundational Decision-Making with A/B Testing

In today’s digital and customer experience landscape, A/B testing is often viewed as a “must-have” skill set, strategy, and discipline, considered by most as a “no-brainer”. Whether it is referred to as A/B testing, A/B/n testing, hypothesis testing, or conversion rate optimization, what our experience at Analytics Metal has shown is that A/B testing is simultaneously one of the easiest concepts to grasp and the most challenging to pull off successfully.

Why is A/B testing so difficult to get right for so many organizations? The answer is that there are a number of factors at play, all of which influence the success or failure of A/B testing. In some cases, internal stakeholders are extremely conservative about making changes to website features and functionality, especially if those changes can affect eCommerce or lead generation pipelines, leading to very minimal or no A/B testing at all. In other cases, there will be a special promotional event, sweepstakes, digital marketing or email marketing campaigns directing traffic to the site, significantly curbing the appetite for A/B testing. Some of the time, the IT or the web development teams will have a code-freeze (a period of time where no changes are published, typically around the holiday season), disallowing any new tags or functionality to be implemented until further notice. Finally, a large reason why A/B testing is so easy but yet so difficult is that an organization will often not have the experience, skill sets, budget or bandwidth to execute and report on even one single and simple A/B test, let alone establish a culture of testing and optimization.

The good news is this: When it comes to A/B testing, the team at Analytics Metal has strategized, envisioned, executed upon and provided actionable insights and recommendations to organizations of all shapes and sizes, demonstrating real results tied to actual return on investment and tied to actual website and marketing performance. We are confident that we can do the same for your organization, working cross-functionally across your company’s digital, web, IT, and marketing teams to establish A/B testing as a core competency for your team.

Our Approach to A/B and Multivariate Testing

We at Analytics Metal realize that there are a lot of factors at play when it comes to A/B testing for any organization that is looking to move the needle on digital performance and earn the quick wins that can lead to a better, faster, more convenient and profitable user experience. Whether your organization engages with Analytics Metal for a single and simple A/B test or for a more complex series of multi-faceted multivariate tests, you will find that we will follow a repeatable structure that is flexible enough to accommodate precise needs.

1. Formulating a Hypothesis (Deciding What To Test)

The very first step in A/B or multivariate testing is deciding on what to test. Our Analytics Metal team can initiate this process by analyzing digital marketing campaign, website, or mobile app data, and utilizing our own internal experience in user-experience design (and in A/B testing) to bring opportunities to the surface for your organization. Often, we find that our clients also have their own hypotheses about what could be tested gleaned from their own pool of insights. Whether our team, your team, or a collaborative hypothesis is formulated, the best next-step is to map out how the test will be executed and how success will be measured. Determining these details leads into an initial discovery and goal-setting session, briefly outlined below.

2. Initial Kick-Off and Goal-Setting

Once a test subject has been identified, or if one is already identified, and after some details are planned for, an initial kick-off and goal-setting session is a typical next step. While gathering the particular details of a test - scope, timing, assets needed, sample size and required approvals - can also happen when formulating an initial hypothesis, our focus here is typically centered around the ultimate end-goal, which is how your organization will measure success or failure for the A/B test in question. Testing for testing’s sake without a tangible and performance-oriented goal to measure against is a fruitless endeavor. Some of the questions that we ask ourselves when we formulate our own hypotheses, and will be sure to ask during an initial discovery session if you bring your own hypotheses to the table, look like the following:

• What key metrics or KPIs will be used to evaluate success?
• How will the results of the A/B test affect decision-making?
• How will the results of the A/B test be reported internally to key stakeholders?
• Will there be a need to show the winning test subject to all visitors after test completion?
• Is there an appetite to retest during a different time of year or while a marketing campaign is in flight?

3. Gathering Assets and Building the A/B Test Experiment

Our team at Analytics Metal has many years of experience in platforms like Google Optimize, Adobe Target, Optimizely, Monetate, Visual Website Optimizer, and other A/B and multivariate testing systems. Regardless of which platform your organization uses, all of the content (call-to-action text, messaging, blurbs, etc…) will need to be collected to build the A/B test variants, as well as any image or creative assets, color hues or hex codes, links, pricing, or other elements that could potentially appear to a segment of your website’s audience. We take great care in precisely following your organization’s style guide and brand identity.

4. Test Launch and Initial Observation Period

After the test experiment(s) are built and tested for accuracy and for visual appeal across all of the major platforms and browser types, the A/B test is ready for launch! After “go-live”, we closely monitor and closely observe the initial experiment data, and begin predicting which test variation - control or treatment - will become the best performer over time.

5. Review of Test Results, Recommendations, and Subsequent Steps

While every A/B or multivariate test is different, a bare-minimum of ten consecutive in-flight days are recommended before results should be analyzed, and before any insights and actionable recommendations should be entertained. Typically, an A/B test reaches maturity within the first few weeks after “go-live”. However, sometimes it can take longer for an A/B test to reach the point where results are valid - the team at Analytics Metal will keep you well informed and current on the progress of any and all A/B tests that are in-flight. It is also worth noting that any A/B test can be switched off at any time and very quickly, which comes in handy if there is an unexpected website change or a sudden shift in strategy.

Our A/B testing approach is repeatable and dependable, traits that Analytics Metal aims to be for your organization.

Want to find out more about our A/B testing philosophy? Does your organization have an A/B test, or perhaps a series of tests that it is looking to conduct? Let our Analytics Metal team know, and we would look forward to speaking with you!