In the marketing world, the value of testing is well known.
Testing allows you to identify what works and doesn’t work when attracting, engaging, and converting your target customers, making improvements as needed. But this can only happen if your approach to testing is flawless. Unfortunately, you’re likely to encounter various testing pitfalls that can render your data useless. In these situations, you’ll end up having to restart the testing process, or only recognize these errors after they’ve skewed your data.
In any case, your best bet is to avoid these pitfalls altogether by recognizing what these problems are and knowing how to handle them when they surface.
More from PostFunnel on testing:
A/B Testing Your Email Campaigns: Benefits and Drawbacks
Testing, Testing, A/B/C
1. Focusing on Surface-Level Information
Whether testing the copy of a landing page, the timing of an email or the call-to-action of a Facebook Ad, marketers use tests to determine which path leads to more engagement. Once identifying the better option, they can implement the changes and move onto the next campaign, right?
Not exactly.
The problem with taking test results at face value like this is that the knowledge isn’t transferable—and won’t provide much value to you in the future. Take a look at the following screenshots:
(Source)
The version on the right converted 100% more visitors than the one on the left. The message: removing the navigation bar on this page makes visitors more likely to convert. This, however, is just the beginning.
From the moment you create a test, you need to know how the data will help you improve not only this marketing initiative, but all of your marketing efforts moving forward. To do so, you’ll need to contextualize the test by forming a hypothesis.
This involves:
- Noticing a discrepancy in engagement data (such as an above- or below-average clickthrough rate for a landing page)
- Determining potential reasons for the discrepancy
- Identifying potential changes to be made
- Defining proper KPIs to measure the effectiveness of the changes made
Going back to the above example, the team at Yuppiechef posited that removing the navigation bar would allow visitors to focus more on the page’s main content (and on the call-to-action, as well)—a hypothesis which proved to be true.
But, again:
In addition to knowing which version of this page to use, Yuppiechef also now knows to remove their navigation bar on similar registration-focused pages on their site.
While knowing what works best for a given initiative is important, knowing why it works best will allow you to implement impactful changes to all your marketing efforts moving forward.
2. Testing Inconsequential Elements
Marketers sometimes conflate the phrase “always be testing” with “test everything.” While you should always test part of your marketing initiatives, testing everything is a waste of valuable time, money, and resources. Often, the reason teams take a “test everything” approach is that they aren’t exactly sure what they should test. Instead of focusing on factors that affect audience engagement levels, marketers can spend their time on tests that have little to no impact on customer engagement.
Let’s go back to our previous example from Yuppiechef:
Imagine if instead of removing the navigation bar, the team changed the color of the “Live Chat” button in the top-right corner from blue to green. Do you think one version of the page would have a 100% higher CTR than the other?
Nope. The Live Chat button has nothing to do with the central focus of the page, and therefore nothing to do with an individual’s decision to engage with the page’s call-to-action.
This is why forming a pre-test hypothesis is so important: it enables you to focus on the elements that actually matter to your audience. Only after determining the potential impact of making a change, should you consider testing the change.
3. Too Little or Too Much Data
The sample size of your tests can have a major impact on the test results and your company’s bottom line. Failure to collect a sufficient amount of data ruins the entire test. You can’t assume that a minimal amount of data is representative of your entire audience base.
This is where statistical significance and credibility come in:
- Statistical Significance refers to the threshold amount at which a set of results can represent the whole
- Statistical Credibility refers to the degree to which a chance initiative led to a change in customer behavior
Marketers must collect enough relevant data to make an informed decision moving forward, without ‘over collecting’ and wasting resources.
Check out Optimove’s blog for more on how to determine statistical significance and credibility for your individual tests.
4. Taking Too Scientific of an Approach
Testing is often a balancing act between maintaining scientifically-sound processes and maintaining focus on your business goals. The entire point of marketing tests is to determine a change’s impact on audience engagement. To measure this impact, you need to run multiple tests, altering only a single element, keeping everything else constant. The scientifically-sound thing to do is to eliminate—or minimize—any other deviations while testing a specific element. For example, if you’re testing multiple versions of copy for a Facebook Ad, you’d use the same version for each image.
Note: While multivariate testing is useful for determining which ad is more appealing, it won’t help you decipher which ad’s copy is more engaging. This goes back to defining your purpose for the test upfront: Since we’re looking for specific information on a single element, we want everything else to stay constant. The reality is that you can’t conduct marketing tests in isolation.
The good news:
Your marketing tests don’t need to be scientifically sound to the same degree as science experiments. As data scientist Lukas Vermeer explains in an interview with ConversionXL, all marketers deal with unavoidable “noise” when running tests; worrying about it will only cause you to fall further behind your competition.
The lesson to take away:
When running marketing tests, take full ownership of every variable you have control over—but don’t let that which is out of your control stop you from running the test in the first place.
5. Failure to Improve Your Bottom Line
Let’s say you test various elements of a marketing campaign, making adjustments to ensure your audience sees the best version of the campaign possible. Your tests did what they set out to do, help you optimize the marketing initiative. The question is:
Did optimizing the marketing initiative lead to significant gains for your business?
Unfortunately, it’s possible to optimize a given initiative—and even spur engagement from it—but still not reap much financial reward from your efforts. For example, you might develop a highly-engaging Facebook Ad through multiple rounds of testing—but come to realize your poor landing page fails to convert your audience. You need to ensure the big pieces of the sales funnel “puzzle” are in place before testing and optimizing the “little things” along the way.
Your ultimate goal for your marketing initiatives isn’t just to get your audience to take the next step—it’s to get them to take all the necessary steps toward conversion. If you’re unsure of whether testing a certain element will allow you to reach this ultimate goal, chances are there are more important elements you should be testing first.
Wrap Up
If your approach to testing your marketing campaigns is flawed, it’ll be impossible for you to optimize your campaigns. In fact, a flawed testing approach can even cause changes that negatively impact the customer experience. If you aren’t aware that your testing is flawed, you won’t even recognize the damage you may be doing. On a more positive note, maintaining a solid approach to testing your marketing initiatives can lead you to do less of what doesn’t work and more of what does. In turn, you’ll be able to continually enhance your customer’s experience with each new initiative.