Mul­tivari­ate testing: the complex process for op­tim­ising the con­ver­sion rate

An optimised con­ver­sion rate is the goal of every developer or web project operator. First-class content and products, as well as con­stantly in­creas­ing traffic, are usually promising signs for upcoming shops and companies. Good as these might sound, they are useless if they don’t result in trans­ac­tions, clicks, or ac­quis­i­tions from completed forms. While the number of visitors can be in­flu­enced by quite different measures (social media marketing, SEO, SEA, link building, etc.), the amount of con­ver­sions is mainly due to two factors: how much trust the user has in the shop or website, and the web project’s user-friend­li­ness, which is also referred to as usability.

In order to con­sist­ently improve the latter, mul­tivari­ate testing is becoming in­creas­ingly popular among de­velopers and marketers. The advanced testing method, which is even more complex than the often-performed A/B testing, can detect weak­nesses among the elements being checked. These can then be rectified to make it easier for the user to access the content. If visitors enjoy their ex­per­i­ence they will spend more time in the shop or on the website and are more likely to carry out the desired action (con­ver­sion).

The following example il­lus­trates the approach to mul­tivari­ate testing: the web page of a certain product should be optimised since the de­scrip­tion and image used were selected as decisive test criteria. The analysis procedure is intended to determine whether and to what extent the two different product de­scrip­tions A and B (as well as the various product images 1 and 2) influence the con­ver­sion rate. For this purpose, the images and de­scrip­tions are combined to form the following com­bin­a­tions:

• De­scrip­tion A, Image 1
• De­scrip­tion A, Image 2
• De­scrip­tion B, Image 1
• De­scrip­tion B, Image 2

This simple example shows that there is a maximum of four different com­bin­a­tions – the number of possible com­bin­a­tions rises with each ad­di­tion­al variable. The four different versions are now presented to potential customers who access the shop’s website. The entire traffic is dis­trib­uted evenly among the four variants. The con­ver­sions (completed trans­ac­tions) are analysed during the review period using an analysis tool such as Webtrends Optimize. This is so that a con­ver­sion rate can be cal­cu­lated for all four variants after the mul­tivari­ate test is completed. The bigger the amount of traffic and the longer the ob­ser­va­tion period, the more sig­ni­fic­ant the results.

Compared to usability tests (which are un­der­taken in the project’s de­vel­op­ment phase), mul­tivari­ate tests have the advantage that a much larger number of par­ti­cipants can be included seeing as these tests are run once the website, app, or shop is online. These are known as quant­it­at­ive research methods. Although mul­tivari­ate testing seems extremely complex, it is actually quite easy when the right tools are used. The various test pages are quickly con­figured and im­ple­men­ted via JavaS­cript code snippet in the web project. The results are shown in real-time in clearly arranged tables so that it’s easy to see which com­bin­a­tions promise the greatest success.

Unlike A/B testing, mul­tivari­ate testing isn’t limited to two versions; there’s actually no lim­it­a­tion. This fa­cil­it­ates the veri­fic­a­tion of several different elements and also provides the marketer with com­pre­hens­ive insights into how various com­pon­ents interact with each other. Mul­tivari­ate tests don’t just reveal which com­bin­a­tions influence the con­ver­sion rate in a positive or negative way when it comes to in­di­vidu­al cases; they also provide a concrete, stat­ist­ic­ally proven picture of which com­pon­ents con­trib­ute to the overall success of a web project and in which ways. The knowledge gained during mul­tivari­ate testing can also play a major role in the de­vel­op­ment of future projects.

High traffic is mandatory for a mul­tivari­ate test to obtain results that are as accurate as possible since the traffic is split and dis­trib­uted to at least four test examples. An ad­di­tion­al problem with this testing method is when one or more of the tested variables have no influence on the con­ver­sion goal – es­pe­cially when it comes to in­ter­pret­ing the results. Here, mul­tivari­ate testing can quickly end up being the wrong choice because it ends up com­plic­at­ing the eval­u­ation process un­ne­ces­sar­ily. In this case, a simple A/B test might have sufficed.

The best way to test the usability of your web project depends on a number of different factors, with the amount of traffic playing the biggest part. If your web project is only recently up and running and you haven’t had many visitors as of yet, mul­tivari­ate testing won’t provide you with reliable results on the success of each variant. In such a case, it is re­com­men­ded to test the func­tion­al­ity of in­di­vidu­al variables in con­sec­ut­ive A/B tests. If on the other hand, your web presence has high traffic, mul­tivari­ate tests are the better option since you save time and effort.

A pre-requisite for both methods is that you have to have for­mu­lated clear hy­po­theses and result quant­it­ies for the tested elements. Otherwise, the test results will be difficult to interpret. In contrast to the usability analysis during the de­vel­op­ment process, you must also expect the con­ver­sion rate to de­teri­or­ate tem­por­ar­ily during this live test. So when it comes to testing out new ideas or variants, mul­tivari­ate tests are neither efficient nor adequate, and A/B testing is often too limited. Pre­lim­in­ary ex­per­i­ments on a smaller scale and with clearly for­mu­lated questions are a much more effective and less risky solution.

If your web project has enough traffic and you decide to perform mul­tivari­ate testing, you shouldn’t see this as a carte blanche to create as many variables or versions as possible. For an optimal result, it is advisable to proceed as stra­tegic­ally as possible and only have the main opponents compete against each other by pre-selecting which ones these are. In order to play it safe, you can even check the test result with a sub­sequent A/B test. However, you should be aware that using one of these testing methods doesn’t guarantee an increase in the con­ver­sion rate, but merely gives you ideas for what could increase it.