# What is the definition of “multivariate testing”

I have seen job adverts that require experience in "multivariate testing". Although I have taken studied a bit statistics, I cant say for sure what this is.

The almighty Wikipedia provides a vague answer, first results on google talk about multivariate testing when it comes to websites, and when I search litterature on Amazon for multivariate testing, there is no "Multivariate testing for dummies" and other results seem irrelevant.

What would you say multivariate testing is (more specific in terms of statistics), and if you work with multivariate testing, what is it you do?

• I would have thought it was something A/B testing, but looking at the impact of combinations of alternatives: e.g. varying the button and the text and the colour with all possible combinations considered – Henry Aug 9 '17 at 18:12
• To a statistician, multivariate usually means having several dependent variables, so things like Hotelling's T^2 test and MANOVA. But this might not be what they mean in job ads. – Flounderer Aug 9 '17 at 18:16
• @Flounderer In the context of "click rate optimization", multivariate testing really just means changing multiple factors on a web page (usually resulting in $p > 2$ new pages) and then comparing those the proportion (or percentage) of clicks that each page gets. In this context, you would have multiple hypothesis tests with 1 variable of interest, "clicks". – Jon Aug 12 '17 at 17:29
• @Jon Thanks for the info! I have upvoted your answer. – Flounderer Aug 14 '17 at 17:21
• google.ch/search?q=multivariate+split+testing Is this your context? Otherwise provide some more context. Since, 'multivariate testing', is very broad and may have more specialized definitions in certain fields (like multivariate testing in relation to websites). You "blame" Wikipedia for being vague, but you answers will only be as broad as your questions. – Sextus Empiricus Aug 16 '17 at 17:18

From someone who just looking on the job market (recently hired), I can assure you that the job poster probably doesn't know what "multivariate testing", but if you want an answer to that, think of multiple t-tests done at once. This requires some correction of the p-value (e.g. Bonferroni). Multivariate testing is a methodology of conducting multiple simultaneous hypothesis tests regarding an outcome variable.

As an example, consider the following example in R. If we'd like if engine cylinder types have statistically significant differences in MPG (miles per gallon) we'd conduct an ANOVA test.

> data("mtcars")
>
> mod1 = lm(formula = mpg ~ as.factor(cyl), data = mtcars)
>
> summary(mod1)

Call:
lm(formula = mpg ~ as.factor(cyl), data = mtcars)

Residuals:
Min      1Q  Median      3Q     Max
-5.2636 -1.8357  0.0286  1.3893  7.2364

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)      26.6636     0.9718  27.437  < 2e-16 ***
as.factor(cyl)6  -6.9208     1.5583  -4.441 0.000119 ***
as.factor(cyl)8 -11.5636     1.2986  -8.905 8.57e-10 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 3.223 on 29 degrees of freedom
Multiple R-squared:  0.7325,    Adjusted R-squared:  0.714
F-statistic:  39.7 on 2 and 29 DF,  p-value: 4.979e-09

>
> anova(mod1)
Analysis of Variance Table

Response: mpg
Df Sum Sq Mean Sq F value    Pr(>F)
as.factor(cyl)  2 824.78  412.39  39.697 4.979e-09 ***
Residuals      29 301.26   10.39
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
>


The ANOVA example above uses cylynder 4 as the reference, so we are testing cylinder means against $\mu_4$. By simply looking at your $F$-statistic and associated p-value, you can reject (or fail to accept) the null hypothesis that $\mu_{4} = \mu_{6} = \mu_{8} = \mu$, and state that at least 1 of these cylynder types has a difference in means from $\mu_4$.

However, if we'd like to know which if cylinder types 6 and 8 are different from each other, we could run a pairwise-t-test as part of the post-hoc ANOVA analysis. Since we have 3 cylinder types, and we're testing for all 3, we'd need to make a correction for the simultaneous hypothesis testing to handle the increase in the false discovery rate (FDR). You can follow this reference for more information why is multiple testing a problem?

> pairwise.t.test(x = mtcars$mpg, g = mtcars$cyl, alternative = "two.sided")

Pairwise comparisons using t tests with pooled SD

data:  mtcars$mpg and mtcars$cyl

4       6
6 0.00024 -
8 2.6e-09 0.00415

P value adjustment method: holm


By default, this pairwise t-test uses a holm correction, which is not too dissimilar from a Bonferroni correction. Now, what this test tells us is that between all combinations of cylinder types, we see a difference at each combination of comparisons. So in effect, $\mu_{4} \neq \mu_{6} \neq \mu_{8}$

For another example, you can read this article, Adjusting for multiple testing when reporting research results: the Bonferroni vs Holm methods, a fairly short article that walks through the difference between the two types (Holm v Bonferroni) of hypothesis testing corrections.

I hope this brief example answers your questions, and you are able to read through the references I've shared.

Best of luck in your job hunt!

## EDIT

### Multivariate statistics v "Multivariate testing"

I wasn't sure if this issue needed to be addressed further until Martijn Weterings' comments. My argument is that the use of multivariate testing is not directly related to multivariate statistics. I'll explain.

First, let's look at OP's initial question,

I have seen job adverts that require experience in "multivariate testing". Although I have taken studied a bit statistics, I cant say for sure what this is.

The name multivariate testing comes up in a job board search, not a text book. This is important because there can be differences between definitions between academia and the outside world. Unfortunately, OP did not direct readers to a specific job posting, so I did a little job hunting on indeed. Here is an example of just a job post with multivariate testing Web Analytics Engineer

Are you looking for an exciting new opportunity in the field of web analytics and conversion rate optimization? Do you have experience programming and enjoy solving interesting problems? If so, you might be the perfect person to join our Engineering team!

This role is one part optimizing clients’ websites through A/B and multivariate tests, and one part web tracking escalation. Strong programming skills, intense intellectual curiosity, and a self-motivated, proactive mindset are the necessary tools for success. In addition, our ideal candidate has a passion for data and an interest in helping our clients achieve ROI-driven marketing results.

Job postings like the one above seem to indicate that A/B testing and multivariate testing are related, so I delve in deeper. What is A/B testing? And how is multivariate testing (currently ambiguously defined) related?

Well, as my comment to OP's question states

@Flounderer In the context of "click rate optimization", multivariate testing really just means changing multiple factors on a web page (usually resulting in $p>2$ new pages) and then comparing those the proportion (or percentage) of clicks that each page gets. In this context, you would have multiple hypothesis tests with 1 variable of interest, "clicks".

So, don't trust my understanding? Well, let's look further and see how multivariate testing relates with topics like clickthrough rate. See this post, Multivariate testing

Multivariate testing is a tactic for testing several variables in an email to find the best possible combination. It is often used when marketers want to test different variables (such as copy, images, color, or font) within an email to find the ideal combination for the end goal (such as click through rate or open rate.) Unlike an A/B test where only one variable is tested, multiple variables are being tested to find the ideal combination.

EXAMPLE In an email, a multivariate test might look like this: One test email uses a template that’s clean and simple with a bold hero image and a single CTA, while the other test email uses a more complex template with two CTAs, moderate body text, and no hero image to see which is more effective at driving click through rate to the company website.

So, to summarise, the outcome variable (only 1) is "click through rate (CRT)", and there are many predictors with multiple levels which the analyst/statistician may be interested in comparing. Multivariate testing here thus relates with testing how multiple factors affect one outcome, CRT.

My conclusion: Multivariate testing $\neq$ multivariate statistics

Now, still can't agree with this answer? Well, I'd like to see someone else delve into the topic. If I am wrong, I am wrong. But I have not seen anyone else try to answer OP's question without directly assuming the job postings that OP saw strictly meant multivariate statistics.

Also, without further information from OP, I don't think we can fully say it's this or that, but I think I do provide a strong argument for a possible definition of multivariate testing in the context of OP's question.

My personal experience as it relates to this type of job posting: I've worked with online marketing firms that work with specifically with business to business online marketing. A/B testing and multivariate testing (we didn't use this term) were commonly used. I am familiar with the terminology, albeit, not an expert.

• Although I know some textbooks (for non-statisticians) that call ANOVA multivariate, the example you give is univariate statistics in my terms. I upvote it anyway, because it is still close enough for some HR staff. – Horst Grünbusch Aug 12 '17 at 15:07
• @HorstGrünbusch, multivariate testing does not imply multivariate dependent data. In multivariate testing you are strictly interested in multiple hypothesis tests. If you'd like a more advanced reference, please refer to Large-Scale Inference by Bradley Efron where he takes this issue through an Emprical Bayes perspective. – Jon Aug 12 '17 at 17:01
• You reference doesn't support that multivariate tests would be multiple tests. Instead, any textbook about "multivariate statistics" includes testing multivariate hypotheses, i.e. hypotheses about variables that may be dependent within the subjects. See e.g. Muirhead or Anderson. Multiple testing is rather omitted therein. – Horst Grünbusch Aug 14 '17 at 11:30
• However, in marketing, procedures that use multiple testing (in a multivariate context) are called multivariate testing. – Horst Grünbusch Aug 14 '17 at 11:36
• I would also call this a bad example of multivariate testing. This 'mpg ~ as.factor(cyl)' is a situation in which a single variable (the number of cylinders) has multiple levels. Yes you can do multi-testing, but do not call that multivariate testing. @MLEN, the wikipedia is not at all that vague en.wikipedia.org/wiki/Multivariate_statistics "Multivariate statistics is a subdivision of statistics encompassing the simultaneous observation and analysis of more than one outcome variable" – Sextus Empiricus Aug 16 '17 at 13:51

Multivariate statistics deals with vector valued observations that can have some more general stochastic dependence within each vector. In contrast, univariate statistics is about data where each value is independent from each other.

So in multivariate statistics, covariance matrices or copulas as models for dependence appear as an important property of the data. In univariate statistics, such words are not needed to describe the data, at most in the "internals" of some statistics. For example, the whole dataset of a large clinical study, where lots of parameters are measured on the probands, is multivariate data.

Multivariate testing is testing with multivariate data, e.g. testing about correlations and regression models, testing differences of whole mean vectors, and multiple testing in which component the mean vectors differ.

See the toc of some textbooks about multivariate statistics, e.g. Anderson or Muirhead.

• As I've gone back to provide context and background information for my definition of multivariate testing $\neq$ multivariate statistics. Can you do the same to support your answer? I would at least like to see 1 job posting where it uses multivariate testing to indicate multivariate statistics. Thanks! – Jon Aug 16 '17 at 16:52
• I should also mention that the definition of multivariate statistics is clear to readers of this post. However, I've not seen anywhere where a statistician has used the term multivariate testing. Maybe you can enlighten me on this? – Jon Aug 16 '17 at 16:54

Different statisticians use "multivariate statistics" to mean different things.

In a wider sense, "multivariate" refers to all procedures that analyse the relations between more than two variables. An example for this would be a two-way analysis of variance (two-way ANOVA), with two independent variables and one dependent variable.

In a more narrower sense, "multivariate" refers to all statistical procedures that examine more than one dependent variable. An example would be a multivariate analysis of variance (MANOVA), but not the aforementioned two-way ANOVA. This is the meaning that Wikipedia uses in the article on multivariate statistics, and it is also what I learned to think of as multivariate when I was taught statistics at university.

Multivariate statistics usually deals with complex, often layered real-world data and employs linear or mixed or generalized linear mixed models, principal component analysis, and so on.

Can you not deduce what your future employer expects from the job description, what they do (see the website), or simply call them and ask (the secretary)?