1
$\begingroup$

I have a data set of about 260,000 observations of 50 variables. Although I highly suspect it's multivariate distribution to be non-normal, I still need a proof of it. I tried QuantPsyc package and it's mult.norm function but got a warning "Error: cannot allocate vector of size 794.5 Gb". I guess my data is too large for such tests. How can I test my data for multivariate normality or rather its absence?

$\endgroup$
4
  • $\begingroup$ Why do you have to "prove" the lack of normality? $\endgroup$
    – Dave
    Commented Mar 24, 2021 at 15:15
  • $\begingroup$ @Dave in my thesis things like that could be questioned $\endgroup$
    – user000
    Commented Mar 24, 2021 at 15:17
  • $\begingroup$ I am willing to wager literally any amount that unless your data were generated by a computer algorithm designed to produce multivariate normal values, your data are not consistent with a multivariate Normal data generating process. When you have such a large number of observations you should not be concerned with such questions: there are more relevant and important ones to address. $\endgroup$
    – whuber
    Commented Mar 24, 2021 at 18:06
  • $\begingroup$ This is very interesting. I have the exact same problem, asked in a review how I know that my data follow normality, when they are hundreds of thousands with a small feature space (5). I tried MVN in R and it needs allocation of around 400GB, that is how I ended up here. It was my initial thought that my data are too many, they most probably follow normality to some extent, but how to I put it? What does "When you have such a large number of observations you should not be concerned with such questions: there are more relevant and important ones to address" mean? $\endgroup$
    – ODstuck
    Commented Apr 3, 2021 at 22:35

1 Answer 1

3
$\begingroup$

You can reduce the problem of refuting multivariate normality to the one-dimensional case. Just use the fact that a random vector $X\in\mathbb{R}^n$ is multivariate normal, if and only if $a^T X\in\mathbb{R}$ is a normal random variable for every vector $a\in\mathbb{R}^n$ (see first bullet of this section.).

Start with the margins, i.e. apply a standard univariate normality test to each $X_i=e_i ^TX.$ If all margins pass your test, either choose an $a$ according to some prior knowledge you have, or sample randomly uniform on the sphere.

If you want a proper test, i.e. with confidence level and such, you would need to account for multiple testing. And all caveats of hypothesis testing apply of course!

But I think actually much harder than performing tests, is thinking about what it is you like to achieve by those tests. Your data is not normal, this is clear, because it is discrete. Even more pertinent: If your data is real world data, even the underlying generating distribution will never be normal. It may be very close to normal. So how close to or far away from normal do you need for your application? And in what sense? There are many ways in which your data may deviate.

$\endgroup$
3
  • $\begingroup$ Re "Your data is not normal, this is clear, because it is discrete:" doesn't this mistakenly confuse data with the data generation hypothesis to be tested? $\endgroup$
    – whuber
    Commented Mar 24, 2021 at 18:07
  • $\begingroup$ Well, I wanted to point out exactly this difference. In think this difference is the basic reason why it is a very courageous endeavour to test a sample in the hope to decide about the underlying distribution. $\endgroup$
    – g g
    Commented Mar 24, 2021 at 18:37
  • $\begingroup$ Thank you @gg. I was originally planning to trim my data from multivariate outliers using robust Mahalanobis distance method, which apparently assumed multivariate normality of the data. But I kept seeing people say that outliers can only be model-specific, and there is no point in searching them before starting to train a model. I thought I needed to remove outliers to conduct t-test feature filtering for my binary classification, but since it seems that only the Quantile Transformer Sccaler can help with that, while destroying linear relationships, I decided to opt for info gain filtering. $\endgroup$
    – user000
    Commented Mar 25, 2021 at 12:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.