1
$\begingroup$

I'm examining the performance of quadratic and linear discriminant models at classification.

My dataset has 250,000 observations, 2 groups and 30 explanatory variables.

I thought it would be worth considering whether or not my dataset displays multivariate normality, since that is one of the assumptions of discriminant function analysis.

The steps i took were:

1: created a data frame of just the 30 explanatory variables

2: converted the data frame into a numeric matrix (explanatoryVariables)

3: I used the mshapiro.test(explanatoryVariables) function and received the following error:

Error: cannot allocate vector of size 465.7 Gb

I have also tried

hzTest(explanatoryVariables)
mardiaTest(explanatoryVariables)
roystonTest(explanatoryVariables)

but also receive the same error.

Would this be due to the fact that I have too much data? Would there be any other way to show multivariate normality like reducing sample size, and would that make a significant impact in any way on the validity of the test?

Regards,

Marvin

$\endgroup$

1 Answer 1

1
$\begingroup$

With such a large ratio of samples to variables you could randomly sample them to use substantially fewer and get likely similar results.

You can also do quantile-quantile (QQ) plots of pairs of variables, multivariate normality also implies pairwise normality.

But I'm not sure you should take statistical normality tests very seriously (as opposed to plots which you should take seriously). Almost all real world data will be found to be non-normal given enough samples, unless generated by a really simple random process. So what impact would knowing that your data is approximately normal but not exactly normal have on your analysis? Likely not much, the main point is "does it make a difference?" and to answer that you have to empirically evaluate whether your models work well and whether some form of transformation would make them work better, judged by some criterion such as how well they predict in cross-validation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.