1
$\begingroup$

I have a data frame with 7 columns that holds numerical and integer values where some columns, even though numerical, are binary values (e.g. a dummy variable for sex; $0=\text{male}$, $1=\text{female}$).

I was asked to check if my data frame is normally distributed and if not I have to normalize it. I found that there’s two ways to check: either by visualization, or by testing. However I tried both I didn’t get the outcome I want!

$\endgroup$
3
  • $\begingroup$ Welcome to CV. Since you’re new here, you may want to take our tour, which has information for new users. I am sure the answers below would be helpful, but you somehow answered the question in the title: you know how to check normality. However, the tests failed and visualization did not help. So, maybe you would like to edit the title to better reflect your problem. $\endgroup$ – T.E.G. Jul 1 '19 at 9:07
  • 3
    $\begingroup$ @Norah Binary variables cannot be normal. Integer variables cannot actually be normal. Variables that are necessarily either positive or non-negative cannot be normal. It would be pointless to try to check what cannot be true. Even continuous variates with infinite range are almost certainly not normal. This may be of little consequence - and trying to transform your variables to achieve normality is almost certainly misguided. What led you to believe you need any of your variables to be normally distributed? $\endgroup$ – Glen_b Jul 1 '19 at 10:35
  • 2
    $\begingroup$ A point of information: whoever asked you to check that your entire data frame was "normally distributed" doesn't know what they were asking, or perhaps you mis-heard them. You can check the distribution of the variables in your data frame (aka the columns). $\endgroup$ – Weiwen Ng Jul 1 '19 at 12:07
9
$\begingroup$

Welcome to CV!

There are several issues with your suggested approach:

  • Contrary to what the name suggests, normalization will not turn an arbitrarily distributed variable into a normally distributed one.
  • Neither can normality testing tell you that your data are normally distributed (only whether there is a significant deviation from normality).
  • Finally, data need rarely be normally distributed. It is also unlikely any of your data truly are normally distributed in the first place. You mentioned an integer variable, this can't be exactly normal, because the normal distribution is continuous, from $-\infty$ to $+\infty$. The same goes for the binary variable. Rather, it is common for models to assume the conditional distribution of the outcome variable to be approximately normally distributed.

As to what approach is best, you may want to have a look here for starters.

$\endgroup$
3
$\begingroup$

Adding to Frans' excellent answer:

A data frame cannot be normal, variables can be. All the variables in a data frame might be multivariate normal, but that's probably not what you meant.

The variables in your data frame cannot be normally distributed. Binary variables (as Glen pointed out) cannot be normal. Integers can't be exactly normal although, with a large enough range, they can come close (e.g. IQ is roughly normally distributed, even though scores are all positive integers). No transformation can make a binary variable even close to normal.

"I didn't get the outcome I want" is really a dangerous attitude in statistics.

Whoever asked you to do this either a) Didn't know anything about your data b) Didn't know anything about statistics or c) Was testing you to see if you would try to do something as silly as this.

$\endgroup$
-1
$\begingroup$

In R, the two most popular functions for testing normalization are the shapiro.test and the lillie.test (package nortest). They've been long discussed in this forum. Keep in mind, however, that those tests will only tell you if your distribution is normal or not: if you don't get the outcome you want maybe simply because your distribution is not normal.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.