How to perform a test using R to see if data follows normal distribution

Question

I have a data set with following structure:

a word | number of occurrence of a word in a document | a document id 

How can I perform a test for normal distribution in R? Probably it is an easy question but I am a R newbie.

Answer 1

If I understand your question correctly, then to test if word occurrences in a set of documents follows a Normal distribution you can just use a shapiro-Wilk test and some qqplots. For example,

## Generate two data sets
## First Normal, second from a t-distribution
words1 = rnorm(100); words2 = rt(100, df=3)

## Have a look at the densities
plot(density(words1));plot(density(words2))

## Perform the test
shapiro.test(words1); shapiro.test(words2)

## Plot using a qqplot
qqnorm(words1);qqline(words1, col = 2)
qqnorm(words2);qqline(words2, col = 2)

The qqplot commands give:

You can see that the second data set is clearly not Normal by the heavy tails (More Info).

In the Shapiro-Walk normality test, the p-value is large for the first data set (>.9) but very small for the second data set (<.01). This will lead you to reject the null hypothesis for the second.

Answer 2

Assuming your dataset is called words and has a counts column, you can plot the histogram to have a visualization of the distribution:

hist(words$counts, 100, col="black")

where 100 is the number of bins

You can also do a normal Q-Q plot using

qqnorm(words$counts)

Finally, you can also use the Shapiro-Wilk test for normality

shapiro.test(word$counts)

Although, look at this discussion: Normality Testing: 'Essentially Useless?'

Answer 3

No test will show you that your data has a normal distribution - it will only be able to show you when the data is sufficiently inconsistent with a normal that you would reject the null.

But counts are not normal in any case, they're positive integers - what's the probability that an observation from a normal distribution will take a value that isn't an integer? (... that's an event of probability 1).

Why would you test for normality in this case? It's obviously untrue.

[In some cases it may not necessarily matter that you can tell your data aren't actually normal. Real data are never (or almost never) going to be actually drawn from a normal distribution.]

If you really need to do a test, the Shapiro-Wilk test (?shapiro.test) is a good general test of normality, one that's widely used.

Answer 4

A more formal way of looking at the normality is by testing whether the kurtosis and skewness are significantly different from zero.

To do this, we need to get:

kurtosis.test <- function (x) {
m4 <- sum((x-mean(x))^4)/length(x)
s4 <- var(x)^2
kurt <- (m4/s4) - 3
sek <- sqrt(24/length(x))
totest <- kurt/sek
pvalue <- pt(totest,(length(x)-1))
pvalue 
}

for kurtosis, and:

skew.test <- function (x) {
m3 <- sum((x-mean(x))^3)/length(x)
s3 <- sqrt(var(x))^3
skew <- m3/s3
ses <- sqrt(6/length(x))
totest <- skew/ses
pt(totest,(length(x)-1))
pval <- pt(totest,(length(x)-1))
pval
}

for Skewness.

Both these tests are one-tailed, so you'll need to multiply the p-value by 2 to become two-tailed. If your p-value become larger than one you'll need to use 1-kurtosis.test() instead of kurtosis.test.

If you have any other questions you can email me at [email protected]

Answer 5

By using the nortest package of R, these tests can be conducted:

• Perform Anderson-Darling normality test

  ad.test(data1)
  

• Perform Cramér-von Mises test for normality

  cvm.test(data1)
  

• Perform Pearson chi-square test for normality

  pearson.test(data1)
  

• Perform Shapiro-Francia test for normality

  sf.test(data1)
  

Many other tests can be done by using the normtest package. See description at https://cran.r-project.org/web/packages/normtest/normtest.pdf

Answer 6

In addition to the Shapiro-Wilk test of the stats package, the nortest package (available on CRAN) provides other normality tests.