I have a data set with following structure:
a word | number of occurrence of a word in a document | a document id
How can I perform a test for normal distribution in R? Probably it is an easy question but I am a R newbie.
If I understand your question correctly, then to test if word occurrences in a set of documents follows a Normal distribution you can just use a shapiro-Wilk test and some qqplots. For example,
The qqplot commands give:
You can see that the second data set is clearly not Normal.
Assuming your dataset is called
where 100 is the number of bins
You can also do a normal Q-Q plot using
Finally, you can also use the Shapiro-Wilk test for normality
Although, look at this discussion: Normality Testing: 'Essentially Useless?'
No test will show you that your data has a normal distribution - it will only be able to show you when the data is sufficiently inconsistent with a normal that you would reject the null.
But counts are not normal in any case, they're positive integers - what's the probability that an observation from a normal distribution will take a value that isn't an integer?
Why would you test for normality in this case?
A more formal way of looking at the normality is by testing whether the kurtosis and skewness are significantly different from zero.
To do this, we need to get:
for kurtosis, and:
Both these tests are one-tailed, so you'll need to multiply the p-value by 2 to become two-tailed. If your p-value become larger than one you'll need to use 1-kurtosis.test() instead of kurtosis.test.
If you have any other questions you can email me at email@example.com
In addition to the Shapiro-Wilk test of the stats package, the nortest package (available on CRAN) provides other normality tests.