I have numerical vectors of data that represent number of emails per day send by user (one variable). I want to run a simulation where I will be able to simulate mail creation process for about 30 users, and so I need a distribution to generate data sets. I analysed data and found that this data follows a negative binomial distribution (I used the fitdist R package to find parameters mu and size). I want to verify if the observed data and data generated by this model follow the same distribution.

  1. Should I use a chi square test goodness of fit test to compare this two datasets? Will my null hypothesis be that data follow the same distribution, so if I get that p-value is P<0.05 then I confirm that H0 is correct.

  2. When I perform the chi square test, the first parameter is a vector of data, and the second parameter is a probability vector. How can I produce this probability vector, or can I pass just two vectors of data (empirical set and generated set of data) for comparison ?

Example in R:

N1 <- rpois(500, lambda = 4)

> table(N1)
  0   1   2   3   4   5   6   7   8   9  10  11 
  9  37  71 113  87  87  46  26  16   5   2   1

I transformed this frequency table in order to group freq. higher than 9 into one bin (since there is smallest number of occurences, and to avoid chi square test error)

> nf<-c(9,37,71,113,87,87,46,26,16,8)

> of<-dpois(0:9, lambda = 4)   / generated expected freq for poisson with lambda(mean) 4

> of
> [1] 0.01831564 0.07326256 0.14652511 0.19536681 0.19536681 0.15629345 0.10419563 0.05954036 0.02977018
[10] 0.01323119 

(I used Excel to find this probability of freq correct :)), then I applied a chi square test with rescale parameter to scale freq possibility sume to 1

> chisq.test(nf, p = of, rescale.p = TRUE)

    Chi-squared test for given probabilities

data:  nf
X-squared = 6.1301, df = 9, p-value = 0.7268

1 Answer 1

  1. Yes, you can use the goodness-of-fit test to test whether the actual distribution is different from the simulated distribution. It is a good idea to try this with totally made-up data first, and confirm to yourself that your implementation of the test actually works. (Note that a p<.05 in this case means that your distributions are significantly different, and then you would REJECT H0. Since we live in a NHST world, we never say that H0 is correct.)

  2. Yes, you can simply pass in the two vectors for comparison. Again, check and double check that the test is doing what you think it is doing. (Edit: to clarify, you need to use vectors of frequencies or probabilities for the goodness-of-fit test, not a vector of raw samples)

  • $\begingroup$ I creted two data vectors using a <- rpois(15, lambda = 3.5) and b <- rpois(15, lambda = 3.5) /first tryed with poisson latter will try with negative binomial. But when I do chisq.test(a,b) result for p = 0.1225 and this means that both vectors data are from same distribution since p is higher then 0.05 ... correct? Then I done another test using expected frequency chisq.test(a, p = b/sum(b)) and p = 2.951 e-06 wich is real small number. How can i interpret this results I know that this vector follow same distribution but this different p values confuse me $\endgroup$
    – explorer
    Oct 11, 2020 at 9:16
  • $\begingroup$ @explorer There a few problems here. First, you need vectors of probabilities or frequencies (see my edit above), not raw samples from rpois(). Second, the goodness-of-fit test will not work if your frequencies are around 5 or below. The Chi estimates will not be accurate below that point. Third, if the test statistic is significant, that means you should reject the null hypothesis (so, conclude that the distributions are likely different). If the test statistic is not significant, that means you fail to reject the null hypothesis (so, the conclude the distributions are likely the same). $\endgroup$
    – KamRa
    Oct 11, 2020 at 22:30
  • $\begingroup$ I know that there should be frequencies but when I do this, then I have different vector length for example b 1 2 3 4 5 6 with freq 1 2 4 5 2 1 and a 1 2 3 4 5 with freq 1 2 4 5 3 ... as you can see one freq heaving 5 members and other one more. If I count freq. then probability in one vector will also have one member more then other one. Should I define bins ?!? and thx for your effort $\endgroup$
    – explorer
    Oct 12, 2020 at 7:36
  • $\begingroup$ I just add code that result in P value that is equal to 0.7268, this means that my distributions are not "significantly different" and can accept H0 that say that my dataset follow poisson distribution. Same test I performed with negative binomial distribution where I used dnegbin function in R to generate expected frequency and a rnbinom to generate dataset that follow negative binomial distribution and again p value was higher then 0.05. Thx @KamRa for valuable comment, and I hope that this will help others to understend chisq.test goodnes of fit in R. $\endgroup$
    – explorer
    Oct 13, 2020 at 19:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.