I have a ~1 million data points. Here is the link to file data.txt Each of them can take a value between 0 to 145. It's a discrete dataset. Below is the histogram of dataset. On x-axis is the count (0-145) and on y-axis is the density.
source of data: I have around 20 reference objects and 1 Million random object in the space. For each of these 1 million random objects i calculated Manhattan distance with respect to these 20 reference objects. However i only considered shortest distance among these 20 reference objects. So i have 1 million Manhattan distances (which you can find in the link to file given in post)
I tried to fit the Poisson and Negative binomial distributions to this data set using R. I found the fit resulting from the negative binomial distributions seems reasonable. Below is the fitted curve (in blue).
Final goal: Once i have fitted this distribution appropriately, i would like to considered this distribution as random distribution of distances. Next time when I calculate the distance (d) of the any object to these 20 reference objects, I should be able to know if the (d) is significant or just part of random distribution.
To evaluate the goodness of fit I calculated the chi squared test using R with the observed frequencies and probabilities I got from negative binomial fit. Although the blue curve nicely fit to distribution, P-value returning from the chi squared test is extremely low.
This put me in confusion a bit. I have two related questions:
Is the choice of negative binomial distribution for this dataset appropriate?
If the chi squared test P-value is so low, should I consider another distribution?
Below is the complete code I used:
# read the file containing count data data <- read.csv("data.txt", header=FALSE) # plot the histogram hist(data[], prob=TRUE, breaks=145) # load library library(fitdistrplus) # fit the negative binomial distribution fit <- fitdist(data[], "nbinom") # get the fitted densities. mu and size from fit. fitD <- dnbinom(0:145, size=25.05688, mu=31.56127) # add fitted line (blue) to histogram lines(fitD, lwd="3", col="blue") # Goodness of fit with the chi squared test # get the frequency table t <- table(data[]) # convert to dataframe df <- as.data.frame(t) # get frequencies observed_freq <- df$Freq # perform the chi-squared test chisq.test(observed_freq, p=fitD)