Skip to main content
added 338 characters in body
Source Link
user1275607
  • 113
  • 1
  • 1
  • 6

I have a ~1 million data points. Here is the link to file data.txt Each of them can take a value between 0 to 145. It's a discrete dataset. Below is the histogram of dataset. On x-axis is the count (0-145) and on y-axis is the density.

source of data: I have around 20 reference objects and 1 Million random object in the space. For each of these 1 million random objects i calculated Manhattan distance with respect to these 20 reference objects. However i only considered shortest distance among these 20 reference objects. So i have 1 million Manhattan distances (which you can find in the link to file given in post)

I tried to fit the Poisson and Negative binomial distributions to this data set using R. I found the fit resulting from the negative binomial distributions seems reasonable. Below is the fitted curve (in blue).

Final goal: Once i have fitted this distribution appropriately, i would like to considered this distribution as random distribution of distances. Next time when I calculate the distance (d) of the any object to these 20 reference objects, I should be able to know if the (d) is significant or just part of random distribution.

enter image description here

To evaluate the goodness of fit I calculated the chi squared test using R with the observed frequencies and probabilities I got from negative binomial fit. Although the blue curve nicely fit to distribution, P-value returning from the chi squared test is extremely low.

This put me in confusion a bit. I have two related questions:

  1. Is the choice of negative binomial distribution for this dataset appropriate?

  2. If the chi squared test P-value is so low, should I consider another distribution?

Below is the complete code I used:

# read the file containing count data
data <- read.csv("data.txt", header=FALSE)

# plot the histogram
hist(data[[1]], prob=TRUE, breaks=145)

# load library
library(fitdistrplus)

# fit the negative binomial distribution
fit <- fitdist(data[[1]], "nbinom")

# get the fitted densities. mu and size from fit.
fitD <- dnbinom(0:145, size=25.05688, mu=31.56127)

# add fitted line (blue) to histogram
lines(fitD, lwd="3", col="blue")

# Goodness of fit with the chi squared test  
# get the frequency table
t <- table(data[[1]])   

# convert to dataframe
df <- as.data.frame(t)

# get frequencies
observed_freq <- df$Freq

# perform the chi-squared test
chisq.test(observed_freq, p=fitD)

I have a ~1 million data points. Here is the link to file data.txt Each of them can take a value between 0 to 145. It's a discrete dataset. Below is the histogram of dataset. On x-axis is the count (0-145) and on y-axis is the density.

source of data: I have around 20 reference objects and 1 Million random object in the space. For each of these 1 million random objects i calculated Manhattan distance with respect to these 20 reference objects. However i only considered shortest distance among these 20 reference objects. So i have 1 million Manhattan distances (which you can find in the link to file given in post)

I tried to fit the Poisson and Negative binomial distributions to this data set using R. I found the fit resulting from the negative binomial distributions seems reasonable. Below is the fitted curve (in blue).

enter image description here

To evaluate the goodness of fit I calculated the chi squared test using R with the observed frequencies and probabilities I got from negative binomial fit. Although the blue curve nicely fit to distribution, P-value returning from the chi squared test is extremely low.

This put me in confusion a bit. I have two related questions:

  1. Is the choice of negative binomial distribution for this dataset appropriate?

  2. If the chi squared test P-value is so low, should I consider another distribution?

Below is the complete code I used:

# read the file containing count data
data <- read.csv("data.txt", header=FALSE)

# plot the histogram
hist(data[[1]], prob=TRUE, breaks=145)

# load library
library(fitdistrplus)

# fit the negative binomial distribution
fit <- fitdist(data[[1]], "nbinom")

# get the fitted densities. mu and size from fit.
fitD <- dnbinom(0:145, size=25.05688, mu=31.56127)

# add fitted line (blue) to histogram
lines(fitD, lwd="3", col="blue")

# Goodness of fit with the chi squared test  
# get the frequency table
t <- table(data[[1]])   

# convert to dataframe
df <- as.data.frame(t)

# get frequencies
observed_freq <- df$Freq

# perform the chi-squared test
chisq.test(observed_freq, p=fitD)

I have a ~1 million data points. Here is the link to file data.txt Each of them can take a value between 0 to 145. It's a discrete dataset. Below is the histogram of dataset. On x-axis is the count (0-145) and on y-axis is the density.

source of data: I have around 20 reference objects and 1 Million random object in the space. For each of these 1 million random objects i calculated Manhattan distance with respect to these 20 reference objects. However i only considered shortest distance among these 20 reference objects. So i have 1 million Manhattan distances (which you can find in the link to file given in post)

I tried to fit the Poisson and Negative binomial distributions to this data set using R. I found the fit resulting from the negative binomial distributions seems reasonable. Below is the fitted curve (in blue).

Final goal: Once i have fitted this distribution appropriately, i would like to considered this distribution as random distribution of distances. Next time when I calculate the distance (d) of the any object to these 20 reference objects, I should be able to know if the (d) is significant or just part of random distribution.

enter image description here

To evaluate the goodness of fit I calculated the chi squared test using R with the observed frequencies and probabilities I got from negative binomial fit. Although the blue curve nicely fit to distribution, P-value returning from the chi squared test is extremely low.

This put me in confusion a bit. I have two related questions:

  1. Is the choice of negative binomial distribution for this dataset appropriate?

  2. If the chi squared test P-value is so low, should I consider another distribution?

Below is the complete code I used:

# read the file containing count data
data <- read.csv("data.txt", header=FALSE)

# plot the histogram
hist(data[[1]], prob=TRUE, breaks=145)

# load library
library(fitdistrplus)

# fit the negative binomial distribution
fit <- fitdist(data[[1]], "nbinom")

# get the fitted densities. mu and size from fit.
fitD <- dnbinom(0:145, size=25.05688, mu=31.56127)

# add fitted line (blue) to histogram
lines(fitD, lwd="3", col="blue")

# Goodness of fit with the chi squared test  
# get the frequency table
t <- table(data[[1]])   

# convert to dataframe
df <- as.data.frame(t)

# get frequencies
observed_freq <- df$Freq

# perform the chi-squared test
chisq.test(observed_freq, p=fitD)
added 391 characters in body
Source Link
user1275607
  • 113
  • 1
  • 1
  • 6

I have a ~1 million data points. Here is the link to file data.txt Each of them can take a value between 0 to 145. It's a discrete dataset. Below is the histogram of dataset. On x-axis is the count (0-145) and on y-axis is the density.

source of data: I have around 20 reference objects and 1 Million random object in the space. For each of these 1 million random objects i calculated Manhattan distance with respect to these 20 reference objects. However i only considered shortest distance among these 20 reference objects. So i have 1 million Manhattan distances (which you can find in the link to file given in post)

I tried to fit the Poisson and Negative binomial distributions to this data set using R. I found the fit resulting from the negative binomial distributions seems reasonable. Below is the fitted curve (in blue).

enter image description here

To evaluate the goodness of fit I calculated the chi squared test using R with the observed frequencies and probabilities I got from negative binomial fit. Although the blue curve nicely fit to distribution, P-value returning from the chi squared test is extremely low.

This put me in confusion a bit. I have threetwo related questions:

  1. Is the choice of negative binomial distribution for this dataset appropriate?

  2. If the chi squared test P-value is so low, should I consider another distribution?

Below is the complete code I used:

# read the file containing count data
data <- read.csv("data.txt", header=FALSE)

# plot the histogram
hist(data[[1]], prob=TRUE, breaks=145)

# load library
library(fitdistrplus)

# fit the negative binomial distribution
fit <- fitdist(data[[1]], "nbinom")

# get the fitted densities. mu and size from fit.
fitD <- dnbinom(0:145, size=25.05688, mu=31.56127)

# add fitted line (blue) to histogram
lines(fitD, lwd="3", col="blue")

# Goodness of fit with the chi squared test  
# get the frequency table
t <- table(data[[1]])   

# convert to dataframe
df <- as.data.frame(t)

# get frequencies
observed_freq <- df$Freq

# perform the chi-squared test
chisq.test(observed_freq, p=fitD)

I have a ~1 million data points. Here is the link to file data.txt Each of them can take a value between 0 to 145. It's a discrete dataset. Below is the histogram of dataset. On x-axis is the count (0-145) and on y-axis is the density.

I tried to fit the Poisson and Negative binomial distributions to this data set using R. I found the fit resulting from the negative binomial distributions seems reasonable. Below is the fitted curve (in blue).

enter image description here

To evaluate the goodness of fit I calculated the chi squared test using R with the observed frequencies and probabilities I got from negative binomial fit. Although the blue curve nicely fit to distribution, P-value returning from the chi squared test is extremely low.

This put me in confusion a bit. I have three related questions:

  1. Is the choice of negative binomial distribution for this dataset appropriate?

  2. If the chi squared test P-value is so low, should I consider another distribution?

Below is the complete code I used:

# read the file containing count data
data <- read.csv("data.txt", header=FALSE)

# plot the histogram
hist(data[[1]], prob=TRUE, breaks=145)

# load library
library(fitdistrplus)

# fit the negative binomial distribution
fit <- fitdist(data[[1]], "nbinom")

# get the fitted densities. mu and size from fit.
fitD <- dnbinom(0:145, size=25.05688, mu=31.56127)

# add fitted line (blue) to histogram
lines(fitD, lwd="3", col="blue")

# Goodness of fit with the chi squared test  
# get the frequency table
t <- table(data[[1]])   

# convert to dataframe
df <- as.data.frame(t)

# get frequencies
observed_freq <- df$Freq

# perform the chi-squared test
chisq.test(observed_freq, p=fitD)

I have a ~1 million data points. Here is the link to file data.txt Each of them can take a value between 0 to 145. It's a discrete dataset. Below is the histogram of dataset. On x-axis is the count (0-145) and on y-axis is the density.

source of data: I have around 20 reference objects and 1 Million random object in the space. For each of these 1 million random objects i calculated Manhattan distance with respect to these 20 reference objects. However i only considered shortest distance among these 20 reference objects. So i have 1 million Manhattan distances (which you can find in the link to file given in post)

I tried to fit the Poisson and Negative binomial distributions to this data set using R. I found the fit resulting from the negative binomial distributions seems reasonable. Below is the fitted curve (in blue).

enter image description here

To evaluate the goodness of fit I calculated the chi squared test using R with the observed frequencies and probabilities I got from negative binomial fit. Although the blue curve nicely fit to distribution, P-value returning from the chi squared test is extremely low.

This put me in confusion a bit. I have two related questions:

  1. Is the choice of negative binomial distribution for this dataset appropriate?

  2. If the chi squared test P-value is so low, should I consider another distribution?

Below is the complete code I used:

# read the file containing count data
data <- read.csv("data.txt", header=FALSE)

# plot the histogram
hist(data[[1]], prob=TRUE, breaks=145)

# load library
library(fitdistrplus)

# fit the negative binomial distribution
fit <- fitdist(data[[1]], "nbinom")

# get the fitted densities. mu and size from fit.
fitD <- dnbinom(0:145, size=25.05688, mu=31.56127)

# add fitted line (blue) to histogram
lines(fitD, lwd="3", col="blue")

# Goodness of fit with the chi squared test  
# get the frequency table
t <- table(data[[1]])   

# convert to dataframe
df <- as.data.frame(t)

# get frequencies
observed_freq <- df$Freq

# perform the chi-squared test
chisq.test(observed_freq, p=fitD)
edited title; added tags; light editing & formatting
Source Link
gung - Reinstate Monica
  • 147.5k
  • 89
  • 406
  • 717

R: fitting Fitting negative binomial distribution to large count data

I have a ~1 million data points. Here is the link to file data.txt Each of them can take a value between 0 to 145. It's a discrete dataset. Below is the histogram of dataset. On x-axis is the count (0-145) and on y-axis is the density.

I tried to fit the Poisson and Negative binomial distributions to this data set using R. I found the fit resulting from the negative binomial distributions seems reasonable. Below is the fitted curve (in blue).   

enter image description here

To evaluate the goodness of fit I calculated the chi squared test using R with the observed frequencies and probabilities I got from negative binomial fit. Although the blue curve nicely fit to distribution, P-value returning from the chi squared test is extremely low.

This put me in confusion a bit. I have three related questions:

a) is the choice of negative binomial distribution for this dataset appropriate?

b) if the chi squared test P-Value is so low, should i considered the another distributions?

  1. Is the choice of negative binomial distribution for this dataset appropriate?

  2. If the chi squared test P-value is so low, should I consider another distribution?

Below is the complete code I used:

//# read the file containing count data
data<data <- read.csv("data.txt", header=FALSE)

//# plot the histogram
hist(data[[1]], prob=TRUE, breaks=145)

//# load library
library(fitdistrplus)

//# fit the negative binomial distribution
fit<fit <- fitdist(data[[1]], "nbinom")

//# get the fitted densities. mu and size from fit.
fitD<fitD <- dnbinom(0:145, size=25.05688, mu=31.56127)

//# add fitted line (blue) to histogram
lines(fitD, lwd="3", col="blue")

//# Goodness of fit with the chi squared test  
//# get the frequency table
t<t <- table(data[[1]])   

//# convert to dataframe
df<df <- as.data.frame(t)

//# get frequencies
observed_freq<observed_freq <- df$Freq

//# perform the chi-squared test
chisq.test(observed_freq, p=fitD)

R: fitting negative binomial distribution to large count data

I have a ~1 million data points. Here is the link to file data.txt Each of them can take a value between 0 to 145. It's a discrete dataset. Below is the histogram of dataset. On x-axis is the count (0-145) and on y-axis is the density.

I tried to fit the Poisson and Negative binomial distributions to this data set using R. I found the fit resulting from the negative binomial distributions seems reasonable. Below is the fitted curve (in blue).  enter image description here

To evaluate the goodness of fit I calculated the chi squared test using R with the observed frequencies and probabilities I got from negative binomial fit. Although the blue curve nicely fit to distribution, P-value returning from the chi squared test is extremely low.

This put me in confusion a bit. I have three related questions:

a) is the choice of negative binomial distribution for this dataset appropriate?

b) if the chi squared test P-Value is so low, should i considered the another distributions?

Below is the complete code I used:

//read the file containing count data
data<-read.csv("data.txt", header=FALSE)

//plot the histogram
hist(data[[1]], prob=TRUE, breaks=145)

//load library
library(fitdistrplus)

//fit the negative binomial distribution
fit<-fitdist(data[[1]], "nbinom")

//get the fitted densities. mu and size from fit.
fitD<-dnbinom(0:145, size=25.05688, mu=31.56127)

//add fitted line (blue) to histogram
lines(fitD, lwd="3", col="blue")

//Goodness of fit with the chi squared test  
//get the frequency table
t<-table(data[[1]])   

//convert to dataframe
df<-as.data.frame(t)

//get frequencies
observed_freq<-df$Freq

//perform the chi-squared test
chisq.test(observed_freq, p=fitD)

Fitting negative binomial distribution to large count data

I have a ~1 million data points. Here is the link to file data.txt Each of them can take a value between 0 to 145. It's a discrete dataset. Below is the histogram of dataset. On x-axis is the count (0-145) and on y-axis is the density.

I tried to fit the Poisson and Negative binomial distributions to this data set using R. I found the fit resulting from the negative binomial distributions seems reasonable. Below is the fitted curve (in blue). 

enter image description here

To evaluate the goodness of fit I calculated the chi squared test using R with the observed frequencies and probabilities I got from negative binomial fit. Although the blue curve nicely fit to distribution, P-value returning from the chi squared test is extremely low.

This put me in confusion a bit. I have three related questions:

  1. Is the choice of negative binomial distribution for this dataset appropriate?

  2. If the chi squared test P-value is so low, should I consider another distribution?

Below is the complete code I used:

# read the file containing count data
data <- read.csv("data.txt", header=FALSE)

# plot the histogram
hist(data[[1]], prob=TRUE, breaks=145)

# load library
library(fitdistrplus)

# fit the negative binomial distribution
fit <- fitdist(data[[1]], "nbinom")

# get the fitted densities. mu and size from fit.
fitD <- dnbinom(0:145, size=25.05688, mu=31.56127)

# add fitted line (blue) to histogram
lines(fitD, lwd="3", col="blue")

# Goodness of fit with the chi squared test  
# get the frequency table
t <- table(data[[1]])   

# convert to dataframe
df <- as.data.frame(t)

# get frequencies
observed_freq <- df$Freq

# perform the chi-squared test
chisq.test(observed_freq, p=fitD)
Source Link
user1275607
  • 113
  • 1
  • 1
  • 6
Loading