Skip to main content

Methods to check if my data fit withfits a distribution function?

My shortenshortened data is:

y <- c (2,2,1,5,6,7,1,2,1,6,6,7,3,2,4,4,4,4,3,3,9,1,1,9)

I firstly normalize my data:

y_scale <- scale(y)

I thenThen, generalI generate a model data setdataset with normal distribution basebased on y_scale's mean and stdev:

y_norm <- rnorm(n=24, m=mean(y_scale), sd=sd(y_scale))

To check if my data fitfits the normal distribution, I do

ks.test(y_scale,y_norm)

I found the resultsresult is as follows:

Two-sample Kolmogorov-Smirnov test

data:  y_scale and y_norm 
D = 0.2083, p-value = 0.6749
alternative hypothesis: two-sided 

Warning message:
In ks.test(y_scale, y_norm) : cannot compute correct p-values with ties

Here, my question is:

(1) My real data set has ~ 700,000 numbers, I found I cannot use shapiro.test.

shapiro.test(y_scale)
Error in shapiro.test(y_scale) : sample size must be between 3 and 5000

(2) Is the p-value calculated above by ks.test as illustrated above is wrong? howHow to solve this problem of p-values?

Warning message:
In ks.test(y_scale, y_norm) : cannot compute correct p-values with ties

(3) The reasons why I tried to use ks.test instead of other methods, is because I want to compare with other model dataset withdatasets that have other distribution functions. It seems to me that I can simply replace y_norm with other model dataset, and compare their p-values or D-values (the smaller is the better), to choose which distribution function fitfits my data most?.

(4) Is it a must to normalize my data first?

Methods to check if my data fit with distribution function?

My shorten data is:

y <- c (2,2,1,5,6,7,1,2,1,6,6,7,3,2,4,4,4,4,3,3,9,1,1,9)

I firstly normalize my data:

y_scale <- scale(y)

I then, general a model data set with normal distribution base on y_scale's mean and stdev:

y_norm <- rnorm(n=24, m=mean(y_scale), sd=sd(y_scale))

To check if my data fit the normal distribution, I do

ks.test(y_scale,y_norm)

I found the results is as follows:

Two-sample Kolmogorov-Smirnov test

data:  y_scale and y_norm 
D = 0.2083, p-value = 0.6749
alternative hypothesis: two-sided 

Warning message:
In ks.test(y_scale, y_norm) : cannot compute correct p-values with ties

Here, my question is:

(1) My real data set has ~ 700,000 numbers, I found I cannot use shapiro.test.

shapiro.test(y_scale)
Error in shapiro.test(y_scale) : sample size must be between 3 and 5000

(2) Is the p-value calculated above by ks.test as illustrated above is wrong? how to solve this problem of p-values?

Warning message:
In ks.test(y_scale, y_norm) : cannot compute correct p-values with ties

(3) The reasons why I tried to use ks.test instead of other methods, is because I want to compare with other model dataset with other distribution functions. It seems to me that I can simply replace y_norm with other model dataset, and compare their p-values or D-values (the smaller is the better), to choose which distribution function fit my data most?

(4) Is it a must to normalize my data first?

Methods to check if my data fits a distribution function?

My shortened data is:

y <- c (2,2,1,5,6,7,1,2,1,6,6,7,3,2,4,4,4,4,3,3,9,1,1,9)

I firstly normalize my data:

y_scale <- scale(y)

Then, I generate a model dataset with normal distribution based on y_scale's mean and stdev:

y_norm <- rnorm(n=24, m=mean(y_scale), sd=sd(y_scale))

To check if my data fits the normal distribution, I do

ks.test(y_scale,y_norm)

I found the result is as follows:

Two-sample Kolmogorov-Smirnov test

data:  y_scale and y_norm 
D = 0.2083, p-value = 0.6749
alternative hypothesis: two-sided 

Warning message:
In ks.test(y_scale, y_norm) : cannot compute correct p-values with ties

Here, my question is:

(1) My real data set has ~ 700,000 numbers, I found I cannot use shapiro.test.

shapiro.test(y_scale)
Error in shapiro.test(y_scale) : sample size must be between 3 and 5000

(2) Is the p-value calculated above by ks.test wrong? How to solve this problem of p-values?

Warning message:
In ks.test(y_scale, y_norm) : cannot compute correct p-values with ties

(3) The reasons why I tried to use ks.test instead of other methods, is because I want to compare with other model datasets that have other distribution functions. It seems to me that I can simply replace y_norm with other model dataset, and compare their p-values or D-values (the smaller the better), to choose which distribution function fits my data most.

(4) Is it a must to normalize my data first?

Source Link
evdstat
  • 201
  • 1
  • 4
  • 6

Methods to check if my data fit with distribution function?

My shorten data is:

y <- c (2,2,1,5,6,7,1,2,1,6,6,7,3,2,4,4,4,4,3,3,9,1,1,9)

I firstly normalize my data:

y_scale <- scale(y)

I then, general a model data set with normal distribution base on y_scale's mean and stdev:

y_norm <- rnorm(n=24, m=mean(y_scale), sd=sd(y_scale))

To check if my data fit the normal distribution, I do

ks.test(y_scale,y_norm)

I found the results is as follows:

Two-sample Kolmogorov-Smirnov test

data:  y_scale and y_norm 
D = 0.2083, p-value = 0.6749
alternative hypothesis: two-sided 

Warning message:
In ks.test(y_scale, y_norm) : cannot compute correct p-values with ties

Here, my question is:

(1) My real data set has ~ 700,000 numbers, I found I cannot use shapiro.test.

shapiro.test(y_scale)
Error in shapiro.test(y_scale) : sample size must be between 3 and 5000

(2) Is the p-value calculated above by ks.test as illustrated above is wrong? how to solve this problem of p-values?

Warning message:
In ks.test(y_scale, y_norm) : cannot compute correct p-values with ties

(3) The reasons why I tried to use ks.test instead of other methods, is because I want to compare with other model dataset with other distribution functions. It seems to me that I can simply replace y_norm with other model dataset, and compare their p-values or D-values (the smaller is the better), to choose which distribution function fit my data most?

(4) Is it a must to normalize my data first?