1
$\begingroup$

I have a dataset comprising 1000 integers. It represented as follows:

1235, 1284, 1258, 1284, 1162, 1286, 1333, 1235, 1286, 1315, 1240, 1488, 1346, 1188, 
1201, 1188, 1152, 1260, 1240, 1346, 1240, 1343, 1196, 1359, 1219, 1266, ...

The histogram doesn't appear to be normally distributed.

enter image description here

A Shapiro-Wilk test (scipy.stats.shapiro) returned 0 p-value.

data = [1235, 1284, 1258, ...]
scipy.stats.shapiro(data)

The dataset has a positive skewness of 0.9083 with a long tail in the right side. The histogram is similar to a Poisson distribution. So I ran the Kolmogorov-Smirnov test (scipy.stats.kstest), which also returned negative (zero p-value):

loc = min(data)
mu = np.mean(data)-loc
scipy.stats.kstest(data,"poisson",args=(mu,loc),alternative="greater",N=1000)

What are the correct values for mu and loc in this kstest? I used the minimum value for loc and "mean - loc" for mu. In this case, I used the following values

mu=168.349 loc=1098

I think mu value 168.349 is too large for a Poisson distribution. What is the correct way to test if the dataset is Poisson distribution or not?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Arbitrarily looking for a location-shifted Poisson distribution is not going to work and you should not do it unless you think there is a good reason to do so from the science of whatever you are looking at. $\endgroup$
    – Henry
    Commented Aug 24, 2023 at 21:51
  • 2
    $\begingroup$ Here it is clear a test against any Poisson is going to fail wherever you start the location, given how high the variance of your data is. A Poisson distribution with that variance is going to have minimal skewness, unlike your graph. Similarly a normal distribution has no skewness, so that test will fail too. Is there any reason to think you data might follow a named distribution? Real world data usually does not and then fails all such tests if you have enough data $\endgroup$
    – Henry
    Commented Aug 24, 2023 at 21:55
  • 2
    $\begingroup$ The KS test is not applicable when there are so many tied values, nor does it directly apply when the distributional parameters are estimated from the data. Why are you even performing distributional testing? What is the intention? $\endgroup$
    – whuber ♦
    Commented Aug 24, 2023 at 22:25
  • $\begingroup$ The only reason I thought my data might follow a named Poisson is because of the shape of the distribution graph. There isn't any other scientific rationale behind it. I just wanted to represent the data distribution with a small number of parameters... $\endgroup$
    – Jihyun
    Commented Aug 24, 2023 at 23:42
  • $\begingroup$ What is the scientific background of the dats? Are these integres counts, or discretized measurements of a continuous variable? And do you have an explanation for the ties that whuber mentions? They make us doubt that these are i.i.d realizations from some distribution $\endgroup$
    – Ute
    Commented Aug 25, 2023 at 2:18

1 Answer 1

Reset to default
2
$\begingroup$

Based on the post and the comments, it looks like you are just trying to find the distribution to best describe your data. And it doesn’t have to be Poisson distribution as you mentioned in the comment.

Instead of guessing the parameters, I’d highly recommend you a Python package called Fitter. And you can find a detailed tutorial in the link below how to identify the best fitted distribution

Improve this answer
$\endgroup$
1
  • $\begingroup$ Following the method you suggested, I found that the log-normal distribution was the best fit. Then applying a log() to the data, it resembled a normal distribution but KS test failed on it. $\endgroup$
    – Jihyun
    Commented Oct 2, 2023 at 21:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.