0
$\begingroup$

I am given a data set of 100k instances and I am being told that it belongs to one of four statistical distributions (normal, truncated normal, poisson and uniform). I am wondering how I may go about finding which stastical model best represents this dataset perhaps using numpy?

What I am thinking is to literally count occurrences of every instance and then divide over size giving the probability of each instance, and then attempting to find that probability using different statistical models such as P(X) = 1/b-a for uniform and see which one it best matches? I feel like this is very tedious and will not work.

Can anyone guide me in the right direction?

$\endgroup$
4
  • 2
    $\begingroup$ Why can you not visualize or plot? $\endgroup$ Feb 4, 2019 at 3:31
  • $\begingroup$ Maybe a $\chi^2$ goodness of fit test. $\endgroup$ Feb 4, 2019 at 7:50
  • 1
    $\begingroup$ It's not possible to distinguish a Normal distribution from a truncated Normal using any amount of data, because the truncation point could be extreme. (It is possible, however, to determine that the data may be truncated Normal but definitely are not Normal.) Given the basic and obvious qualitative differences among the Normal, Poisson, and Uniform families, even the most primitive plot or test with 100K values ought to perform well unless the Poisson parameter is huge. $\endgroup$
    – whuber
    Feb 4, 2019 at 14:37
  • $\begingroup$ Can you elaborate on the above point? What sort of "tests" can one use? I can't plot because it is a requirement for one of my assignments. $\endgroup$
    – skidjoe
    Feb 4, 2019 at 18:15

1 Answer 1

0
$\begingroup$

It is a strange assignment, but:

  • Poisson have mean=variance. Calculate both from data!
  • Uniform has mean midways between min and max, and a variance easily found from min and max
  • Normal has no relation between mean and variance, so if not passes first two ... besides normal is symmetric around mean, so just look at some quantiles symmetrically around the mean ...
  • Truncated normal (assuming truncation point is not very extreme, which it probably not is given this is an exercise is not symmetric ...

The relations above are exact for the true parameter values, but with 100k observations your empirical mean and variance ... will be very close.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.