I've always wondered this. Let's say someone gives you a spreadsheet with a large set of numbers, say 10,000, and says "What distribution does this set of numbers follow (Gaussian, exponential, Poisson, etc.)"? How do you go about determining the answer?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Can you explain what you mean by 'ordered numbers'? In the context of the word 'distribution', some of the answers I can think of cause me some worry that my answer heads down the wrong path. $\endgroup$– Glen_bDec 3, 2013 at 17:54
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
I think you would formulate hypotheses about the distributions which you test on the data. The structure of data may give you hints on whether a discrete or a continuous distribution is at hand. A famous test is Kolmogorov Smirnov, which compares expected to observed cummulative distribution functions to test a hypothesized null distribution. See the wikipedia entry for this test for details:
http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test
To test discrete distributions, a Chi² test is usually applied.
-
$\begingroup$ Thanks .. it's kinda of the opposite of most data analysis where you start with a set of data knowing its distribution and build a model using the assumed distribution. Will have a look at your link. $\endgroup$– BobLDec 2, 2013 at 21:58
-
$\begingroup$ By the way, to test discrete distributions, a KS cannot be used. A chi-squared test is then an option. $\endgroup$– tomkaDec 2, 2013 at 22:01
-
1$\begingroup$ You can use a KS test for a completely specified discrete distribution if you simulate the distribution under the null. The bigger problem tends to be that the KS test is unsuitable when you estimate parameters, and that problem holds for the continuous case as well. e.g. you can't test 'normality' by a KS without estimating the parameters, in which case the tables don't apply. If you estimate parameters you have to simulate the distribution of the test statistic, leaving you with a Lilliefors' test. $\endgroup$– Glen_bDec 2, 2013 at 22:38
$\begingroup$
$\endgroup$
2
Let's say someone gives you a spreadsheet with a large set of numbers, say 10,000, and says "What distribution does this set of numbers follow (Gaussian, exponential, Poisson, etc.)"? How do you go about determining the answer?
If its real data, the actual answer is generally 'none of the above, nor anything else specific you can put a name to'.
You may be able to find one that is a reasonable description, but with a large enough sample size you could potentially reject every 'standard' distribution you try.
Note, particularly, that failure to reject some particular distributional guess with a hypothesis test doesn't imply that's the distribution you have -- nor does rejection imply that using the 'rejected' distribution as a model would yield results that aren't simultaneously reasonably accurate and quite informative.
With a sample size of 10000, I'd be saying "what do I need such an approximate model for? I have a sample of size ten thousand! I can estimate the distribution more accurately nonparametrically, whether by using the ECDF itself or by kernel methods or log-spline density estimation, or whatever"
You may find some useful insight in the discussion of these somewhat related questions here, here or here
-
$\begingroup$ This question is more of a thought experiment that's been in the back of my mind. Yes, with 10K observations you'd likely have enough to build a model rather than approximate one. $\endgroup$– BobLDec 3, 2013 at 15:01
-
$\begingroup$ BobL, I've added a few links to some other, somewhat related questions where you may find the discussion helpful $\endgroup$– Glen_bDec 4, 2013 at 21:22