# How to use/interpret empirical distribution?

First of all I'd like to apologize for the vague title, I couldn't really formulate a better one just now, please feel free to change, or advice me to change, the title to make it better fit the core of the question.

Now about the question itself, I have been working on a software in which I have come across the idea of using an empirical distribution for sampling, however now that it's implemented I am not sure how to interpret it all. Allow me to describe what I have done, and why:

I have a bunch of calculations for a set of objects, yielding a final score. The score as it is however is very ad-hoc. So in order to make some sense out of the score of a particular object, what I do is to do a large number of (N = 1000) calculations of scores with mock/randomly generated values, yielding 1000 mock scores. Estimating an empirical "score distribution" for that particular object is then achieved by these 1000 mock score values.

I have implemented this in Java (as the rest of the software is also written in Java environment) using Apache Commons Math library, in particular the EmpiricalDistImpl class. According to the documentation this class uses:

what amounts to the Variable Kernel Method with Gaussian smoothing: Digesting the input file

1. Pass the file once to compute min and max.
2. Divide the range from min-max into binCount "bins."
3. Pass the data file again, computing bin counts and univariate statistics (mean, std dev.) for each of the bins
4. Divide the interval (0,1) into subintervals associated with the bins, with the length of a bin's subinterval proportional to its count.

Now my question is, does it make sense to sample from this distribution in order to calculate some sort of an expected value? In other words how much could I trust/rely on this distribution? Could I for instance draw conclusion about significance of observing a score $S$ by checking the distribution?

I realize that this is perhaps an unorthodox way looking at a problem like this but I think it would be interesting to get a better grip over the concept of empirical distributions, and how they can/cant be used in analysis.

-
If I understood you correctly, your final distribution is basically as good as your "mock/randomly generated values" for the objects. So - do you think you've sampled the distribution of your "objects" well? –  AVB Mar 20 '11 at 19:06
@AVB: not so much the distribution of the object but the distribution of the scores for one particular object. I guess my goal is to: A) Make sure the score calculation is sound and not-biased towards different properties of the objects in question; and B) To be able to say something about the significance of the score calculated from real experimental data –  posdef Mar 21 '11 at 8:40
You might consider taking more samples from your objects. –  John Salvatier Mar 21 '11 at 17:43
@John: you mean increasing the number from 1000 to, say 10K? I was thinking about that as well, I wasn't sure how well it would pay off though, considering the computation time. Any ideas on that? –  posdef Mar 22 '11 at 13:41
I don't know of an easy way to quantify the difference. It might be instructive to look at a bunch of histograms or Kernel Densities (with your eyes) with different numbers of points. –  John Salvatier Mar 22 '11 at 15:06

Original samples refer to the $N=1000$ samples you used to construct your empirical distribution. You mentioned using this distribution to compute statistics; let's say for simplicity the mean. The way to use the bootstrap would be to compute a mean with the initial $N$ points, let's call it $\hat{\mu}_N$. Now we take $M$ bootstrap samples and compute $M$ estimates for the mean, $\{\hat{\mu}_i^*\}_{i=1}^M$. You can then order this set and the order statistics will give you confidence intervals on $\hat{\mu}_N$ –  Gary Mar 21 '11 at 10:15
Thanks again for the explanation. I am not sure if this is useful however, when I try to estimate $P(x > x_{obs})$ where $x \in \hat{F}_{1000}(x)$. If you mean that I can go from the estimate $\hat{\mu}_{N}$ to a probability calculation, then I certainly interested.. –  posdef Mar 22 '11 at 13:49
I guess I'm a little confused by what you're seeking so I'll do my best to use the example you've posed. For a fixed value, let's say $y$, we wish to estimate $F(y)$. Then we take the $N=1000$ samples and get some value $\hat{q}$. Now take $M$ replicants of the original sample and you repeat the same computation yielding $M$ values of $\hat{q}^*$, giving a CI on $\hat{q}$. The key point is that the original sample can get you the statistic and the bootstrap gives the CI. Perhaps the real question is what you want to do with the empirical distribution. –  Gary Mar 23 '11 at 1:33