Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

First of all I'd like to apologize for the vague title, I couldn't really formulate a better one just now, please feel free to change, or advice me to change, the title to make it better fit the core of the question.

Now about the question itself, I have been working on a software in which I have come across the idea of using an empirical distribution for sampling, however now that it's implemented I am not sure how to interpret it all. Allow me to describe what I have done, and why:

I have a bunch of calculations for a set of objects, yielding a final score. The score as it is however is very ad-hoc. So in order to make some sense out of the score of a particular object, what I do is to do a large number of (N = 1000) calculations of scores with mock/randomly generated values, yielding 1000 mock scores. Estimating an empirical "score distribution" for that particular object is then achieved by these 1000 mock score values.

I have implemented this in Java (as the rest of the software is also written in Java environment) using Apache Commons Math library, in particular the EmpiricalDistImpl class. According to the documentation this class uses:

what amounts to the Variable Kernel Method with Gaussian smoothing: Digesting the input file

  1. Pass the file once to compute min and max.
  2. Divide the range from min-max into binCount "bins."
  3. Pass the data file again, computing bin counts and univariate statistics (mean, std dev.) for each of the bins
  4. Divide the interval (0,1) into subintervals associated with the bins, with the length of a bin's subinterval proportional to its count.

Now my question is, does it make sense to sample from this distribution in order to calculate some sort of an expected value? In other words how much could I trust/rely on this distribution? Could I for instance draw conclusion about significance of observing a score $S$ by checking the distribution?

I realize that this is perhaps an unorthodox way looking at a problem like this but I think it would be interesting to get a better grip over the concept of empirical distributions, and how they can/cant be used in analysis.

Thanks in advance,

share|improve this question
    
If I understood you correctly, your final distribution is basically as good as your "mock/randomly generated values" for the objects. So - do you think you've sampled the distribution of your "objects" well? –  AVB Mar 20 '11 at 19:06
    
@AVB: not so much the distribution of the object but the distribution of the scores for one particular object. I guess my goal is to: A) Make sure the score calculation is sound and not-biased towards different properties of the objects in question; and B) To be able to say something about the significance of the score calculated from real experimental data –  posdef Mar 21 '11 at 8:40
    
You might consider taking more samples from your objects. –  John Salvatier Mar 21 '11 at 17:43
    
@John: you mean increasing the number from 1000 to, say 10K? I was thinking about that as well, I wasn't sure how well it would pay off though, considering the computation time. Any ideas on that? –  posdef Mar 22 '11 at 13:41
    
I don't know of an easy way to quantify the difference. It might be instructive to look at a bunch of histograms or Kernel Densities (with your eyes) with different numbers of points. –  John Salvatier Mar 22 '11 at 15:06
add comment

1 Answer 1

up vote 2 down vote accepted

Empirical distributions are used all the time for inference so you're definitely on the right track! One of the most common use of empirical distributions is for bootstrapping. In fact, you don't even have to use any of the machinery you've described above. In an nutshell, you make many draws (with replacement) from the original samples in a uniform fashion and the results can be used to calculate the confidence intervals on your previously calculated statistical quantities. Furthermore, these samples have well developed theoretical convergence properties. Check out the wikipedia article on the topic here.

share|improve this answer
    
Thanks for your answer, Gary. I am familiar with bootstrapping methods, however I am not sure how (read: where in the analysis) do you suggest I implement bootstrapping. Which original samples are we talking about? –  posdef Mar 21 '11 at 8:42
    
Original samples refer to the $N=1000$ samples you used to construct your empirical distribution. You mentioned using this distribution to compute statistics; let's say for simplicity the mean. The way to use the bootstrap would be to compute a mean with the initial $N$ points, let's call it $\hat{\mu}_N$. Now we take $M$ bootstrap samples and compute $M$ estimates for the mean, $\{\hat{\mu}_i^*\}_{i=1}^M$. You can then order this set and the order statistics will give you confidence intervals on $\hat{\mu}_N$ –  Gary Mar 21 '11 at 10:15
    
Thanks again for the explanation. I am not sure if this is useful however, when I try to estimate $P(x > x_{obs})$ where $x \in \hat{F}_{1000}(x)$. If you mean that I can go from the estimate $\hat{\mu}_{N}$ to a probability calculation, then I certainly interested.. –  posdef Mar 22 '11 at 13:49
    
I guess I'm a little confused by what you're seeking so I'll do my best to use the example you've posed. For a fixed value, let's say $y$, we wish to estimate $F(y)$. Then we take the $N=1000$ samples and get some value $\hat{q}$. Now take $M$ replicants of the original sample and you repeat the same computation yielding $M$ values of $\hat{q}^*$, giving a CI on $\hat{q}$. The key point is that the original sample can get you the statistic and the bootstrap gives the CI. Perhaps the real question is what you want to do with the empirical distribution. –  Gary Mar 23 '11 at 1:33
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.