3
$\begingroup$

Given a set of data (~5000 values) I'd like to draw random samples from the same distribution as the original data. The problem is there is no way to know for sure what distribution the original data comes from.

It makes sense to use normal distribution in my case, although I'd like to be able to motivate that decision, and of course I also need to estimate the $(\mu,\sigma)$ pair.

Any idea on how to accomplish this, preferably within Java environment. I have been using Apache Commons Math and recently stumbled upon Colt library. I was hoping to get it done without bothering with MATLAB and R.

$\endgroup$
  • $\begingroup$ you should mention java environment in the title of your question this will maximize the chance to get an answer. Also is there a tag "java" ? $\endgroup$ – robin girard Feb 9 '11 at 17:09
  • $\begingroup$ @robin Yes it is. $\endgroup$ – user88 Feb 9 '11 at 17:14
  • $\begingroup$ @mbq: cheers! I wasnt too sure about tagging/mentioning java too much; my interest is primarily if it can be done relatively easily, then a Java implementation. But this works as well. $\endgroup$ – posdef Feb 9 '11 at 17:20
  • 1
    $\begingroup$ @posdef, if it is normal, then simulation is easy, just calculate mean and standard deviation. For motivation you will need to test whether the sample is normal. $\endgroup$ – mpiktas Feb 9 '11 at 20:01
  • 1
    $\begingroup$ @posdef, answers require better quality than comments, this is why I commented. As for commercial use, try google to find non comercial. This seems free. In worst case scenario you will need to reimplement this test, or another normality test. I suggest asking here or on stackoverflow , which of the normality tests is easiest to implement. $\endgroup$ – mpiktas Feb 10 '11 at 10:48
3
$\begingroup$

How big are the samples that you need? If substantially smaller than the 5000 points you have, say maximum 100 points or so, you could just take a random subset of your sample. Then you don't even need to assume normality - it's guaranteed to come from the distribution you want!

Otherwise, it seems that the org.apache.commons.math.stat.descriptive.moment package has a Mean and StandardDeviation class which use the correct formulas. These should give you $\mu$ and $\sigma$, respectively.

$\endgroup$
  • $\begingroup$ unfortunately I need to create samples as large as the original dataset, plus I need to do this process a large number of times... So picking random values doesn't seem like a viable option in my case. $\endgroup$ – posdef Feb 10 '11 at 9:30
1
$\begingroup$

You can generate the empirical cumulative distribution function (ecdf). With this, you can generate random variables with uniform distribution and map them to your ecdf. This way, you will generate samples with the desired distribution.

In Java, using Colt, you have the class Empirical, where is written:

Implementation: A uniform random number is generated using a user supplied generator. The uniform number is then transformed to the user's distribution using the cumulative probability distribution constructed from the pdf. The cumulative distribution is inverted using a binary search for the nearest bin boundary.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.