0
$\begingroup$

I have an aggregated dataframe where each row contains a mean, standard deviation, and sample size N similar to below:

Mean | Stdev | N

$1023 | 1507 | 23

$3951 | 4136 | 17

$864 | 306 | 112

And so on for about 4000 entries. The two histograms below show the distribution of the means, and the distribution of the log-adjusted means, respectively.

Hist of means

Hist of log-adj means (notice it's bimodal)

I thought about creating N random samples drawn from the Normal(mean, stdev) of each row, but clearly I can't assume they are normal.

I want to fit a probability distribution to the dataframe so I can randomly sample from that estimated distribution. I haven't been able to fit distributions to the regular or log adj distributions because they are bimodal.

How does one typically create the best representative probability distribution of the original dataframe I have?

$\endgroup$
  • $\begingroup$ You could try to find a distribution which fits the data well enough, for ex. what PauZen suggested. You could try to use rejection sampling to sample from this particular distribution. $\endgroup$ – user2974951 Jul 8 at 11:28
0
$\begingroup$

Honestly, there is no good way to answer to your problem since the aggregation lost a part of the information forever.

First, you can fit a gaussian mixture on your log adjusted mean. This will answer your problem of bimodal distribution.

For me, the best you can do is what you propose then. Sample from the data frame distribution and sample from a normal.

In fact there is a simple way to see there is no good way.

Assume the initial data are gaussian (and independent and with same distribution). In this case, their mean is also gaussian.

However if you original are beta distribution for example, then the mean is no more a beta distribution. (see https://math.stackexchange.com/questions/85535/sum-of-n-i-i-d-beta-distributed-variables ). So there is no good answer according to me.

$\endgroup$
  • $\begingroup$ Thanks for the feedback, really helpful! What are your thoughts on using a KDE and drawing random samples from that fitted distribution instead of fitting a GMM? $\endgroup$ – J Doe Jul 7 at 20:53
  • $\begingroup$ The problem is, sample from a KDE is not very efficient (except if you have gaussian kernel where in fact you have a mixture of gaussian). You could without losing too much efficiency use a gaussian kernel KDE (but to be honest, you create more problem, since you need to be sure you won't overfit...) $\endgroup$ – PauZen Jul 7 at 22:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.