Imagine that I have an unnormalised distribution $P$ with density function $p(x)$ for $x \in \mathbb{R}^d$. $P$ has two well-separated modes and there are two sets of i.i.d. samples with the size $n$ that each of them has been taken from one of the modes. I would like to know how I can combine these samples in an unbiased way. In other words, let $\hat{\mu}_i$ be the empirical estimation of $\mu_i$, then I would like to find the $w_i$ that minimises the following error:

$\min \quad (\sum_{i}w_i(\hat{\mu}_i-\mu_i))^2$

Thanks for your help.


  • 2
    $\begingroup$ Please explain more precisely how you can "sample ... from one of the modes." The language sounds like you might have a mixture. This is crucial because the relative amounts of the components in the mixture determine the solution, but you haven't provided any information about those amounts. $\endgroup$ – whuber Mar 24 '18 at 19:46
  • $\begingroup$ Thanks, consider my problem as a stratified sampling that we do not know the weights of each strata. $\endgroup$ – KiaSh Mar 24 '18 at 19:52
  • 1
    $\begingroup$ Then it would appear you have insufficient information to combine the results in any objective way. $\endgroup$ – whuber Mar 24 '18 at 22:26
  • $\begingroup$ But I have $p(x)$; is not possible to use it to find the weights ? $\endgroup$ – KiaSh Mar 25 '18 at 13:03
  • 1
    $\begingroup$ If you have all of $p$, then there's no question to be asked and your samples aren't needed: simply normalize $p$ to integrate to unity. $\endgroup$ – whuber Mar 25 '18 at 14:03

Here is a solution (see section 3.3) which uses the Renyi entropy:


According to the paper, assuming that we have the Renyi entropy of the samples from each mode and the $p(x_i)$ for those samples, we can find the weight of each mode and the resulting estimator is asymptotically unbiased.

In practice, however, the problem/challenge is how to find the entropy: The authors suggested to use the Information Theoretical Estimators package [1] for estimating the entropy.

[1] Szabó, Zoltán. Information theoretical estimators toolbox. Journal of Machine Learning Research,15:283–287, 2014


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.