4
$\begingroup$

Does this bounded, continuous probability distribution over $x$ have a name?

$P(x|y) \propto \big(\frac{y}{x}\big)^x\big(\frac{1-y}{1-x}\big)^{(1-x)}$ for $x, y \in (0,1)$.

This comes about by setting $P(x|y) \propto 2^{-\mathrm{KL}(X\|Y)}$, where $\mathrm{KL}(X\|Y)$ is the KL-Divergence from $Y$ to $X$, and $X$ and $Y$ are Bernoulli distributions with parameters $x$ and $y$, respectively.

More generally, is there a name for the approach of defining the Bayesian prior probability of a distribution, e.g., $X$, conditioned on another distribution, e.g., $Y$, to be equal to $2^{-\mathrm{KL}(X\|Y)}$? This seems like a natural choice when you want $X$ to be similar to $Y$, but I've never come across it before.

$\endgroup$
4
  • $\begingroup$ I don't understand. You have $X \sim Bern(x)$ and $Y \sim Bern(y)$, and then you end up with the distribution of $(x,y)$, that are the bernulli parameters? is that right? $\endgroup$
    – niandra82
    May 15 '18 at 20:28
  • $\begingroup$ Yes, that's right. I'm defining a distribution $P(x|y)$ on the Bernoulli parameter for $X$ conditioned on the Bernoulli parameter for $Y$. This distribution is defined according to the KL-Divergence from $Y$ to $X$. $\endgroup$ May 15 '18 at 20:39
  • 1
    $\begingroup$ I think you need to provide a lot more information for readers to understand the value you see in this. Based on my own reading and the existing answers, there is not enough information provided to help understand why you feel it is useful. Could you provide some context and a derivation? $\endgroup$
    – ReneBt
    May 16 '18 at 4:01
  • $\begingroup$ @ReneBt: This approach is useful as a very simple way to describe a Bayesian prior over a distribution $X$, conditioned on another distribution $Y$, such that $X$ is encouraged to be somewhat similar to $Y$. We define a prior distribution over $X$ so that its prior probability is equal to $2^{-\mathrm{KL}(X\|Y)}$, where $\mathrm{KL}(X\|Y)$ is the standard KL-Divergence of $X$ from $Y$.This is a satisfying prior from an information theoretic point of view, and is very easy to describe and justify. $\endgroup$ May 16 '18 at 12:50
1
$\begingroup$

There are three problems with this distribution:

  1. the phrase " ... Bayesian prior probability of a distribution, e.g., X, conditioned on another distribution, e.g., Y" for me makes no sense, since your distribution, for your admission, is a distribution for the parameters of the Bernoulli and not the X and Y.

  2. the second one is that it has only one parameter ($y$), which is equal to the mode (and not the mean, and this could be another problem) and you have no way to specify the variability of your distribution around the mode.

  3. if you are trying to model the bernoulli parameters, I suspect that then you need to specify also a prior over $y$. $y$ appears in the normalization constant of $p(x|y)$ that (i may be wrong) is hard to compute.

And, to answer the question, I don't think it has a name because it is not very useful

$\endgroup$
3
  • $\begingroup$ Thanks for your thoughts. To reply: 1. The distribution on the parameters implies a distribution on the distributions, in the obvious way. 2. I don't care about specifying the variance; one parameter works just fine for this purpose. 3. No prior over y is necessary; you can take it as a fixed but unknown constant. As to your final comment, it is useful because it is specific case if the general strategy I described at the end of my question. $\endgroup$ May 16 '18 at 0:09
  • $\begingroup$ 1) I still don't understand, but it is not important here. 2) ok 3) if it is unknown and you want to estimate it, you must specify a prior if you are Bayesian, or maximize the likelihood if you are frequentist. In both cases you need the normalisation constant. If I'm wrong, you should explain better what you mean $\endgroup$
    – niandra82
    May 16 '18 at 7:45
  • $\begingroup$ 1. A distribution over $x$ implies a distribution over $X$, since there is a 1-to-1 correspondence between any value of $x$ and the shape of the distribution $X$ that it produces. 3. I don't want to estimate $y$. I want to understand the distribution on $x$ conditioned on some particular value of $y$. You're right that I've omitted the normalization constant, which will depend on $y$. But I'm not trying to estimate $y$, so there is no need for a prior over it. $\endgroup$ May 16 '18 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.