# Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

302 questions
Filter by
Sorted by
Tagged with
10 views

### Use Effect Size VS p-Value when determining the best fitting distribution?

I have a large sample size of about 50,000 I want to determine what theoretical distribution fits the distribution of my samples the best. What I did is to fit all distributions I know to the data ...
147 views

### What is the relation between ELBO and SGVB?

Evidence lower bound (ELBO) can be minimised, so that to find the most appropriate approximative distribution of the target distribution, which is equivalent to the maximisation of the corresponding ...
45 views

### Different optimization behaviours on delta vs non-delta targets

I built a simple classification model that is required to predict a probability distribution for a set of two available classes. The target distributions are not necessarily delta distributions. i.e ...
168 views

### Does this bounded, continuous probability distribution have a name?

Does this bounded, continuous probability distribution over $x$ have a name? $P(x|y) \propto \big(\frac{y}{x}\big)^x\big(\frac{1-y}{1-x}\big)^{(1-x)}$ for $x, y \in (0,1)$. This comes about by ...
410 views

### KL-divergence between two products

Given factorizations of two joint densities $p(x_1,...,x_n)=\prod_{i=1}^n p(x_i\mid \textrm{cond}(x_i))$ and $q(x_1,...,x_n)=\prod_{i=1}^n q(x_i\mid \textrm{cond}(x_i))$, where $\textrm{cond}(\bullet)$...
213 views

### Label smoothing formula

I recently came across this paper in section 3.2 it talks about label smoothing loss and how it's equivalent to s equivalent to adding the KL divergence between the uniform distribution u and the ...
7 views

25 views

### Why does variational bayes use $KL(Q || P)$ and not $KL(P || Q)$

In variational Bayes, we approximate the intractable posterior $P(Z | X)$ with a tractable $Q(Z)$ and minimize $KL(Q || P)$. Why do we not minimize $KL(P || Q)$ instead?
28 views

### Weigh Kullback Leibler Divergence with P entropy?

I'm wondering if it makes sense to weight Kullback-Leibler Divergences to highlight divergences on highly distinguishing features. I am however not very well versed in Information Theory, and would ...
In a VAE, the encoder learns to output two vectors: $$\mathbf{\mu} \in\ \mathbb{R}^{z}$$ $$\mathbf{\sigma} \in\ \mathbb{R}^{z}$$ which are the mean and variances for the latent vector $\mathbf{z}$, ...