# Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

296 questions
Filter by
Sorted by
Tagged with
0answers
109 views

### Relation between the AIC and the Kullback-Leibler Divergence

I am searching a formal derivation of the Akaike Information Criterion from the Kullback-Leibler Divergence. Can you show me one, or point me toward a book/article in which this is done? Here I set ...
1answer
100 views

### A different proof for KL divergence non-negativity

KL divergence's non-negativity can be proved in many ways. One could use the inequality $\log x \leq x - 1$ as a main step in the proof, another one could leverage the property of concave of the ...
1answer
276 views

### clustering with KL divergence

Recently I read some papers which related to clustering. The paper is https://xifengguo.github.io/papers/ICONIP17-DCEC.pdf In this paper, they calculate Loss function as KL Divergence, KL(P||Q). ...
0answers
25 views

### Are there two motivations for Bayesian information criteria?

Are there two motivations for all these Bayesian information criteria? I am only aware of the motivation of "expected out-of-sample prediction score." Let the in-sample data be $y$ and the parameter ...
1answer
52 views

### Bayesian consistency in compact uncountable parameter space

Let $p(y_i \mid \theta)$ be the likelihood we are using of a single data point, $p(\theta)$ be the prior, and $f(y_i)$ the true distribution of the data. Also, let $\theta_0$ be the parameter that ...
1answer
98 views

1answer
167 views

### KL Divergence between parallel lines

I am trying to understand the example described in the WGAN paper about learning parallel lines with various divergences. More specifically the setup is as follows: Let $Z \sim [0, 1]$ the uniform ...
3answers
137 views

### Simulating KL Divergence between Cauchy RV and the MLE estimate of the RV - Multimodality seems wrong

I'm working on a (what I think is a fairly simple, straightforward) explanation of how it's really hard to approximate distributions with fat tails accurately in the tails. I started looking at an ...
1answer
1k views