# Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

296 questions
Filter by
Sorted by
Tagged with
79 views

### Properties of the KL topology [reference request]

I'm trying to understand better what are the implications of a sequence of random variables $X_n$ converging toward some limit $X$ in the KL topology, ie the probability density functions are such ...
42 views

### KL divergence and probabilities of 0 for P(i)

Why do probabilities of 0 for $P(i)$ not affect the result of the KL Divergence equation? Regardless of what probabilities we have for $Q(i)$, the product is 0. What are the benefits of this? Is ...
541 views

### Hypothesis test based on entropy

I am reading the wikipedia page on hypothesis testing, but a I can't find any reference to tests based on entropy. Which are good hypothesis tests based on entropy or quantities derived from it?
2k views

### variational inference with KL

i am self-studying variational inference - and in Murphy's book "A probabilistic perspective on machine learning" it is discussed that minimizing the forward KL divergence (which is stated to be zero-...
391 views

### How to use KL-divergence in naive bayes classifier to weight features?

I have a dataset consisting of 4 classes. I have implemented the Gaussian Naive Classifier (in Matlab). In the training phase I calculate the mean and variance for each feature and each class as well ...
49 views

440 views

### KL-divergence as a negative log likelihood for exponential families

I am reading Distributed Estimation, Information Loss and Exponential Families, where the authors consider and compare two estimators for $\theta$ in the parametric model $p(x\mid\theta)$: the ...
587 views

### Why is there a E in the name EM algorithm?

I understand where the E step happens in the algorithm (as explicated in the math section below). In my mind, the key ingenuity of the algorithm is the use of the Jensen's inequality to create a lower ...
885 views

### Relationship between Poisson generation and generalized Kullback-Leibler divergence

I have read that, in the context of matrix factorization, performing maximum likelihood estimation under the assumption that the entries are Poisson generated is equivalent to minimizing the ...
1k views

### When KL Divergence and KS test will show inconsistent results?

I know that Kullback–Leibler divergence and Kolmogorov–Smirnov test are different and should be used in different scenarios. But they are similar in many ways and given two distributions, we could ...
36 views

### Approximation of objective based on statistical distance

I am a computer science researcher (mostly theoretical) currently in midst of statistics and not able to figure out how to proceed. At an abstract level, I have a hypothesis for an unknown ...