Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

Filter by
Sorted by
Tagged with
1
vote
1answer
41 views

How to compare if two multinomial distributions are significantly different

We can use T test to check if two proportions are significantly different. Similarly is there a way to test if two multinomial distributions or "2 samples with more than 2 unique values" are ...
1
vote
0answers
25 views

KL-Divergence to evaluate regression model performance [duplicate]

Would it make sense to use KL-Divergence to measure the difference in predictions versus ground truth for a regression problem? I've tuned four models and serve the average as a prediction in the ...
0
votes
0answers
28 views

KL Divergence Between Ground Truth and Prediction

I've got four (non-linear, tree-based) models in production and using the average of them as the served prediction. We get ground truth data immediately. During training the optimized candidate models ...
0
votes
0answers
8 views

Rising “sub-losses” when optimizing sum of losses with ADAM optimizer (KL divergence, neg. loglik)

I am optimizing the ELBO as part of the variational inference using a neural network for a dynamic topic model (D-ETM). The loss being optimized is the sum of the negative log-likelihood and KL ...
0
votes
0answers
17 views

If entropy is the underlying measure for KL-divergence, what is the underlying measure for the Wasserstein distance?

If entropy is the basis measure underlying KL-divergence (aka relative entropy), what is the basis measure underlying the Wasserstein distance?
3
votes
1answer
47 views

Is it always better to average out parameter uncertainty?

Setup If we have a data set $y_1, \ldots, y_t := y_{1:t}$, and we're trying to predict $y_{t+1}$, a Bayesian would try to use the posterior predictive distrbution $$ p(y_{t+1} \mid y_{1:t}) = \int p(...
2
votes
1answer
55 views

Optimizing forward/reverse KL divergence for Gaussian distributions

The forward/reverse formulations of KL divergence are distinguished by having mean/mode-seeking behavior. The typical example for using KL to optimize a distribution $Q_\theta$ to fit a distribution $...
1
vote
0answers
26 views

Is there a rate of change performance measure for KL-divergence?

In the example figure below, KL-divergence is being used to measure how far the distribution of different parameterizations of Poisson are from an empirical distribution (real data). The minimum of ...
0
votes
1answer
29 views

Multivariate Jensen-Shannon divergence

This paper says multivariate Jensen-Shannon divergence is $$JS(\mathbf{p}_1,\dots,\mathbf{p}_K) = \frac{1}{m} \sum KL(\mathbf{p}_i || \bar{\mathbf{p}})$$ with $KL$ being the KL-divergence of the ...
0
votes
0answers
49 views

To compare two KL-divergence scores, does the prior model have to be the same for both?

The KL-divergence compares a theoretical model $p$'s distribution with the empirical model $q$'s distribution, giving a score of $0$ if they, or their information contents, are identical. Say we have ...
0
votes
0answers
14 views

Can $f$-divergences narrow the discrepancy between train and test fits in machine learning?

Machine learning models whose task is to predict unseen test data would work best if the test data's distribution turns out to be the same as the training data's distribution. Real data seldom works ...
1
vote
2answers
38 views

Kullback-Leibler divergence between marginals and joint distribution? (It's not mutual information)

Mutual information is defined as the Kullback-Leibler divergence between a joint distribution and its marginals: $$I(X,Y) = \mathrm{KL}(P(x,y)||P(x)P(y)) = \sum_{x,y}P(x,y)\ln\left(\frac{P(x,y)}{P(x)P(...
5
votes
1answer
165 views

Are Mutual Information and Kullback–Leibler divergence equivalent?

From my readings, I understand that: Mutual information $\mathit{(MI)}$ is a metric as it meets the triangle inequality, non-negativity, indiscernability and symmetry criteria. The Kullback–Leibler ...
1
vote
0answers
21 views

Does it make sense to use the KL-divergence between joint distributions of synthetic and real data, as a evaluation metric?

The KL-divergence is defined as: $D_{KL}(p(x_1)∥q(x_1))=\sum p(x_1)\, \log \Big( \dfrac{p(x_1)}{q(x_1)} \Big)$ I consider the Kullback-Leibler (KL) divergence as a performance metric for data ...
2
votes
2answers
68 views

Should reconstruction loss be computed as sum or average over input for variational autoencoders?

I am following this variational autoencoder tutorial: https://keras.io/examples/generative/vae/. I have included the loss computation part of the code below. I know VAE's loss function consists of the ...
1
vote
0answers
29 views

Is KL-divergence just the multiplication rule for independent events, reformulated in terms of entropy?

We know KL-divergence is sometimes expressed like this: which shows it's capturing the deviation between the joint distribution of X and Y, and the product of marginals for X and Y. This suggests KL-...
0
votes
0answers
18 views

Measurement to compare probability distributions

I have a set of probability distribution that I want to compare. Right now, I'm relying on (subjective) ocular evaluations of the plot. Is there any more appropriate statistical measurement? ...
2
votes
1answer
60 views

Kullback-Leibler divergence and marginals

Let $P(x,y)$ and $Q(x,y)$ be two probability distributions, with marginals $$P(x)=\sum_y P(x,y),\quad P(y)=\sum_x P(x,y),\quad Q(x)=\sum_y Q(x,y),\quad Q(y)=\sum_x Q(x,y)$$ What is the relation ...
0
votes
0answers
18 views

Finding the lower bound for relative entropy $D(f||g)$ where, $f$, $g$ are two different distribution?

I am trying to find a tighter bound of the relative entropy $D(f||g)$. Problem statement: Let, $f$ and $g$ be two discrete probability distribution. $f \in \left[ {Q(x + a),Q( - x - a)} \right]$ and $...
9
votes
1answer
148 views

What's an intuitive way to understand how KL divergence differs from other similarity metrics?

The general intuition I have seen for KL divergence is that it computes the difference in expected length sampling from distribution $P$ with an optimal code for $P$ versus sampling from distribution $...
5
votes
1answer
85 views

KL-divergence: P||Q vs. Q||P

Assume, that we have several data generating measures $P_{1}, \dots, P_{k}$ and $Q$, all defined on the same probability space. Next, assume, we have the same amount of independently sampled data from ...
0
votes
0answers
15 views

Minimize the limit of K-L (Kullback Leibler) divergence for a given conditional probability $p(y|x)$ distribution?

Let, $p(x);p(y)$ are the probability distribution function of random variable $X$, $Y$ and the Conditional probability $p(y|x)$ is given e.g. $p(y|x)=Q(x+2y)$. where, $Q(x) = \frac{1}{{\sqrt {2\pi } }}...
2
votes
1answer
41 views

Sensitivity of KL Divergence

I am very new to the concept of KL divergence. Although I have grasped the fundamental formulations, I have a confusion comparing the KL divergence across the different distributions. Suppose I have 3 ...
1
vote
1answer
40 views

estimation of KL-divergence of continuous distributions

Assume we have two independently sampled datasets, $X = \{x_{1}, \dots, x_{n}\}$ and $Y = \{y_{1}, \dots, y_{m}\}$ from continuous distributions $f$ and $g$. I aim to estimate the KL-divergence ...
0
votes
0answers
25 views

Does minimizing KL-divergence result in maximum entropy principle?

The Kullback-Leibler divergence (or relative entropy) is a measure of how a probability distribution differs from another reference probability distribution. I want to know what connection it has to ...
1
vote
0answers
28 views

Similarity between two conditional discrete distribution

Having data of X:Y where X is a categorical feature and Y is multiclass label (i.e Sweet:Apple , Red: Apple, Sweet:Orange, Sugary: Banana ) I've created for each X the conditional frequency/ ...
0
votes
0answers
13 views

Power-density divergence for logistic regression

How is power-density (or $\beta$-divergence) written for the binary logistic regression problem?
5
votes
2answers
422 views

Using cross-entropy for regression problems

I usually see a discussion of the following loss functions in the context of the following types of problems: Cross entropy loss (KL divergence) for classification problems MSE for regression ...
2
votes
1answer
26 views

How much additional information should be provided if approximating a distribution? (KL Divergence)

"If we use a distribution that is different from the true one, then we must necessarily have a less efficient coding, and on average the additional information that must be transmitted is (at ...
3
votes
1answer
179 views

Cross entropy vs KL divergence: What's minimized directly in practice?

My understanding is that in ML one can establish a connection between these quantities using the following line of reasoning: Assuming we plan to use ML to make decisions, we choose to minimize our ...
0
votes
0answers
43 views

Understanding concrete dropout

I am trying to understand a technique call Concrete Dropout for automatically tuning the dropout rate during the network training. However, I am unable to follow the work described in the paper ...
0
votes
1answer
20 views

Lemma KL-Divergence (Differential Privacy)

I am studying differential privacy and I got stuck again in proof of a lemma. Which is: "$D_{\infty}^\delta(Y||Z) \leq \epsilon$ if and only if there exists a random variable $Y'$ such that $\...
1
vote
1answer
52 views

Laplace and Normal Distribution Cross Entropy

I need the following integral and struggle with calculating it or finding a citable source. $$\int_{-\infty}^{\infty}(x-\mu)^2\exp\!\left(-\frac{|x-\nu|}{\tau}\right)dx.$$ Background: I want to find ...
0
votes
0answers
90 views

KL Divergence Normal and Laplace densities

I want to calculate the KL-Divergence between a Laplacian density g and a normal density f. I can decompose $KL(G|F)$ to $\mathbb{E}_g[\log g(X)]-\mathbb{E}_g[\log f(X)]$. I am already stuck with my ...
0
votes
0answers
44 views

Transformation of discrete KL-Divergence in Continuous KL-Divergence possible?

I Use Python and the following definition of KL-Divergence def kl_divergence(p, q): return np.sum(np.where(p != 0, p * np.log(p / q), 0)) to calculate the ...
2
votes
1answer
28 views

what is -0.5 in VAE loss function with KL term

The VAE loss is composed of two terms: Reconstruction loss KLD loss in the implementation there is -0.5 applied to KLD loss. Kindly let me know what is this -0.5
0
votes
0answers
16 views

How to minimize KL Divergence in VAE loss?

I am training VAE autoencoder model. VAE has loss combining MSE+KL divergence. When I train the model, KL loss is increasing over or near 100 while MSE loss is decreasing. So, can anyone tell me what ...
1
vote
0answers
32 views

Should I use the sum of KL divergences for multi-objective model selection?

I have a model implemented in Python with 2 free parameters. I would like to find the parameter values that provide the best fit to empirical data comprising of response times and accuracy of human ...
1
vote
1answer
82 views

Which definition of Kullback-Leibler-Divergence for discrete & continuos variables makes sense?

Motivation In variational inference, one tries to approximate a posterior density $p(\theta|y)$ by another,easier, density $q(\theta)$ in terms of the Kullback-Leibler-Divergence $$D_{KL}(q(\theta)||p(...
2
votes
0answers
48 views

When do expected KL-divergence and expected MSE coincide?

The AIC is an approximately unbiased estimator of the (relative) risk of the Kullback-Leibler loss. I read that If you use AIC to choose among a family of models, AIC (approximately) yields the model ...
0
votes
0answers
5 views

Centroid wrt Kullback-Leibler divergence with Kronecker deltas

I have the following problem. Suppose I have a set of discrete probabilities $f_i:A \rightarrow[0,1], \ \ i=1,...,m$, such that $\forall i \ \ \exists a_i : f_i(a_i) =1$, that is they are constant. ...
7
votes
1answer
589 views

Is my proof that relative entropy is never negative correct?

I wish to prove that relative entropy(Kullback-Liebler divergence) is always non-negative. I.e. that $$I^{KL}(F;G)=E_F\left[\log\frac{f(X)}{g(X)}\right]\geq0$$ where F,G are two different probability ...
1
vote
2answers
40 views

When to use distance between distributions instead of using median based statistics?

I have a problem where I have to compare the effect of X ( univariate random variable) on distribution of Y (univariate random variable) between 2 different cases. Y is not following a Normal ...
0
votes
0answers
33 views

Kullback–Leibler divergence between two inverse Wishart distributions

Is there a closed form formula for the KL divergence between two inverse Wishart distributions?
1
vote
1answer
48 views

Variational autoencoders: Computational vs. analytical intractability of KL divergence

I am currently trying to understand the ideas behind variational autoencoders. Specifically, I am a trying to understand why the KL divergence between the approximate posterior $q(z | x)$ and true ...
2
votes
0answers
76 views

If the KL divergence is not a metric or a measure, what is it?

The KL divergence is not a metric because e.g. it does not satisfy the symmetry property that metrics posses. According to the definition of measure, the KL divergence doesn't seem to be a measure, ...
0
votes
1answer
42 views

KL-Divergence and Entropy for marginals

I am going through this paper, where the following claim is unproven (page 3, after the first equality): Let $r,c \in \mathbb{R}_+^d$ be discrete probability histograms, and $P \in \mathbb{R}_+^{d,d}$...
5
votes
2answers
307 views

Relationship between KL divergence and entropy

In Bishop's Pattern Recognition and Machine Learning, there is a small discussion in section 10.1.2 of the difference between minimizing $D_{KL}(p \:||\: q)$ and $D_{KL}(q \:||\: p)$ with respect to ...
5
votes
1answer
118 views

Why is Kullback Leibler Divergence always positive?

I know there have been mathematical treatments of this question on here. What I'd like help with is my intuitive understanding though. Take the example given on Wikipedia: $$\begin{array}{|c|c|c|c|} \...
2
votes
1answer
27 views

Can $G^2$ statistic in log-linear model for contingency tables be negative?

Can $G^2$ statistic of log-linear (unsaturated) model in contingency tables be negative? Since saturated model with perfect fit has $G^2=0$ I don't think the unsaturated models can get negative $G^2$. ...

1
2 3 4 5
8