Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

Filter by
Sorted by
Tagged with
0
votes
1answer
27 views

Maximum likelihood and Minimizing Kullback–Leibler divergence to the ecdf?! (i.e.: finite sample statement?)

There is a known relationship stating that finding MLE is asymptotically the same is minimizing Kullback–Leibler divergence (see wiki here), or just the cross entropy. I'm wondering if there is a ...
3
votes
1answer
32 views

“Any meaning to the concept of ‘Self Mutual Information?”

** “Any meaning to the concept of ‘Self Mutual Information?” ** A blog post entitled, “Entropy in machine learning” dated May 6, 2019 (https://amethix.com/entropy-in-machine-learning/) gave a very ...
1
vote
0answers
18 views

KL divergence between samples from a unknown distribution and a Normal distribution with zero mean and unit variance

If you draw samples of unknown distribution, how can you measure the KL-divergence between the unknown distribution and a gaussian distribution with zero mean and unit variance N(0,1)? Can we use ...
0
votes
0answers
7 views

Is Kullback-Leibler distance the same as the Kullback Information Criterion?

I have seen some old papers, where they seem to be referring to Kullback-Leibler divergence as the Kullback Information Criterion (e.g. Gourieroux et al (1987) "Kullback Causality Measures"). I just ...
1
vote
0answers
44 views

Statistical estimators distance for close empirical distributions

Is it valid to argue that two empirical distributions $ p_1, p_2 $ having small Wasserstein distance $W_r(\cdot)$ for an order $ r $ will yield close MLE estimators for a statistical model ...
1
vote
1answer
47 views

Kullback Leibler Divergence between two Normal Whishart Distributions

I'm having trouble to compute the KL Divergence between two normal-Wishart distributions. KL divergence from $Q$ to $P$ is defined as: $$D_{\mathrm{KL}}(P \Vert Q) = \int p(x) \log \frac{p(x)}{q(x)}...
0
votes
0answers
21 views

Clarifying notation for the Kullback-Leibler divergence in terms of expectations

We know that for discrete variables \begin{equation} D(p(x),q(x))=\mathbb{E}_{p}\left(\log\frac{p(x)}{q(x)}\right) \end{equation} where $p(x)$ and $q(x)$ are probability mass functions. Can this be ...
1
vote
0answers
27 views

Distance measure for two probability distribution of unequal sample size

Context: I have 100 stores and these stores are divided into 10 business markets. I want to select 3 markets where each market is a good representation of the 100 stores i.e. the population. There ...
0
votes
0answers
18 views

Kullback–Leibler and the Brier score?

Both seem to be quite obviously about prediction and sort of map one probability distribution onto another one. Whereas with the DKL (https://en.wikipedia.org/wiki/Kullback%E2%80%...
1
vote
1answer
101 views

KL divergence between two Asymmetric Laplace distributions?

Consider two asymmetric Laplace distribution $L_1(\mu_1,\sigma_1,\tau_1)$ and $L_2(\mu_2,\sigma_2,\tau_2)$ where \begin{equation} L(x;\mu,\sigma,\tau) =\frac{\tau(1-\tau)}{\sigma} \begin{cases} ...
1
vote
1answer
35 views

Which KL Divergence is larger D(P|Q) or D(Q|P)?

From the perspective of information theory, I understand how D(P|Q) is non-negative and why the KL divergence is asymmetric, i.e. $D(P|Q) \neq D(Q|P)$, given two gaussian univariate gaussian ...
0
votes
0answers
25 views

Maximum likelihood as minimizing the dissimilarity between the empirical distriution and the model distribution

I am reading Ian Goodfellow "Deep Learning" book. At page 128 it says One way to interpret maximum likelihood estimation is to view it as minimizing the dissimilarity between the empirical ...
0
votes
0answers
7 views

Computing KL Divergence for distributions over sets

I have a distribution over a set of (hundreds of) discrete terms, and I'd like to describe the difference between I see a couple of options, and none seems really attractive: Take the KL divergence ...
1
vote
1answer
266 views

Gradients of KL divergence and ELBO for variational inference

When doing variational inference, due to intractability we typically maximize the evidence lower bound (ELBO) instead of minimizing Kullback-Leibler divergence (KLD) between our approximate and exact ...
0
votes
0answers
11 views

Comparing two coin tossing experiments using Kullback-Leibler divergence

In a coin-tossing, suppose two people have conducted two separate experiments to find out the probability of head $p_H$ of the same coin. They both are Bayesians and have started from the prior $$...
2
votes
1answer
36 views

How is Akaike Information Criterion related to Information theory?

How is the Akaike Information Criterion (AIC) related to Information theory ? I mean from the equation (below), it is not at all intuitive how information theory comes into picture. Also is AIC ...
0
votes
1answer
62 views

How to solve for the minimum KL Divergence when the distribution is discrete?

Say we have a simple case of $p(x,y)$ is a 3x3 matrix: $$\begin{bmatrix} 1/6 & 0 & 0 \\ 1/6 & 3/6 & 0 \\ 0 & 0 & 1/6 \end{bmatrix}$$ And $q(x,y)=...
0
votes
1answer
20 views

Use Effect Size VS p-Value when determining the best fitting distribution?

I have a large sample size of about 50,000 I want to determine what theoretical distribution fits the distribution of my samples the best. What I did is to fit all distributions I know to the data ...
0
votes
0answers
12 views

Proof help for multivariate mutual information as a sum of entropies

I'm following this paper on ICA and I got to equation (1) describing the multivariate mutual information contrast function as a sum of entropies. $J(Y) = \int p(y_1,...y_D)log(\frac{p(y_1,...y_D)}{p(...
0
votes
1answer
64 views

Variational inference with dependent variables

Classical variational inference uses mean field theory because of its computational benefits, i.e. assumes that the latent variables are independent. For gaussian distribution, it wants to find a ...
0
votes
1answer
60 views

Different optimization behaviours on delta vs non-delta targets

I built a simple classification model that is required to predict a probability distribution for a set of two available classes. The target distributions are not necessarily delta distributions. i.e ...
1
vote
1answer
44 views

The expected log-Likelihood in Kullback Leibler Divergence

Given a true normal distribution $g(x)$ with mean $\mu_G$ and variance $\sigma_G$, and a model $f(x)$, the KL divergence involves the expected log-likelihood $\mathbb{E}_G[log f(x|\theta]$. The ...
1
vote
1answer
31 views

Why does variational bayes use $KL(Q || P)$ and not $KL(P || Q)$

In variational Bayes, we approximate the intractable posterior $P(Z | X)$ with a tractable $Q(Z)$ and minimize $KL(Q || P)$. Why do we not minimize $KL(P || Q)$ instead?
0
votes
0answers
35 views

Weigh Kullback Leibler Divergence with P entropy?

I'm wondering if it makes sense to weight Kullback-Leibler Divergences to highlight divergences on highly distinguishing features. I am however not very well versed in Information Theory, and would ...
0
votes
1answer
133 views

KL Divergence of two standard normal arrays

I generated two 9000,1 np arrays with a = np.random.standard_normal(9000) b = np.random.standard_normal(9000) Then I check the KL Divergence with ...
1
vote
0answers
39 views

Expectation of Log Likelihood Function with given parameters Proof

I have been looking for AICc value derivation employing Kullback-Leibler distance but as a result of my search I got stuck with expectation of loglikelihood. In the link loglikelihood is given as $InL(...
1
vote
0answers
65 views

How to show an alternate data processing inequality concerning KL divergence between conditionals?

Suppose $(Y,X) \sim F \in \mathcal{P(\mathbb{R^d})}$. Consider an arbitrary transformation $f$ that acts on $X$. My intuition is that the following should be a result in information theory: $$ \...
0
votes
1answer
165 views

Meaning of expectation with respect to a function?

This might be a trivial question, but I've come across a paper where some expectation is said to be taken with respect to some pdf. See example: How am I to interpret this, and is there some ...
2
votes
1answer
90 views

Variational inference with deterministic dependencies between variables

Suppose I have a probabilistic graphical model shown in the picture, in which all variables are binary, $c_1$ and $c_2$ are observed, and I want to use mean-field variational inference to estimate ...
0
votes
0answers
25 views

What is a reasonable value for KL Divergence

I am using scipy.stats.entropy for KL Divergence. When I create distributions to test I never receive values greater than .5. When I run the function on data I ...
1
vote
1answer
81 views

KL divergence minimization

While reading Unsupervised Data Augmentation for Consistency Training, I came across an equation that describes the minimization of KL divergence. $$\min_\theta \mathscr{J}_{UDA}(\theta) = \mathbb{E}...
0
votes
0answers
24 views

“Compressing” or downsampling a discrete probability distribution

I have a discrete probability distribution $P$ which I obtained by applying a softmax transformation, with an automatically-derived exponent $-\beta$, to a set of measurements (potentially large, in ...
2
votes
2answers
128 views

Expectation of the log-likelihood under the posterior

Suppose $L(X \mid \theta)$ is a likelihood function, i.e., a probability distribution over $X \in \mathcal{X}$ indexed by a parameter $\theta \in \Theta$. Suppose further we have a prior $\pi(\theta)$,...
0
votes
0answers
118 views

Measure the distance between two probability transition matrices

I have a probability transition matrix $P$ that contains some values very close to zero. I want to sparsify this matrix by taking the k largest values for each row and setting the others to zero. For ...
0
votes
0answers
18 views

How to minimise and expectation with respect to a parameter?

Suppose that $X$ is a random variable with distribution $G$. Let $H(X;\theta)$ be a parametric function with $\theta \in \Theta \subset {\mathbb R}^p$. I want to maximize the function $$\varphi(\...
1
vote
0answers
94 views

How to use KL divergence to compare two distributions?

I am trying to model the probability distribution of a multi-dimensional dataset where all the values are discrete. Suppose the training data (represented by T) is of the shape (m, n) where n is the ...
0
votes
1answer
108 views

KL divergence of a uniform prior and a custom posterior

So I was reading the Google's paper on VQ-VAE and have stumbled upon the derivation of KL divergence of the uniform prior and the given distribution: $$q(z=k \mid x)=\left\{\begin{array}{ll}{1} & ...
0
votes
0answers
27 views

How to prove KL(q(z)|p(z)) = E_q(z) [ log f(z) ] where f is the CDF of p?

As title. It was used in https://arxiv.org/abs/1905.10549 without proving.
0
votes
0answers
46 views

Kullback-Leibler divergence Loss With different-length vectors

I am new to KL Divergence Loss (and indeed all similar comparisons between discrete series data). The output of my network produces a series of tuples of a length that varies during training. The ...
1
vote
0answers
8 views

Markov type bound based on KL divergence?

Given two discrete distribution $p, q$ on some universe $U$, if I know they have a bounded KL divergence, say some number $c$, can I say anything about how much each point in the universe differs in ...
9
votes
1answer
912 views

Why is Kullback-Leilbler divergence a better metric for measuring distance between two probability distributions than squared error? [duplicate]

I know that KL-divergence is a metric that is more suitable when we want to measure the distance between numbers which a probability form. However, I am still confused what is the benefit of using KL-...
1
vote
0answers
22 views

A divergence that can be extended to logistic functions?

If I have data $\{(x_i, y_i)\}_{i=1}^n$ where the dependent variable is binary $(y_i = 0,1)$ I can model it using a logistic function: $$f(x; \alpha, \beta) = \frac{1}{1 + e^{-(\alpha + \beta x)}}$$ ...
2
votes
0answers
116 views

Relation Between Wasserstein Distance and Relative Entropy

Consider the Wasserstein metric of order one $W_1$ (aka the Earth Movers Distance). I would like to know whether it is possible to link $W_1$ and relative entropy and what this would mean intuitively. ...
1
vote
1answer
84 views

Can multivariate Gaussians KL divergence be a negative value?

I'm trying to find if two hidden neurons in RBF Network overlap with each other or not? It's an online classification problem, it means data come to our network one-by-one and then discard completely. ...
1
vote
1answer
578 views

KL divergence between gaussian and uniform distribution

Is the KL divergence not defined because uniform has bounded support and gaussian has unbounded support? How else can I calculate the distance of my gaussian to a 'maximum entropy' distribution if I ...
0
votes
0answers
22 views

Quantifying information loss (KL divergence?) between a multivariate and a univariate discrete distribution

Let's say I have n discrete variables, n1, n2, ... n_n, each with a different scale, and another discrete variable ...
1
vote
0answers
37 views

Running two MCMC chains in parallel while minimizing Kullback-Leibler divergence between both sample distributions

I want to sample from a distribution $p(X)$ with $X \in R^n$. However, I can only evaluate the likelihoods of $Z = AX$ and $Z = BX$ with $A,B \in R^{m \times n}$ and $m = n-1$. Now my idea is to run ...
2
votes
1answer
335 views

What is the relation between ELBO and SGVB?

Evidence lower bound (ELBO) can be maximized, so that to find the most appropriate approximative distribution of the target distribution, which is equivalent to the minimization of the corresponding ...
2
votes
0answers
43 views

Name/definition of $\int \log F(x) \cdot g(x)dx$?

We know that: $$-\int \log f(x) \cdot g(x)dx,$$ where $f$ and $g$ are density functions, is known as the cross entropy. Does $$-\int \log F(x) \cdot g(x)dx,$$ where $F$ is the cumulative ...
0
votes
2answers
41 views

Mysteriously defined KL-divergence term [duplicate]

I am trying to re-create a variational autoencoder. The loss function has two terms: reconstruction loss and KL-divergence term. KL-divergence is defined as $$ D_{KL}(P||Q) = -\sum_{x\in X}{P(X)\log\...