The tag has no wiki summary.

learn more… | top users | synonyms

0
votes
0answers
9 views

R “Entropy” package gives weird KL divergence results

Using the R "entropy" package, I tried some KL divergence computations as a sanity check, but I'm getting weird results. For instance, shouldn't the following all be 2*log2(2)= 2 ? Instead, I'm ...
2
votes
0answers
20 views

KL-divergence as a negative log likelihood for exponential families

I am reading Distributed Estimation, Information Loss and Exponential Families, where the authors consider and compare two estimators for $\theta$ in the parametric model $p(x\mid\theta)$: the ...
1
vote
1answer
42 views

f(y | x) or f(y,x) in regression and MLE

In $Y = aX + b + \epsilon$ where $\epsilon$ ~ $N(0,\sigma^2)$ and i.i.d regression setting If X is stochastic and $E(\epsilon\mid X) =0$, then which one is correct: (1) $f(x,y) = ...
0
votes
1answer
36 views

Relationship between Poisson generation and generalized Kullback-Leibler divergence

I have read that, in the context of matrix factorization, performing maximum likelihood estimation under the assumption that the entries are Poisson generated is equivalent to minimizing the ...
1
vote
0answers
15 views

When KL Divergence and KS test will show inconsistent results?

I know that Kullback–Leibler divergence and Kolmogorov–Smirnov test are differnt and should be used in different scenarios. But they are similar in many ways and given two distributions, we could ...
0
votes
0answers
16 views

Type I errors on Hypothesis testing with KL divergence

I am performing a hypothesis test for data from an empirical distribution, $q$, where my null hypothesis is that the data is sampled from distribution $p_0$ and the alternative is that it is sampled ...
2
votes
0answers
14 views

Approximation of objective based on statistical distance

I am a computer science researcher (mostly theoretical) currently in midst of statistics and not able to figure out how to proceed. At an abstract level, I have a hypothesis for an unknown ...
3
votes
0answers
34 views

Asymmetry of the Kullback-Leibler distance in hypothesis testing

My question is related to the asymmetry of the Kullback-Leibler distance. I'm using the discrete definition of the Kullback-Leibler dinstance, so we have: $KL(p,q) = \sum_{s \in S} p(s) ...
1
vote
1answer
46 views

KL divergence between a gamma distribution and a lognormal distribution?

Is there a closed-form formula for the KL divergence $D_{KL}(X,Y)$ where $X \sim \mathrm{Gamma}(k,\theta)$ and $Y \sim \mathrm{LogNormal}(\mu,\sigma^2)$ ? Many thanks.
0
votes
0answers
7 views

How good is 0.01 Jensen-Shannon divergence score?

I want to compare a density function $p$ with the ground truth $g$ by Jensen-Shannon divergence. I am a bit unclear of how to interpret the results. In particular, how do people often conclude $p$ ...
2
votes
0answers
94 views

Minimizing KL divergence from a given distribution, according to a graph

Given $n$ discrete random variables $X_1,...,X_n$, a distribution $p$ on $X=(X_1,...,X_d)$ and a DAG (Directed Acyclic Graph) $G$ on $\{1,...,d\}$, which is the distribution $q$ factorizing with $G$ ...
2
votes
1answer
71 views

Information theory without normalization

I'd like to know if there is a way anyone knows of for doing information theory with unnormalized densities. Specifically, I hav two log likelihoods $\phi(x), \psi(x)$ and so I can write: $p(x) = ...
1
vote
0answers
27 views

Kullback Leibler divergence “efficient” upper bound

For a distribution of N values, how can I efficiently upper-bound the largest divergence between all non-negative distributions over the same random field? For example, for all distributions of a ...
0
votes
0answers
27 views

Want to findout satistical distance unsing another procedure except kullback liebler divergence

i want to find the distance between two pdf(pdf is calculated using kernel density estimator from two random data set of different size ) .Is there any alternative and efficent way to calculate ...
2
votes
2answers
91 views

Why KL-Divergence uses “ln” in its formula?

I notice in KL-Divergence formula a $ln$ function is used: $${D_{KL}}(P||Q) = \sum\limits_i {P(i)} \ln \frac{{P(i)}}{{Q(i)}},$$ where $i$ is a point and $P(i)$ the true discrete probability ...
1
vote
1answer
122 views

Jensen Shannon Divergence vs Kullback-Leibler Divergence?

I know that KL Divergence is not symmetric and it cannot be strictly considered as a metric. If so, why is it used when JS Divergence satisfies the required properties for a metric? Are there ...
0
votes
0answers
20 views

lower bound on square root of KL divergence

$\mu_1$ and $\mu_2$ are two probability measures on $\Omega$. Assume they have probability mass functions $f_1$ and $f_2$. I wonder if the following is a correct usage of Jensen' inequality: $$ ...
0
votes
0answers
44 views

Difference between two data distributions

I have the following problem: I have two datasets: A and B. From each of the datasets, I extracted the same features. I also know the data labels. So, I would like to perform 10-fold CV to see how a ...
1
vote
0answers
33 views

Occupancy octree metrics (Kullback-Leibler)

As I'm currently working on scan matching for outdoor environments I was wondering about the best metric to compare two occupancy octrees (one resulted from the scan matching and one ground truth ...
0
votes
3answers
154 views

Analysis of Kullback-Leibler divergence

Let us consider the following two probability distributions ...
1
vote
0answers
54 views

Neuroscience Equations

I am trying to understand a neuroscience article by Karl Friston. In it he gives three equations that are, as I understand him, equivalent or inter-convertertable and refer to both physical and ...
0
votes
1answer
149 views
1
vote
0answers
146 views

Under what conditions will Kullback-Leibler divergence/mutual information be infinity?

For two perfectly correlated Gaussian variables, the mutual information between them, and thus the KL divergence between the product of the marginal distributions and the joint distribution, is ...
1
vote
0answers
50 views

How to estimate similarity between several probability distributions?

I have several set of probability distributions. I want to reliably estimate consistency across distributions inside each set. Literature contains methods to compare two distributions: ...
0
votes
0answers
47 views

sufficient statistic and KL-divergence: Confusion with an equation

I am reading a paper, which talks about minimising KL-divergence of any arbitrary distribution over a family of exponential distribution. So, given a distribution $p$, we want to compute its ...
1
vote
0answers
65 views

computing KL divergence: M projections for arbitrary distributions

Background I have a generative model for a process that can be described as follows: $$ y = t(x, w) + e $$ where $x$ and $y$ observations of a set of random variables which are related by a ...
0
votes
0answers
57 views

Kullback-Leibler divergence Pareto Distribution

What is the Kullback-Leibler divergence for a Pareto Distribution? Given $p(x)$ = $ \alpha$ $\frac{x^{\alpha}_{min,1}}{x^{a+1}}$.
7
votes
3answers
3k views

Calculate the Kullback-Leibler Divergence in practice?

I am using KL Divergence as a measure of dissimilarity between 2 $p.m.f.$ $P$ and $Q$. $$D_{KL}(P||Q) = \sum_{i=1}^N \ln \left( \frac{P_i}{Q_i} \right) P_i$$ $$=-\sum P(X_i)ln\left(Q(X_i)\right) + ...
3
votes
1answer
141 views

KL divergence between an uninformative (?) Gaussian and a Gaussian

I have to calculate the KL divergence between a distribution $q$ and a prior distribution $p$, both of which are univariate Gaussians, i.e. $KL(q|p), q \sim \mathcal{N}(\mu, \sigma^2), p \sim ...
1
vote
0answers
76 views

Problem with Kullback–Leibler divergence criteria

I am using Kullback–Leibler divergence criteria for comparing my estimation and true density functions, but I have zero value on my estimation function when I have a testing set of size 10000, mostly ...
10
votes
2answers
433 views

What is the relationship between the GINI score and the log-likelihood ratio

I am studying classification and regression trees, and one of the measures for the split location is the GINI score. Now I am used to determining best split location when the log of the likelihood ...
2
votes
0answers
68 views

Quantify the information lost given by the Kullback-Leibler divergence measure

Consider there are $N$ individuals and these measure a quantity $X\in \mathbb{R}^{N\times M}$ where $M$ is the number of measurements and let $P(X)$ denote a probability distribution over $X$. The ...
2
votes
1answer
160 views

How to test whether a series data follow Ornstein-Uhlenbeck process (OU process)?

I have some measures which seems to have some mean-reverting properties and I'm wondering whether they can be modeled as Ornstein-Uhlenbeck process (OU process). And actually I quite expect it because ...
0
votes
1answer
38 views

How to use Kullback–Leibler divergence to help me choose the best distribution function?

I am now using Exponential distribution to model the the time intervals of a sequence of random events.Since I can choose several different lamdas for this model , I want to find out which lamda of ...
1
vote
1answer
61 views

KL divergence minimisation equation

I am looking at some literature on KL divergence minimisation and am having trouble understanding the derivation of the second order moment. So, if we have a distribution from the exponential family, ...
1
vote
0answers
25 views

Modeling a list with a tunable degree of disorder/shuffling

Imagine we have a list of ordered numbers $L = (1, 2,\dots, N)$. I want to add an arbitrary amount of "disorder" to that list. For instance: Adding a little bit of disorder would permute a few ...
1
vote
1answer
61 views

Renyi divergence identity

I'm reading the paper, T. van Erven and P. Harremoës, Rényi Divergence and Kullback-Leibler Divergence, arXiv 1206.2459 on the Renyi divergence, and I'm trying to make sense of "Example 1". I think ...
1
vote
0answers
58 views

Formal statistical test for comparing likelihood distributions obtained via MCMC

I am trying to formally compare the distribution of the likelihood values generated using two different models with marginal posterior values of the parameters obtained using MCMC in order to assess ...
2
votes
0answers
224 views

Disadvantages of the Kullback-Leibler divergence

I'm working on a calibration problem which involves the usage of the Kullback-Leibler divergence as an error between some empirical distribution $p$ and a theoretical distribution $q$. In the model, ...
2
votes
0answers
79 views

Does relative Kullback-Leibler divergence exist?

Suppose I have two multivariate normal distributions. I have computed the KL divergence ($d_{LK}(N_1, N_2)$). Is there a way to measure a relative divergence between these two distributions? For ...
3
votes
1answer
500 views

KL-divergence between two categorical/multinomial distributions gives negative values?

If $$P = [0,0.9,0,0.1]$$ $$Q = [0,1,0,0]$$ Then $$KL(P||Q) = 0 + \ln(0.9/1)\cdot0.9 + 0 + 0 = -0.094$$ This shouldn't be possible from the Gibbs inequality. What am I misunderstanding?
3
votes
1answer
187 views

Distance between two Gaussian mixtures to evaluate cluster solutions

I'm running a quick simulation to compare different clustering methods, and currently hit a snag trying to evaluate the cluster solutions. I know of various validation metrics (many found in ...
3
votes
1answer
124 views

G-test statistic and KL divergence

According to Wikipedia, the G-test statistic is "proportional to the Kullback–Leibler divergence of the empirical distribution from the theoretical distribution." To get the relationship between $ G $ ...
3
votes
2answers
566 views

Is it possible to apply KL divergence between discrete and continuous distribution?

I am not a mathematician. I have searched the internet about KL Divergence. What I learned is the the KL divergence measures the information lost when we approximate distribution of a model with ...
2
votes
1answer
437 views

Symmetrised Kullback - Leibler divergence

I have trouble understanding KL divergence, where P is probability mass function of true distribution of data and Q is the approximation of P. The definition of KL divergence is: ...
3
votes
1answer
782 views

Kullback-Leibler divergence of two normal distributions

I was recently trying to find a way to compute the KL-divergence between 2 populations that are normally distributed using the mean and variance of each population. But I found several different ...
3
votes
1answer
110 views

Select best distance for feature selection

Suppose I have matrix $X \in R^{n \times m}$, where $n$ is the number of individuals and $m$ is the number of features and $X[i,j] \in \{0,1\}$; $1$ indicates that the individual $i$ has the feature ...
4
votes
1answer
3k views

KL divergence between two multivariate Gaussians

I'm having trouble deriving the KL divergence formula assuming two multivariate normal distributions. I've done the univariate case fairly easily. However, it's been quite a while since I took math ...
5
votes
2answers
3k views

How to calculate Kullback-Leibler divergence/distance?

I have three data sets X, Y and Z. Each data set defines the frequency of an event occurring. For example: Data Set X: E1:4, E2:0, E3:10, E4:5, E5:0, E6:0 and so on.. Data Set Y: E1:2, E2:3, E3:7, ...
-1
votes
2answers
837 views

Symmetric Kullback-Leibler divergence OR Mutual Information as a metric of distance between two distributions?

I need some metric of divergence of two distributions. (They are complex and don't fit with exponential family, normal, log-normal, power-law. Maybe some mixture of that, but I'm not feeling right ...