The tag has no wiki summary.

learn more… | top users | synonyms

1
vote
0answers
23 views

Occupancy octree metrics (Kullback-Leibler)

As I'm currently working on scan matching for outdoor environments I was wondering about the best metric to compare two occupancy octrees (one resulted from the scan matching and one ground truth ...
0
votes
3answers
75 views

Analysis of Kullback-Leibler divergence

Let us consider the following two probability distributions ...
1
vote
0answers
51 views

Neuroscience Equations

I am trying to understand a neuroscience article by Karl Friston. In it he gives three equations that are, as I understand him, equivalent or inter-convertertable and refer to both physical and ...
0
votes
1answer
55 views
1
vote
0answers
54 views

Under what conditions will Kullback-Leibler divergence/mutual information be infinity?

For two perfectly correlated Gaussian variables, the mutual information between them, and thus the KL divergence between the product of the marginal distributions and the joint distribution, is ...
1
vote
0answers
33 views

How to estimate similarity between several probability distributions?

I have several set of probability distributions. I want to reliably estimate consistency across distributions inside each set. Literature contains methods to compare two distributions: ...
0
votes
0answers
28 views

sufficient statistic and KL-divergence: Confusion with an equation

I am reading a paper, which talks about minimising KL-divergence of any arbitrary distribution over a family of exponential distribution. So, given a distribution $p$, we want to compute its ...
1
vote
0answers
24 views

computing KL divergence: M projections for arbitrary distributions

Background I have a generative model for a process that can be described as follows: $$ y = t(x, w) + e $$ where $x$ and $y$ observations of a set of random variables which are related by a ...
0
votes
0answers
32 views

Kullback-Leibler divergence Pareto Distribution

What is the Kullback-Leibler divergence for a Pareto Distribution? Given $p(x)$ = $ \alpha$ $\frac{x^{\alpha}_{min,1}}{x^{a+1}}$.
6
votes
3answers
516 views

Calculate the Kullback-Leibler Divergence in practice?

I am using KL Divergence as a measure of dissimilarity between 2 $p.m.f.$ $P$ and $Q$. $$D_{KL}(P||Q) = \sum_{i=1}^N \ln \left( \frac{P_i}{Q_i} \right) P_i$$ $$=-\sum P(X_i)ln\left(Q(X_i)\right) + ...
3
votes
1answer
113 views

KL divergence between an uninformative (?) Gaussian and a Gaussian

I have to calculate the KL divergence between a distribution $q$ and a prior distribution $p$, both of which are univariate Gaussians, i.e. $KL(q|p), q \sim \mathcal{N}(\mu, \sigma^2), p \sim ...
1
vote
0answers
57 views

Problem with Kullback–Leibler divergence criteria

I am using Kullback–Leibler divergence criteria for comparing my estimation and true density functions, but I have zero value on my estimation function when I have a testing set of size 10000, mostly ...
10
votes
2answers
221 views

What is the relationship between the GINI score and the log-likelihood ratio

I am studying classification and regression trees, and one of the measures for the split location is the GINI score. Now I am used to determining best split location when the log of the likelihood ...
0
votes
0answers
97 views

Kullback-Liebler Divergence function in R package “FNN” returns NAN for certain data? Why?

I am using the "FNN" package to calculate the Kullback-Liebler Divergence between two numeric vectors: require(FNN) load("~/xy.RData") KL.divergence(x,y) (Link ...
2
votes
0answers
44 views

Quantify the information lost given by the Kullback-Leibler divergence measure

Consider there are $N$ individuals and these measure a quantity $X\in \mathbb{R}^{N\times M}$ where $M$ is the number of measurements and let $P(X)$ denote a probability distribution over $X$. The ...
1
vote
1answer
80 views

How to test whether a series data follow Ornstein-Uhlenbeck process (OU process)?

I have some measures which seems to have some mean-reverting properties and I'm wondering whether they can be modeled as Ornstein-Uhlenbeck process (OU process). And actually I quite expect it because ...
0
votes
0answers
23 views

How to use Kullback–Leibler divergence to help me choose the best distribution function?

I am now using Exponential distribution to model the the time intervals of a sequence of random events.Since I can choose several different lamdas for this model , I want to find out which lamda of ...
1
vote
1answer
52 views

KL divergence minimisation equation

I am looking at some literature on KL divergence minimisation and am having trouble understanding the derivation of the second order moment. So, if we have a distribution from the exponential family, ...
1
vote
0answers
24 views

Modeling a list with a tunable degree of disorder/shuffling

Imagine we have a list of ordered numbers $L = (1, 2,\dots, N)$. I want to add an arbitrary amount of "disorder" to that list. For instance: Adding a little bit of disorder would permute a few ...
1
vote
1answer
54 views

Renyi divergence identity

I'm reading the paper, T. van Erven and P. Harremoës, Rényi Divergence and Kullback-Leibler Divergence, arXiv 1206.2459 on the Renyi divergence, and I'm trying to make sense of "Example 1". I think ...
1
vote
0answers
41 views

Formal statistical test for comparing likelihood distributions obtained via MCMC

I am trying to formally compare the distribution of the likelihood values generated using two different models with marginal posterior values of the parameters obtained using MCMC in order to assess ...
2
votes
0answers
140 views

Disadvantages of the Kullback-Leibler divergence

I'm working on a calibration problem which involves the usage of the Kullback-Leibler divergence as an error between some empirical distribution $p$ and a theoretical distribution $q$. In the model, ...
2
votes
0answers
66 views

Does relative Kullback-Leibler divergence exist?

Suppose I have two multivariate normal distributions. I have computed the KL divergence ($d_{LK}(N_1, N_2)$). Is there a way to measure a relative divergence between these two distributions? For ...
3
votes
1answer
340 views

KL-divergence between two categorical/multinomial distributions gives negative values?

If $$P = [0,0.9,0,0.1]$$ $$Q = [0,1,0,0]$$ Then $$KL(P||Q) = 0 + \ln(0.9/1)\cdot0.9 + 0 + 0 = -0.094$$ This shouldn't be possible from the Gibbs inequality. What am I misunderstanding?
3
votes
1answer
132 views

Distance between two Gaussian mixtures to evaluate cluster solutions

I'm running a quick simulation to compare different clustering methods, and currently hit a snag trying to evaluate the cluster solutions. I know of various validation metrics (many found in ...
3
votes
1answer
83 views

G-test statistic and KL divergence

According to Wikipedia, the G-test statistic is "proportional to the Kullback–Leibler divergence of the empirical distribution from the theoretical distribution." To get the relationship between $ G $ ...
3
votes
2answers
339 views

Is it possible to apply KL divergence between discrete and continuous distribution?

I am not a mathematician. I have searched the internet about KL Divergence. What I learned is the the KL divergence measures the information lost when we approximate distribution of a model with ...
2
votes
1answer
267 views

Symmetrised Kullback - Leibler divergence

I have trouble understanding KL divergence, where P is probability mass function of true distribution of data and Q is the approximation of P. The definition of KL divergence is: ...
2
votes
1answer
426 views

Kullback-Leibler divergence of two normal distributions

I was recently trying to find a way to compute the KL-divergence between 2 populations that are normally distributed using the mean and variance of each population. But I found several different ...
3
votes
1answer
106 views

Select best distance for feature selection

Suppose I have matrix $X \in R^{n \times m}$, where $n$ is the number of individuals and $m$ is the number of features and $X[i,j] \in \{0,1\}$; $1$ indicates that the individual $i$ has the feature ...
4
votes
1answer
2k views

KL divergence between two multivariate Gaussians

I'm having trouble deriving the KL divergence formula assuming two multivariate normal distributions. I've done the univariate case fairly easily. However, it's been quite a while since I took math ...
4
votes
2answers
2k views

How to calculate Kullback-Leibler divergence/distance?

I have three data sets X, Y and Z. Each data set defines the frequency of an event occurring. For example: Data Set X: E1:4, E2:0, E3:10, E4:5, E5:0, E6:0 and so on.. Data Set Y: E1:2, E2:3, E3:7, ...
-1
votes
2answers
369 views

Symmetric Kullback-Leibler divergence OR Mutual Information as a metric of distance between two distributions?

I need some metric of divergence of two distributions. (They are complex and don't fit with exponential family, normal, log-normal, power-law. Maybe some mixture of that, but I'm not feeling right ...
4
votes
3answers
910 views

Is the square root of the symmetric Kullback-Leibler divergence a metric?

It is well known that the square root of the Jensen-Shannon divergence is a true metric, but how about the square root of symmetric KL: D(P||Q)+D(Q||P)? I have reasons to believe that it also is a ...
1
vote
0answers
24 views

Is there any meta-approach for variable selection based of measures of similarity between each two variables?

Is there any meta-approach ( or mayby I should say universal approach which works with different measures ) for variable selection which is based on similarity matrix which every entry ...
2
votes
0answers
153 views

Comparing divergence from uniform distributions with differing supports (discrete)

Imagine we have a potentially biased coin and a potentially biased six-sided die and we want to know which is more biased than the other. Firstly, is this a reasonable goal? Could it make sense to ...
0
votes
2answers
811 views

Bhattacharyya distance for histograms

One of the ways to measure the similarity of two discrete probability distributions is the Bhattacharyya distance. In computer vision, for example, it is used to evaluate the degree of similarity ...
4
votes
0answers
83 views

Orthogonal intersection in a Riemannian manifold

Let $S$ be the set of all probability distributions on $\mathbb{R}$ and $S_n=\{p_\theta\}$ be an $n$ dimensional submanifold of parameterized family of probability distributions on $\mathbb{R}$ where ...
3
votes
1answer
136 views

Orthogonal intersection of linear family and exponential family

I asked the following question in MSE for which I couldn't get any answer yet. I thought this would be a better place for that question. In statistical maniolds ...
5
votes
2answers
219 views

Connection between Fisher metric and the relative entropy

Can someone prove the following connection between Fisher information metric and the relative entropy (or KL divergence) in a purely mathematical rigorous way? $$D( p(\cdot , a+da) \| p(.,a) ) ...
1
vote
1answer
334 views

Calculating Hellinger Divergence from Results of Kernel Density Estimates in Matlab

Using the ksdensity function in matlab returns a density estimation in the form of 2 vectors f and xi. Where f are the density values and xi the corresponding points for the density values. How do I ...
2
votes
0answers
195 views

Kullback-Leibler vs Hellinger Distance

I am working on this problem in which I have a dataset of n-dimensional examples that come from different and unknown distributions. Given a new sample, I wish to find k examples from the dataset that ...
4
votes
0answers
126 views

Estimating parameters using Kullback-Leibler or Kolmogorov-Smirnoff via Nelder-Mead

I want to find the parameters of a model which specifies a set of classification probabilities, for say M classes. (I'll use the parameters in another model later.) Given a set of parameters ...
0
votes
0answers
241 views

Logistic regression, loss function and KL divergence

In decision theory, a loss function signature is supposed to be output space * output space -> error There seems to be many different definition of 'the ...
1
vote
0answers
139 views

On the uniform convergence of relative frequencies of events to their probabilities

I have read the article by Vapnik, Chervonenkis "On the uniform convergence of relative frequencies of events to their probabilities" Theory of Probability and Its Applications, vol XVI, n. , 1971. ...
3
votes
1answer
1k views

Kullback-Leibler divergence: negative values?

Wikipedia - KL properties says that KL can never be negative. But e.g. for texts where the probabilities are very small I somehow get negative values? E.g. Collection A: - word count: 321 doc ...
2
votes
0answers
222 views

KL divergence between 2 distributions with unequal cardinalities?

Say $X$ is a discrete random variable with cardinality $|X|$ and $Y$ is a discrete random variable with cardinality $|Y|$. Does it make sense to talk about the KL divergences $D_{KL}(X||Y)$ or ...
3
votes
0answers
138 views

Sample distribution for Kullback-Leibler distance

For two $n$ dimensional multivariate normal distributions $X_{1}\sim N\left(\mu_{1},\Sigma_{1}\right)$ and $X_{2}\sim N\left(\mu_{2},\Sigma_{2}\right)$, the Kullback-Leibler distance is given by ...
3
votes
0answers
62 views

Lomax distributions - Kullback Leibler divergence

Does anyone know of a reference for an expression for the Kullback-Leibler divergence between two Lomax (Pareto II) distributions? Not really worried which way the Lomax is parameterized.
1
vote
1answer
242 views

Multivariate time series model evaluation with conditional moments

Consider multivariate time series models that estimate potentially time-varying conditional means, variances, and correlations (one type of model might be a VAR(p)+Garch(1,1)+DCC Gaussian Copula ...