The tag has no wiki summary.

learn more… | top users | synonyms

0
votes
0answers
26 views

Kullback-Liebler Divergence function in R package “FNN” returns NAN for certain data? Why?

I am using the "FNN" package to calculate the Kullback-Liebler Divergence between two numeric vectors: require(FNN) load("~/xy.RData") KL.divergence(x,y) (Link ...
2
votes
0answers
20 views

Quantify the information lost given by the Kullback-Leibler divergence measure

Consider there are $N$ individuals and these measure a quantity $X\in \mathbb{R}^{N\times M}$ where $M$ is the number of measurements and let $P(X)$ denote a probability distribution over $X$. The ...
0
votes
0answers
32 views

How to test whether a series data follow Ornstein-Uhlenbeck process (OU process)?

I have some measures which seems to have some mean-reverting properties and I'm wondering whether they can be modeled as Ornstein-Uhlenbeck process (OU process). And actually I quite expect it because ...
0
votes
0answers
16 views

How to use Kullback–Leibler divergence to help me choose the best distribution function?

I am now using Exponential distribution to model the the time intervals of a sequence of random events.Since I can choose several different lamdas for this model , I want to find out which lamda of ...
1
vote
1answer
43 views

KL divergence minimisation equation

I am looking at some literature on KL divergence minimisation and am having trouble understanding the derivation of the second order moment. So, if we have a distribution from the exponential family, ...
1
vote
0answers
20 views

Modeling a list with a tunable degree of disorder/shuffling

Imagine we have a list of ordered numbers $L = (1, 2,\dots, N)$. I want to add an arbitrary amount of "disorder" to that list. For instance: Adding a little bit of disorder would permute a few ...
1
vote
1answer
42 views

Renyi divergence identity

I'm reading the paper, T. van Erven and P. Harremoës, Rényi Divergence and Kullback-Leibler Divergence, arXiv 1206.2459 on the Renyi divergence, and I'm trying to make sense of "Example 1". I think ...
1
vote
0answers
33 views

Formal statistical test for comparing likelihood distributions obtained via MCMC

I am trying to formally compare the distribution of the likelihood values generated using two different models with marginal posterior values of the parameters obtained using MCMC in order to assess ...
2
votes
0answers
91 views

Disadvantages of the Kullback-Leibler divergence

I'm working on a calibration problem which involves the usage of the Kullback-Leibler divergence as an error between some empirical distribution $p$ and a theoretical distribution $q$. In the model, ...
2
votes
0answers
56 views

Does relative Kullback-Leibler divergence exist?

Suppose I have two multivariate normal distributions. I have computed the KL divergence ($d_{LK}(N_1, N_2)$). Is there a way to measure a relative divergence between these two distributions? For ...
2
votes
1answer
193 views

KL-divergence between two categorical/multinomial distributions gives negative values?

If $$P = [0,0.9,0,0.1]$$ $$Q = [0,1,0,0]$$ Then $$KL(P||Q) = 0 + \ln(0.9/1)\cdot0.9 + 0 + 0 = -0.094$$ This shouldn't be possible from the Gibbs inequality. What am I misunderstanding?
2
votes
1answer
84 views

Distance between two Gaussian mixtures to evaluate cluster solutions

I'm running a quick simulation to compare different clustering methods, and currently hit a snag trying to evaluate the cluster solutions. I know of various validation metrics (many found in ...
2
votes
1answer
66 views

G-test statistic and KL divergence

According to Wikipedia, the G-test statistic is "proportional to the Kullback–Leibler divergence of the empirical distribution from the theoretical distribution." To get the relationship between $ G $ ...
3
votes
2answers
204 views

Is it possible to apply KL divergence between discrete and continuous distribution?

I am not a mathematician. I have searched the internet about KL Divergence. What I learned is the the KL divergence measures the information lost when we approximate distribution of a model with ...
2
votes
1answer
205 views

Symmetrised Kullback - Leibler divergence

I have trouble understanding KL divergence, where P is probability mass function of true distribution of data and Q is the approximation of P. The definition of KL divergence is: ...
2
votes
1answer
239 views

Kullback-Leibler divergence of two normal distributions

I was recently trying to find a way to compute the KL-divergence between 2 populations that are normally distributed using the mean and variance of each population. But I found several different ...
3
votes
1answer
97 views

Select best distance for feature selection

Suppose I have matrix $X \in R^{n \times m}$, where $n$ is the number of individuals and $m$ is the number of features and $X[i,j] \in \{0,1\}$; $1$ indicates that the individual $i$ has the feature ...
0
votes
0answers
32 views

Measure stability of distribution over time

I have the following thing I want to investigate: Given a system where users add resources, I want to concretely investigate whether this system stabilizes over time. This would mean that most users ...
3
votes
1answer
941 views

KL divergence between two multivariate Gaussians

I'm having trouble deriving the KL divergence formula assuming two multivariate normal distributions. I've done the univariate case fairly easily. However, it's been quite a while since I took math ...
3
votes
1answer
650 views

How to calculate Kullback-Leibler divergence/distance?

I have three data sets X, Y and Z. Each data set defines the frequency of an event occurring. For example: Data Set X: E1:4, E2:0, E3:10, E4:5, E5:0, E6:0 and so on.. Data Set Y: E1:2, E2:3, E3:7, ...
0
votes
1answer
139 views

Symmetric Kullback-Leibler divergence OR Mutual Information as a metric of distance between two distributions?

I need some metric of divergence of two distributions. (They are complex and don't fit with exponential family, normal, log-normal, power-law. Maybe some mixture of that, but I'm not feeling right ...
4
votes
3answers
729 views

Is the square root of the symmetric Kullback-Leibler divergence a metric?

It is well known that the square root of the Jensen-Shannon divergence is a true metric, but how about the square root of symmetric KL: D(P||Q)+D(Q||P)? I have reasons to believe that it also is a ...
1
vote
0answers
21 views

Is there any meta-approach for variable selection based of measures of similarity between each two variables?

Is there any meta-approach ( or mayby I should say universal approach which works with different measures ) for variable selection which is based on similarity matrix which every entry ...
2
votes
0answers
111 views

Comparing divergence from uniform distributions with differing supports (discrete)

Imagine we have a potentially biased coin and a potentially biased six-sided die and we want to know which is more biased than the other. Firstly, is this a reasonable goal? Could it make sense to ...
0
votes
2answers
631 views

Bhattacharyya distance for histograms

One of the ways to measure the similarity of two discrete probability distributions is the Bhattacharyya distance. In computer vision, for example, it is used to evaluate the degree of similarity ...
4
votes
0answers
76 views

Orthogonal intersection in a Riemannian manifold

Let $S$ be the set of all probability distributions on $\mathbb{R}$ and $S_n=\{p_\theta\}$ be an $n$ dimensional submanifold of parameterized family of probability distributions on $\mathbb{R}$ where ...
3
votes
1answer
125 views

Orthogonal intersection of linear family and exponential family

I asked the following question in MSE for which I couldn't get any answer yet. I thought this would be a better place for that question. In statistical maniolds ...
4
votes
2answers
168 views

Connection between Fisher metric and the relative entropy

Can someone prove the following connection between Fisher information metric and the relative entropy (or KL divergence) in a purely mathematical rigorous way? $$D( p(\cdot , a+da) \| p(.,a) ) ...
1
vote
1answer
213 views

Calculating Hellinger Divergence from Results of Kernel Density Estimates in Matlab

Using the ksdensity function in matlab returns a density estimation in the form of 2 vectors f and xi. Where f are the density values and xi the corresponding points for the density values. How do I ...
2
votes
0answers
173 views

Kullback-Leibler vs Hellinger Distance

I am working on this problem in which I have a dataset of n-dimensional examples that come from different and unknown distributions. Given a new sample, I wish to find k examples from the dataset that ...
4
votes
0answers
120 views

Estimating parameters using Kullback-Leibler or Kolmogorov-Smirnoff via Nelder-Mead

I want to find the parameters of a model which specifies a set of classification probabilities, for say M classes. (I'll use the parameters in another model later.) Given a set of parameters ...
0
votes
0answers
191 views

Logistic regression, loss function and KL divergence

In decision theory, a loss function signature is supposed to be output space * output space -> error There seems to be many different definition of 'the ...
1
vote
0answers
123 views

On the uniform convergence of relative frequencies of events to their probabilities

I have read the article by Vapnik, Chervonenkis "On the uniform convergence of relative frequencies of events to their probabilities" Theory of Probability and Its Applications, vol XVI, n. , 1971. ...
3
votes
1answer
969 views

Kullback-Leibler divergence: negative values?

Wikipedia - KL properties says that KL can never be negative. But e.g. for texts where the probabilities are very small I somehow get negative values? E.g. Collection A: - word count: 321 doc ...
2
votes
0answers
203 views

KL divergence between 2 distributions with unequal cardinalities?

Say $X$ is a discrete random variable with cardinality $|X|$ and $Y$ is a discrete random variable with cardinality $|Y|$. Does it make sense to talk about the KL divergences $D_{KL}(X||Y)$ or ...
2
votes
0answers
120 views

Sample distribution for Kullback-Leibler distance

For two $n$ dimensional multivariate normal distributions $X_{1}\sim N\left(\mu_{1},\Sigma_{1}\right)$ and $X_{2}\sim N\left(\mu_{2},\Sigma_{2}\right)$, the Kullback-Leibler distance is given by ...
3
votes
0answers
55 views

Lomax distributions - Kullback Leibler divergence

Does anyone know of a reference for an expression for the Kullback-Leibler divergence between two Lomax (Pareto II) distributions? Not really worried which way the Lomax is parameterized.
1
vote
1answer
223 views

Multivariate time series model evaluation with conditional moments

Consider multivariate time series models that estimate potentially time-varying conditional means, variances, and correlations (one type of model might be a VAR(p)+Garch(1,1)+DCC Gaussian Copula ...
4
votes
2answers
239 views

Maximum entropy sampler

I want to sample from a distribution which has fixed to a given values mean(=0), standard deviation(=1), skewness(=0) and kurtosis. I also want this distribution to be as general as possible, i.e. to ...
2
votes
1answer
359 views

Markov chain convergence, total variation and KL divergence

I have a few related questions regarding the convergence of continuous-state Markov chains. The theorems that I found claim that Markov chains converge in total variation if they are ...
2
votes
0answers
157 views

Cramer-Rao type bound for Information Gain

I am interested in the Bayes risk of some distribution $\pi$ $$ r(\pi) = \mathbb{E}_{\pi(x)}[ \mathbb{E}_{\Pr(y|d,x)}[L(x,\hat x(y|d))]], $$ where $L$ is some loss function and $\hat x$ is the ...
4
votes
1answer
763 views

Kullback-Leibler divergence

Suppose we seek to approximate an arbitrary distribution $p_1(x)$ by a normal $p_2(x) \sim \mathcal N(\mu, \Sigma)$. How can I show that the values that lead to the smallest Kullback–Leibler ...
3
votes
1answer
170 views

Property of KL-divergence

Let $p_1$ and $p_2$ be two distinct probability distributions. Define $$ L(q)=D(q||p_1)-D(q||p_2) $$ where $D$ is the usual Kullback-Leibler divergence. Assume the support of $p_2$ is included in ...
2
votes
1answer
413 views

Kullback–Leibler divergence between two Wishart distributions

The result is shown in: [1] W.D. Penny, KL-Divergences of Normal, Gamma, Dirichlet, and Wishart densities, Available at: www.fil.ion.ucl.ac.uk/~wpenny/publications/densities.ps But could anyone help ...
1
vote
1answer
103 views

How does one express the decrease in minimal type II error bound for each observation added?

Problem: I have a "classifier" that uses some arbitrary hypothesis test on observations from one of two known probability distributions: $P_0$ (null hypothesis $H_0$) is a zero-mean Gaussian ...
7
votes
2answers
1k views

Hypothesis testing and total variation distance vs. Kullback-Leibler divergence

In my research I have run into the following general problem: I have two distributions $P$ and $Q$ over the same domain, and a large (but finite) number of samples from those distributions. Samples ...
0
votes
0answers
137 views

Shall I use sqrt of J-S divergence or K-L divergence as a measure of deviation from uniform pdf?

Some colleagues are using 'normalized' K-L divergence to measure deviation of unit-area histogram (pseudo discrete pdf) from corresponding uniform distribution ($N$ equal-length bins). Maybe it is ...
3
votes
2answers
307 views

Similarity / dissimilarity of two large bimodal datasets

I am interested in assessing the divergence, or similarity or dissimilarity of 2 datasets that are the results of 2 different lidar instrument measurements. Each dataset has over 90,000 values and ...
6
votes
3answers
4k views

Measure of similarity or distance between two symmetric covariance matrices

are there any measures of similarity or distance between two symmetric covariance matrices (both having the same dimension)? I am thinking here of analogues to KL divergence of two probability ...
5
votes
3answers
2k views

How to compute the Kullback-Leibler divergence when the PMF contains 0s?

I have the following timeseries obtained using the data posted below. For a sliding window size of 10, I am trying to compute the KL-divergence between the PMF of values within the current sliding ...