Tagged Questions

an asymmetric measure of distance (or dissimilarity) between probability distributions.

7 views

Evaluation metrics for kernel density estimation

Given a true (benchmark) density $p(x)$, and several density estimation algorithms, I want to empirically compare which one works better than the other. In this case, what kind of evaluation metric is ...
10 views

Derivation of autoencoders in backpropagation

I'm following the basics of autoencoders here: http://ufldl.stanford.edu/wiki/index.php/Autoencoders_and_Sparsity Here are some of the important parts: But I don't understand the last part: Why is ...
18 views

What is a multivariate random variable?independent vectors analysis?

I like to use the Kullback-Leibler divergence to define the contrast function for mutivariate random variables.How does it work? What does "multivariate components' means? What is a multivariate ...
25 views

50 views

68 views

What are the assumptions that we make when we compute KL Divergence between two distributions?

Let us assume that we compute the KL Divergence between p and q. Is it necessary that both p and q belong to the exponential family of distributions. Moreover, is it necessary, that both p and q ...
28 views

What tests can I use to compare these two probability distributions

I am trying to compare two one-dimensional distributions. I am using Kullback-Leibler divergence function for this but it requires me to have both the distributions of equal length. I am not sure how ...
35 views

163 views

Use of KL Divergence in practice

It's not symmetric, so it can't really be used as a distance metric. I suppose given two known distributions p(x) and q(x), if one found another distribution z(x) but knew it came from either p or q,...
42 views

Does the local triangle inequality holds for Kullback-Leibler divergence

Does the local triangle inequality holds for the Kullback-Leibler divergence? For the local triangle inequality, I mean the $$d(\theta', \theta) + d(\theta'', \theta) \geq A d(\theta', \theta'')$$ ...
50 views

Can you write a probability based on the relative entropy?

Suppose we have a graphical model $X\rightarrow \Theta \rightarrow D$ where all the distributions are Gaussian Mixture Models. Suppose further that the distribution of $X$ has more components than the ...
62 views

Kullback-Leibler Divergence for Graph Sampling

I am from Computer Science background and need to apply Kullback Leibler Divergence to find the divergence between two distributions of unknown types. Let's say I have a graph G(V,E) and I make a ...
46 views

Negative KL Divergence Values

I am computing KL divergence between two documents. I have got the tf-idf vectors for the top 5 features as follows: ...
62 views

Name of an $f$-divergence

The term divergence means a function $D$, which, given two probability distributions $P,Q$, assigns a non-negative real number $D(P,Q)$ such that $D(P,Q) = 0$ iff $P(x)=Q(x) \forall x$. The relative ...
79 views

Algorithm for approximating a density by a mixture density

Given a density $f(x)$ (e.g. the log-normal distribution or log-$t_{\nu=3}$ distribution), I was wondering what algorithm are known/typically used to find a mixture of distributions $g_r(x)$ from ...
23 views

Properties of the KL topology [reference request]

I'm trying to understand better what are the implications of a sequence of random variables $X_n$ converging toward some limit $X$ in the KL topology, ie the probability density functions are such ...
28 views

KL divergence and probabilities of 0 for P(i)

Why do probabilities of 0 for $P(i)$ not affect the result of the KL Divergence equation? Regardless of what probabilities we have for $Q(i)$, the product is 0. What are the benefits of this? Is ...
92 views

Hypothesis test based on entropy

I am reading the wikipedia page on hypothesis testing, but a I can't find any reference to tests based on entropy. Which are good hypothesis tests based on entropy or quantities derived from it?
181 views

variational inference with KL

i am self-studying variational inference - and in Murphy's book "A probabilistic perspective on machine learning" it is discussed that minimizing the forward KL divergence (which is stated to be zero-...
44 views

How to use KL-divergence in naive bayes classifier to weight features?

I have a dataset consisting of 4 classes. I have implemented the Gaussian Naive Classifier (in Matlab). In the training phase I calculate the mean and variance for each feature and each class as well ...
23 views

61 views

KL-divergence as a negative log likelihood for exponential families

I am reading Distributed Estimation, Information Loss and Exponential Families, where the authors consider and compare two estimators for $\theta$ in the parametric model $p(x\mid\theta)$: the ...
60 views

f(y | x) or f(y,x) in regression and MLE

In $Y = aX + b + \epsilon$ where $\epsilon$ ~ $N(0,\sigma^2)$ and i.i.d regression setting If X is stochastic and $E(\epsilon\mid X) =0$, then which one is correct: (1) \$f(x,y) = \frac{1}{\sigma\...
215 views

Relationship between Poisson generation and generalized Kullback-Leibler divergence

I have read that, in the context of matrix factorization, performing maximum likelihood estimation under the assumption that the entries are Poisson generated is equivalent to minimizing the ...