Questions tagged [information-geometry]

The geometrical analysis of probability spaces

Filter by
Sorted by
Tagged with
4 votes
0 answers
72 views

Intuition/meaning of information geometry distances and geodesics?

In information geometry, we consider a manifold of probability distributions, together with the Fisher Information metric (given by the Fisher Information matrix). I have some intuition (see ...
dherrera's user avatar
  • 1,258
0 votes
0 answers
28 views

Fisher information reparametrization formula in non-bijective cases

The formula writes as this: $$I(X) = \left(\frac{\partial Z}{\partial X}\right)^\top I(Z) \left(\frac{\partial Z}{\partial X}\right),$$ in particular, if $Z = AX$, we have $$I(X) = A^\top I(Z) A.$$ ...
metricspace's user avatar
3 votes
0 answers
149 views

Ay (2017) vs Amari (2016) for information geometry

I am interested in learning information geometry in detail with a focus on applications, and am currently considering two main texts: Amari (2016)'s "Information Geometry and Its Applications&...
1 vote
1 answer
123 views

How does computing the bias of model parameters make sense?

I've been studying statistics recently and was thinking about the fact that computing the expectation of a random variable $E(X)$ only really makes sense if $X$ is a random variable defined over a ...
puzzleshark's user avatar
1 vote
0 answers
123 views

How to learn information geometry from a statistical background [closed]

I have started learning information geometry from the book Differential geometry and Statistics by M.K.murray and J.W. Rice.But I am facing huge difficulties in understanding mathematics behind it. I ...
Avishek Dutta's user avatar
2 votes
0 answers
85 views

Book recommendation: which ones should I study in order to learn information geometry, copulas and everything in between?

So far I have been studying the theory of copulas for a while and I became fascinated by it. More recently, I read about information geometry which also caught my attention. Having said that, could ...
5 votes
1 answer
1k views

Difference of two KL-divergence

The Kullback-Leibler (KL) divergence between two distributions $P$ and $Q$ is defined as $$\mbox{KL}(P \| Q) = \mathbb{E}_P\left[\ln \frac{\mbox{d}P}{\mbox{d}Q}\right].$$ My question is that suppose ...
Matics's user avatar
  • 301
3 votes
1 answer
283 views

A divergence that can be extended to logistic functions?

If I have data $\{(x_i, y_i)\}_{i=1}^n$ where the dependent variable is binary $(y_i = 0,1)$ I can model it using a logistic function: $$f(x; \alpha, \beta) = \frac{1}{1 + e^{-(\alpha + \beta x)}}$$ ...
Jason d'Eon's user avatar
4 votes
0 answers
159 views

Fisher information matrix and gradients

I'm a math Ph.D. without formal training in statistics. Quite a few papers on normalization methods in deep learning mention the Fisher information matrix and how it's related to the Riemannian metric ...
del's user avatar
  • 171
2 votes
0 answers
231 views

Diffusion tensor as a covariance matrix

TLDR: In nuclear magnetic resonance (NMR), to study molecular diffusion we assume that molecules displace in 3D space according to a trivariate gaussian distribution. The variables are then the ...
user241848's user avatar
2 votes
0 answers
63 views

Is the Franke-Wolfe algorithm the same as Manifold optimization?

The Frank-Wolfe optimization algorithm describes optimization over a constrained domain. In the Manifold Optimization literature (e.g. [1]) a Gradient-Step is done using an exponential map. This maps ...
JohnRos's user avatar
  • 5,684
6 votes
1 answer
117 views

Is MLE intrinsically connected to logs?

My mathematical exploration led me the following claim: Claim: MLE is fundamentally connected to logs (and KL divergence, which also uses logs). It’s not correct to say log shows up simply to make ...
Yatharth Agarwal's user avatar
4 votes
1 answer
2k views

How to picture EM algorithm and KL-divergence geometrically?

In reading up on the Expectation-Maximization algorithm on Wikipedia, I read this short and intriguing line, under the subheading "Geometric Intuition": In information geometry, the E step ...
Apprentice's user avatar
5 votes
1 answer
1k views

KL Divergence, Bregman, and uniqueness

While reading the following paper on Bregman Divergence (link) Banerjee, Arindam, et al. "Clustering with Bregman divergences." Journal of machine learning research 6.Oct (2005): 1705-1749. In ...
sheppa28's user avatar
  • 1,636
6 votes
1 answer
3k views

What is the most beginner-friendly book for information geometry?

Question: What is the most beginner-friendly book for information geometry? The book: Amari and Nagaoka, Methods of Information Geometry, is often mentioned as a reference for information ...
Chill2Macht's user avatar
  • 6,259
7 votes
1 answer
432 views

Most accessible introduction to directional statistics?

Question: Does anyone have a recommendation for a reference which is "the most accessible" introduction to directional statistics? When I say "accessible", I mean that many authors are so experienced ...
7 votes
1 answer
791 views

Information geometry tutorial

I would like to know more about the topic of information geometry, but i don't want to delve deeply into it. The resources i found was either too complicated or just very messy presentations like this ...
15 votes
2 answers
3k views

Graphical intuition of statistics on a manifold

On this post, you can read the statement: Models are usually represented by points $\theta$ on a finite dimensional manifold. On Differential Geometry and Statistics by Michael K Murray and ...
Antoni Parellada's user avatar
4 votes
1 answer
347 views

Coefficient of variation of correlation values

I have the following problem that I am not sure of whether I treat it correctly: I run repeated simulations, where the output of a single simulation replicate is a single correlation value. The ...
PScho's user avatar
  • 41
0 votes
1 answer
101 views

What is Geometrical Probability? [closed]

Is it a technique where Geometry is used to solve probabilistic problems? Is it a kind of probability which grows Geometrically when we conduct experiments? Is it a kind of distribution? I am ...
user366312's user avatar
  • 2,088
4 votes
2 answers
2k views

KL divergence vs Absolute Difference between two distributions? [duplicate]

Why should I use KL divergence over just giving the abs difference from two PDFs?
user avatar
2 votes
2 answers
2k views

Cramer-Rao Lower Bound for the estimation of Pearson correlation

Given a bivariate Gaussian distribution $\mathcal{N}\left(0,\begin{pmatrix} 1 & \rho \\ \rho & 1 \end{pmatrix}\right)$, I am looking for information on the distribution of $\hat{\rho}$ ...
mic's user avatar
  • 4,328
23 votes
0 answers
1k views

Is there a general expression for ancillary statistics in exponential families?

An i.i.d sample $X_1,\dots,X_n$ from a scale family with c.d.f. $F(\frac{x}{\sigma})$ has $S(X)$ as an ancillary statistic if $S(X)$ depends on the sample only through $\frac{X_1}{X_n},\cdots,\frac{X_{...
Henry.L's user avatar
  • 2,470
1 vote
0 answers
44 views

classification real vs. made-up words

I am interesting in building a classifier that can separates made-up words (such as brands) from real words (belonging to the English dictionary for example). I have tried using a Soundex ...
mic's user avatar
  • 4,328
4 votes
0 answers
151 views

A Fisher information metric which doesn't refer to any exponential family

If $\mathcal E$ is some exponential family of distributions, then we can view it as a Riemannian manifold with local metric the Fisher information matrix. We can then define the Fisher metric: the ...
Guillaume Dehaene's user avatar
1 vote
0 answers
66 views

Request for literature on applications of manifold-valued random variables in medical imaging

I'd appreciate some references/literature on the applications of manifold-valued random variables, i.e., random variables $X:\Omega\to M$, where M is a manifold (could be even infinite dimensional), ...
Mathmath's user avatar
  • 761
16 votes
1 answer
5k views

Determinant of Fisher information

(I posted a similar question on math.se.) In information geometry, the determinant of the Fisher information matrix is a natural volume form on a statistical manifold, so it has a nice geometrical ...
geodude's user avatar
  • 331
11 votes
1 answer
475 views

Clarification in information geometry

This question is concerned with the paper Differential Geometry of Curved Exponential Families-Curvatures and Information Loss by Amari. The text goes as follows. Let $S^n=\{p_{\theta}\}$ be an $n$-...
Ashok's user avatar
  • 1,181
4 votes
0 answers
1k views

Are there efficient estimators for the variance of an exponential family?

Let us consider the Gaussian model $\mathcal{N}(\mu,\sigma^2)$, where both $\mu$ and $\sigma$ are unknown. I have learnt that (for example, from Amari's information geometry book) the exponential ...
Kumara's user avatar
  • 617
25 votes
3 answers
11k views

Does differential geometry have anything to do with statistics?

I am doing master in statistics and I am advised to learn differential geometry. I would be happier to hear about statistical applications for differential geometry since this would make me motivated. ...
LaTeXFan's user avatar
  • 1,306
7 votes
0 answers
290 views

Reference Request: Information Geometry for Ridge Regression

I am reading the book "regression estimators" by Gruber 2010 where he uses this technique to compare Ridge Regressors, however he concentrates on deriving the mathematical results without ...
Baz's user avatar
  • 1,763
8 votes
0 answers
166 views

Orthogonal intersection in a Riemannian manifold

Let $S$ be the set of all probability distributions on $\mathbb{R}$ and $S_n=\{p_\theta\}$ be an $n$ dimensional submanifold of parameterized family of probability distributions on $\mathbb{R}$ where $...
Kumara's user avatar
  • 617
4 votes
1 answer
578 views

Orthogonal intersection of linear family and exponential family

I asked the following question in MSE for which I couldn't get any answer yet. I thought this would be a better place for that question. In statistical maniolds $S=\{p_\theta\}$,$\theta=(\theta_1,\...
Kumara's user avatar
  • 617
5 votes
3 answers
628 views

References for use of symplectic geometry in statistics?

I have heard that many problems in Mathematical Statistics can be stated and solved in terms of Symplectic Geometry. Unfortunately this was a pretty vague statement and I am interested in something ...
adhalanay's user avatar
  • 153
14 votes
3 answers
768 views

Using information geometry to define distances and volumes…useful?

I came across a large body of literature which advocates using Fisher's Information metric as a natural local metric in the space of probability distributions and then integrating over it to define ...
Yaroslav Bulatov's user avatar
6 votes
2 answers
4k views

Orthogonal parametrization

In general inference, why orthogonal parameters are useful, and why is it worth trying to find a new parametrization that makes the parameters orthogonal ? I have seen some textbook examples, not so ...
Alekk's user avatar
  • 549
14 votes
4 answers
10k views

How to interpret KL divergence quantitatively?

I am comparing two distributions with KL divergence which returns me a non-standardized number that, according to what I read about this measure, is the amount of information that is required to ...
Ampleforth's user avatar