# Clarification in information geometry

This question is concerned with the paper Differential Geometry of Curved Exponential Families-Curvatures and Information Loss by Amari.

The text goes as follows.

Let $S^n=\{p_{\theta}\}$ be an $n$-dimensional manifold of probability distributions with a coordinate system $\theta=(\theta_1,\dots,\theta_n)$, where $p_{\theta}(x)>0$ is assumed...

We may regard every point $\theta$ of $S^n$ as carrying a function $\log p_{\theta}(x)$ of $x$...

Let $T_{\theta}$ be the tangent space of $S^n$ at $\theta$, which is, roughly speaking, identified with a linearized version of a small neighborhood of $\theta$ in $S^n$. Let $e_i(\theta), i=1,\dots,n$ be the natural basis of $T_{\theta}$ associated with the coordinated system...

Since each point $\theta$ of $S^n$ carries a function $\log p_{\theta}(x)$ of $x$, it is natural to regard $e_i(\theta)$ at $\theta$ as representing the function $$e_i(\theta)=\frac{\partial}{\partial\theta_i}\log p_{\theta}(x).$$

I don't understand the last statement. This appears in section 2 of the above mentioned paper. How's the basis of the tangent space are given by the above equation? It would be helpful if someone in this community familiar with this kind of material can help me understand this. Thanks.

Update 1:

Although I agree that (from @aginensky) if $\frac{\partial}{\partial\theta_i}p_{\theta}$ are linearly independent then $\frac{\partial}{\partial\theta_i}\log p_{\theta}$ are also linearly independent, how these are members of the tangent space in the first place is not very clear. So how can $\frac{\partial}{\partial\theta_i}\log p_{\theta}$ be considered as basis for the tangent space. Any help is appreciated.

Update 2:

@aginensky: In his book Amari says the following:

Let us consider the case where $S^n=\mathcal{P}(\mathcal{X})$, the set of all (strictly) positive probability measures on $\mathcal{X}=\{x_0,\dots,x_n\}$, where we regard $\mathcal{P}(\mathcal{X})$ as a subset of $\mathbb{R}^{\mathcal{X}}=\{X\big|X:\mathcal{X}\to \mathbb{R}\}$. In fact, $\mathcal{P}(\mathcal{X})$ is an open subset of the affine space $\{X\big |\sum_x X(x)=1\}$.

Then the tangent space $T_p(S^n)$ of $S^n$ at every point can naturally be identified with the linear subspace $\mathcal{A}_0=\{X\big |\sum_x X(x)=0\}$. For the natural basis $\frac{\partial}{\partial\theta_i}$ of a coordiante system $\theta=(\theta_1,\dots,\theta_n)$, we have $(\frac{\partial}{\partial\theta_i})_{\theta}=\frac{\partial}{\partial\theta_i}p_{\theta}$.

Next, let us take another embedding $p\mapsto \log p$, and identify $S^n$ with the subset $\log S^n:=\{\log p\big |p\in S^n\}$ of $\mathbb{R}^{\mathcal{X}}$. A tangent vector $X\in T_p(S^n)$ is then represented by the result of operating $X$ to $p\mapsto \log p$, which we denote by $X^{(e)}$. In particular we have $(\frac{\partial}{\partial\theta_i})_{\theta}^{(e)}=\frac{\partial}{\partial\theta_i}\log p_{\theta}$. It is obvious that $X^{(e)}=X(x)/p(x)$ and that $$T_p^{(e)}(S^n)=\{X^{(e)}\big |X\in T_p(S^n)\}=\{A\in \mathbb{R}^{\mathcal{X}}\big |\sum_x A(x)p(x)=0\}.$$

My question: If both $\frac{\partial}{\partial\theta_i}$ and $(\frac{\partial}{\partial\theta_i})^{(e)}$ are basis for the tangent space then would this not contradict the fact that $T_p$ and $T_p^{(e)}$ are distinct and $\frac{\partial}{\partial\theta_i}^{(e)}\in T_p^{(e)}$?

I guess there seems to be an association between ($S^n,T_p$) and $(\log S^n,T_p^{(e)})$. If you can clarify this, it would be of great help. You may give it as an answer.

• Personally, I understand your confusion. It seems p not natural to use the coordinates "$e_i(\theta)=\frac{\partial}{\partial\theta_i}\log p_{\theta}(x)$" for the tangent space. Your question is local, so we cab take $\theta_{i}$ as local coordinates. The usual coordinates for the tangent space are $\frac{\partial}{\partial\theta_i}$. Given reasonable conditions on $p_{\theta}$ of smoothness, non vanishing derivative,etc., then by the chain rule, one is taking the standard basis of the tangent space and multiplying it by functions, which in general, will still be a basis.
– meh
Apr 7, 2014 at 13:34
• I tried to edit my comment for clarity and wasn't allowed to. Let me know if you want more details.
– meh
Apr 7, 2014 at 13:41
• Thank you @aginensky: You mean, because $\frac{\partial}{\partial\theta_i}\log p_{\theta}(x)=1/p_{\theta}(x)\frac{\partial}{\partial\theta_i}p_{\theta}(x)$, this is also a basis for the tangent space, right? Apr 7, 2014 at 14:50
• The final statement is a (corrupted) version of one definition of a tangent space. Strictly speaking, the tangent space at a point of a differentiable manifold is the (vector space) dual to the space of derivations of germs of differentiable functions in a neighborhood of that point. A basis for the dual is $\{d\theta_i\}$ and, by definition, the $\{\frac{\partial}{\partial\theta_i}\}$ is its dual basis. A standard reference on this material is Volume 1 of Michael Spivak's Differential Geometry, amazon.com/….
– whuber
Apr 7, 2014 at 15:20
• @ Ashok - yes. I would consider what I wrote to be based on a terse version of one definition of a tangent space. Of course since the cotangent space is dual to the tangent space, one could equally argue that $d\theta$ are the true dual basis. In any event as long as the $p_{\theta}$ don't vanish, I think you're good.
– meh
Apr 8, 2014 at 0:18

My comments are so long, I am putting them in as an answer.

I think the question is more philosophical than mathematical at this point. Namely, what do you mean by a space, and in this case, a manifold? The typical definition of a manifold does not involve an embedding into an affine space. This is the 'modern' (150 year old?) approach. For example, to Gauss, a manifold was a manifold with a specific embedding into a specific affine space ($R^n$). If one has a manifold with an embedding in a specific $R^n$, then the tangent space (at any point of the manifold) is isomorphic to a specific subspace of the tangent space to $R^n$ at that point. Note that the tangent space to $R^n$ at any point is identified with the 'same' $R^n$.

I think the point is that in the Amari article, the space he refers to as $S^n$ comes with some 'natural' embedding in an affine space with coordinates the $\theta_{i}$ for which the $p_{\theta}$ can be considered as coordinates on the tangent space of $S^n$. I might add that it is only clear if the function $p$ is 'general' in some sense- for degenerate $p$, this will fail. For example if the function didn't involve all the variables $\theta_{i}$ . The main point is that this embedding of the manifold in a specific $R^n$, gives rise to a specific identification of the tangent space with the $p_{\theta}$. His next point is that because of the properties of $p$, he can map his manifold using the log function to another affine space in which the tangent space has a different identification in terms of the new coordinates (the logs and their derivatives). He then says that because of properties of his situation, the two manifolds are isomorphic and the map induces an isomorphism on the tangent spaces. That leads to an identification (i.e., isomorphism) of the two tangent spaces.

The key idea is that the two tangent spaces are not the same sets, but are isomorphic (which is basically Greek for 'same') after the correct identification. For example, is the group of all permutations of $\{1,2,3\}$ the 'same' group as the group of all permutations of $\{a,b,c\}$? As a simple thought experiment, consider $R^{+}$, the positive reals mapping to $R$, all the reals under the map log. Pick your favorite real number $>0$ and consider what the map is on tangent spaces. Am I finally understanding your question? A caveat is in order, namely that differential geometry is not my main area of expertise. I think I've got it right, but feel free to criticize or still question this answer.

• Your meaning of "isomorphic" is not fully clear, but it seems to be only a very weak one; namely, the one given by the pushforward $f_{*}$ of an invertible differentiable map, which is just some invertible linear transformation. The key idea to doing geometry is to obtain a meaningful and useful Riemanninan metric defined on the manifold. The relevant sense of "isomorphism" would be isometry: that is, the map between the tangent spaces must be distance-preserving.
– whuber
Apr 15, 2014 at 17:15
• @whuber. Indeed, my comments are only on the differential geometry of the situation and the tangent space. I am not at all clear on what conditions on the $p$ would be necessary to make the map an isometry. But as I understood the question, it was really getting at what was the difference between an identification ('the same') and an isomorphism.
– meh
Apr 15, 2014 at 18:05
• @whuber: The relevant Riemannian metric here is given by $G=[g_{i,j}]$, where $g_{i,j}=\sum_x\partial_i p_{\theta}(x)~\partial_j\log p_{\theta}(x)$. Does this suggest $\partial_j\log p_{\theta}$ can also be regarded as tangent vectors? Apr 16, 2014 at 5:49