Clarification in information geometry

Question

This question is concerned with the paper Differential Geometry of Curved Exponential Families-Curvatures and Information Loss by Amari.

The text goes as follows.

Let $S^n=\{p_{\theta}\}$ be an $n$-dimensional manifold of probability distributions with a coordinate system $\theta=(\theta_1,\dots,\theta_n)$, where $p_{\theta}(x)>0$ is assumed...

We may regard every point $\theta$ of $S^n$ as carrying a function $\log p_{\theta}(x)$ of $x$...

Let $T_{\theta}$ be the tangent space of $S^n$ at $\theta$, which is, roughly speaking, identified with a linearized version of a small neighborhood of $\theta$ in $S^n$. Let $e_i(\theta), i=1,\dots,n$ be the natural basis of $T_{\theta}$ associated with the coordinated system...

Since each point $\theta$ of $S^n$ carries a function $\log p_{\theta}(x)$ of $x$, it is natural to regard $e_i(\theta)$ at $\theta$ as representing the function $$e_i(\theta)=\frac{\partial}{\partial\theta_i}\log p_{\theta}(x).$$

I don't understand the last statement. This appears in section 2 of the above mentioned paper. How's the basis of the tangent space are given by the above equation? It would be helpful if someone in this community familiar with this kind of material can help me understand this. Thanks.

---

Update 1:

Although I agree that (from @aginensky) if $\frac{\partial}{\partial\theta_i}p_{\theta}$ are linearly independent then $\frac{\partial}{\partial\theta_i}\log p_{\theta}$ are also linearly independent, how these are members of the tangent space in the first place is not very clear. So how can $\frac{\partial}{\partial\theta_i}\log p_{\theta}$ be considered as basis for the tangent space. Any help is appreciated.

Update 2:

@aginensky: In his book Amari says the following:

Let us consider the case where $S^n=\mathcal{P}(\mathcal{X})$, the set of all (strictly) positive probability measures on $\mathcal{X}=\{x_0,\dots,x_n\}$, where we regard $\mathcal{P}(\mathcal{X})$ as a subset of $\mathbb{R}^{\mathcal{X}}=\{X\big|X:\mathcal{X}\to \mathbb{R}\}$. In fact, $\mathcal{P}(\mathcal{X})$ is an open subset of the affine space $\{X\big |\sum_x X(x)=1\}$.

Then the tangent space $T_p(S^n)$ of $S^n$ at every point can naturally be identified with the linear subspace $\mathcal{A}_0=\{X\big |\sum_x X(x)=0\}$. For the natural basis $\frac{\partial}{\partial\theta_i}$ of a coordiante system $\theta=(\theta_1,\dots,\theta_n)$, we have $(\frac{\partial}{\partial\theta_i})_{\theta}=\frac{\partial}{\partial\theta_i}p_{\theta}$.

Next, let us take another embedding $p\mapsto \log p$, and identify $S^n$ with the subset $\log S^n:=\{\log p\big |p\in S^n\}$ of $\mathbb{R}^{\mathcal{X}}$. A tangent vector $X\in T_p(S^n)$ is then represented by the result of operating $X$ to $p\mapsto \log p$, which we denote by $X^{(e)}$. In particular we have $(\frac{\partial}{\partial\theta_i})_{\theta}^{(e)}=\frac{\partial}{\partial\theta_i}\log p_{\theta}$. It is obvious that $X^{(e)}=X(x)/p(x)$ and that $$T_p^{(e)}(S^n)=\{X^{(e)}\big |X\in T_p(S^n)\}=\{A\in \mathbb{R}^{\mathcal{X}}\big |\sum_x A(x)p(x)=0\}.$$

My question: If both $\frac{\partial}{\partial\theta_i}$ and $(\frac{\partial}{\partial\theta_i})^{(e)}$ are basis for the tangent space then would this not contradict the fact that $T_p$ and $T_p^{(e)}$ are distinct and $\frac{\partial}{\partial\theta_i}^{(e)}\in T_p^{(e)}$?

I guess there seems to be an association between ($S^n,T_p$) and $(\log S^n,T_p^{(e)})$. If you can clarify this, it would be of great help. You may give it as an answer.

Answer 1

My comments are so long, I am putting them in as an answer.

I think the question is more philosophical than mathematical at this point. Namely, what do you mean by a space, and in this case, a manifold? The typical definition of a manifold does not involve an embedding into an affine space. This is the 'modern' (150 year old?) approach. For example, to Gauss, a manifold was a manifold with a specific embedding into a specific affine space ($R^n$). If one has a manifold with an embedding in a specific $R^n$, then the tangent space (at any point of the manifold) is isomorphic to a specific subspace of the tangent space to $R^n$ at that point. Note that the tangent space to $R^n$ at any point is identified with the 'same' $R^n$.

I think the point is that in the Amari article, the space he refers to as $S^n$ comes with some 'natural' embedding in an affine space with coordinates the $\theta_{i}$ for which the $p_{\theta}$ can be considered as coordinates on the tangent space of $S^n$. I might add that it is only clear if the function $p$ is 'general' in some sense- for degenerate $p$, this will fail. For example if the function didn't involve all the variables $\theta_{i}$ . The main point is that this embedding of the manifold in a specific $R^n$, gives rise to a specific identification of the tangent space with the $p_{\theta}$. His next point is that because of the properties of $p$, he can map his manifold using the log function to another affine space in which the tangent space has a different identification in terms of the new coordinates (the logs and their derivatives). He then says that because of properties of his situation, the two manifolds are isomorphic and the map induces an isomorphism on the tangent spaces. That leads to an identification (i.e., isomorphism) of the two tangent spaces.

The key idea is that the two tangent spaces are not the same sets, but are isomorphic (which is basically Greek for 'same') after the correct identification. For example, is the group of all permutations of $\{1,2,3\}$ the 'same' group as the group of all permutations of $\{a,b,c\}$? As a simple thought experiment, consider $R^{+}$, the positive reals mapping to $R$, all the reals under the map log. Pick your favorite real number $>0$ and consider what the map is on tangent spaces. Am I finally understanding your question? A caveat is in order, namely that differential geometry is not my main area of expertise. I think I've got it right, but feel free to criticize or still question this answer.