Consider a Bernoulli random variable $X\in\{0,1\}$ with parameter $\theta$ (probability of success). The likelihood function and Fisher information (a $1 \times 1$ matrix) are:
$$ \begin{align} \mathcal{L}_1(\theta;X) &= p(\left.X\right|\theta) = \theta^{X}(1-\theta)^{1-X} \\ \mathcal{I}_1(\theta) &= \det \mathcal{I}_1(\theta) = \frac{1}{\theta(1-\theta)} \end{align} $$
Now consider an "over-parameterized" version with two parameters: the probability of success $\theta_1$ and the probability of failure $\theta_0$. (Note that $\theta_1+\theta_0=1$, and this constraint implies that one of the parameters is redundant.) In this case the likelihood function and Fisher information matrix (FIM) are:
$$ \begin{align} \mathcal{L}_2(\theta_1,\theta_0;X) &= p(\left.X\right|\theta_1,\theta_0) = \theta_1^{X}\theta_0^{1-X} \\ \mathcal{I}_2(\theta_1,\theta_0) &= \left( \begin{matrix} \frac{1}{\theta_1} & 0 \\ 0 & \frac{1}{\theta_0} \end{matrix} \right) \\ \det \mathcal{I}_2(\theta) &= \frac{1}{\theta_1 \theta_0} = \frac{1}{\theta_1 (1-\theta_1)} \end{align} $$
Notice that the determinants of these two FIMs are identical. Furthermore, this property extends to the more general case of categorical models (i.e. more than two states). It also appears to extend to log-linear models with various subsets of parameters constrained to be zero; in this case, the extra "redundant" parameter corresponds to the log partition function, and the equivalence of the two FIM determinants can be shown based on the Schur complement of the larger FIM. (Actually, for log-linear models the smaller FIM is just the Schur complement of the larger FIM.)
Can someone explain whether this property extends to a larger set of parametric models (e.g. to all exponential families), allowing the option of deriving the FIM determinants based on such an "extended" set of parameters? I.e. assume any given statistical model with $n$ parameters which lie on a $n$-dimensional manifold embedded in a $(n+1)$-dimensional space. Now, if we extend the set of parameters to include one more dimension (which is totally constrained based on the others) and compute the FIM based those $(n+1)$ parameters, will we always get the same determinant as that based on the original $n$ (independent) parameters? Also, how are these two FIMs related?
The reason I ask this question is that the $(n+1) \times (n+1)$ FIM with the extra parameter often appears simpler. My first thought is that this shouldn't work in general. The FIM involves computing partial derivatives of the log likelihood wrt each parameter. These partial derivatives assume that, while the parameter in question changes, all other parameters remain constant, which is not true once we involve the extra (constrained) parameter. In this case, it seems to me that the partial derivatives are no longer valid because we cannot assume the other parameters are constant; however, I have yet to find evidence that this is actually a problem. (If partial derivatives are problematic in cases with dependent parameters, are total derivatives needed instead? I have not yet seen an example of computing the FIM with total derivatives, but maybe that is the solution...)
The only example I could find online which computes the FIM based on such an "extended" set of parameters is the following: these notes contain an example for the categorical distribution, computing the required partial derivatives as usual (i.e. as if each parameter is independent, even though a constraint is present among the parameters).