Fisher information matrix determinant for an overparameterized model

Question

Consider a Bernoulli random variable $X\in\{0,1\}$ with parameter $\theta$ (probability of success). The likelihood function and Fisher information (a $1 \times 1$ matrix) are:

$$ \begin{align} \mathcal{L}_1(\theta;X) &= p(\left.X\right|\theta) = \theta^{X}(1-\theta)^{1-X} \\ \mathcal{I}_1(\theta) &= \det \mathcal{I}_1(\theta) = \frac{1}{\theta(1-\theta)} \end{align} $$

Now consider an "over-parameterized" version with two parameters: the probability of success $\theta_1$ and the probability of failure $\theta_0$. (Note that $\theta_1+\theta_0=1$, and this constraint implies that one of the parameters is redundant.) In this case the likelihood function and Fisher information matrix (FIM) are:

$$ \begin{align} \mathcal{L}_2(\theta_1,\theta_0;X) &= p(\left.X\right|\theta_1,\theta_0) = \theta_1^{X}\theta_0^{1-X} \\ \mathcal{I}_2(\theta_1,\theta_0) &= \left( \begin{matrix} \frac{1}{\theta_1} & 0 \\ 0 & \frac{1}{\theta_0} \end{matrix} \right) \\ \det \mathcal{I}_2(\theta) &= \frac{1}{\theta_1 \theta_0} = \frac{1}{\theta_1 (1-\theta_1)} \end{align} $$

Notice that the determinants of these two FIMs are identical. Furthermore, this property extends to the more general case of categorical models (i.e. more than two states). It also appears to extend to log-linear models with various subsets of parameters constrained to be zero; in this case, the extra "redundant" parameter corresponds to the log partition function, and the equivalence of the two FIM determinants can be shown based on the Schur complement of the larger FIM. (Actually, for log-linear models the smaller FIM is just the Schur complement of the larger FIM.)

Can someone explain whether this property extends to a larger set of parametric models (e.g. to all exponential families), allowing the option of deriving the FIM determinants based on such an "extended" set of parameters? I.e. assume any given statistical model with $n$ parameters which lie on a $n$-dimensional manifold embedded in a $(n+1)$-dimensional space. Now, if we extend the set of parameters to include one more dimension (which is totally constrained based on the others) and compute the FIM based those $(n+1)$ parameters, will we always get the same determinant as that based on the original $n$ (independent) parameters? Also, how are these two FIMs related?

The reason I ask this question is that the $(n+1) \times (n+1)$ FIM with the extra parameter often appears simpler. My first thought is that this shouldn't work in general. The FIM involves computing partial derivatives of the log likelihood wrt each parameter. These partial derivatives assume that, while the parameter in question changes, all other parameters remain constant, which is not true once we involve the extra (constrained) parameter. In this case, it seems to me that the partial derivatives are no longer valid because we cannot assume the other parameters are constant; however, I have yet to find evidence that this is actually a problem. (If partial derivatives are problematic in cases with dependent parameters, are total derivatives needed instead? I have not yet seen an example of computing the FIM with total derivatives, but maybe that is the solution...)

The only example I could find online which computes the FIM based on such an "extended" set of parameters is the following: these notes contain an example for the categorical distribution, computing the required partial derivatives as usual (i.e. as if each parameter is independent, even though a constraint is present among the parameters).

Answer 1

For normal $X\sim N(\mu,\sigma^2)$, information matrix is $$\mathcal{I}_1 = \left( \begin{matrix} \frac{1}{\sigma^2} & 0 \\ 0 & \frac{1}{2\sigma^4} \end{matrix} \right) $$ For curved normal $X\sim N(\mu,\mu^2)$ $$\mathcal{I}_2=\frac{3}{\mu^2}.$$So, your observation that determinants being equal is not universal, but that is not the whole story.

Generally, if $\mathcal{I}_g$ is the information matrix under the reparametrization $$g(\theta)=(g_1(\theta),...,g_k(\theta))',$$ then, it is not difficult to see that the information matrix for the original parameters is $$I(\theta)=G'I_g(g(\theta))G$$ where $G$ is the Jacobian of the transformation $g=g(\theta)$.

For Bernoulli example $(\theta_0,\theta_1)=(p,1-p)$ and $g(p)=(p,1-p)$. So, the Jacobian is $(1,-1)'$ and thus $$\mathcal{I}(p) = \left( \begin{matrix} 1& -1 \end{matrix} \right)\left( \begin{matrix} \frac{1}{p} & 0 \\ 0 & \frac{1}{1-p} \end{matrix} \right) \left( \begin{matrix} 1 \\ -1 \end{matrix} \right)=\frac{1}{p(1-p)}$$

For curved normal example, $$\mathcal{I}_2 = \left( \begin{matrix} 1& 2\mu \end{matrix} \right)\left( \begin{matrix} \frac{1}{\mu^2} & 0 \\ 0 & \frac{1}{2\mu^4} \end{matrix} \right) \left( \begin{matrix} 1 \\ 2\mu \end{matrix} \right)=\frac{3}{\mu^2}.$$

I think now you can easily relate the determinants.

Follow-up after the comment

If I understood you correctly, the FIM is valid as long as you extend the parameters in meaningful way: the likelihood under new parametrization should be a valid density. Hence, I called the Bernoulli example a unfortunate one.

I think the link you provided has a serious flaw in the derivation of the FIM for categorical variables, as we have $E(x_i^2)=\theta_i(1-\theta_i)\neq \theta_i$ and $E(x_ix_j)=\theta_i\theta_j\neq 0$. Expectection of the negative Hessian gives $\mathrm{diag}\{1/\theta_i\}$, but not for the covariance of the score vectors. If you neglect the constraints, the information matrix equality doesn't hold.

Answer 2

It appears that the result holds for a specific kind of relation between the parameters.

Without claiming full generality for the results below, I stick to the "one to two parameters" case. Denote $g(\theta_0,\theta_1) =0$ the implicit equation that expresses the relationship that must hold between the two parameters. Then the "correct extended", "two-parameter" log-likelihood (not what the OP calculates -we will arrive there)

$$L^e=L^*(\theta_0,\theta_1) +\lambda g(\theta_0,\theta_1)$$ is equivalent to the true likelihood $L$, since $g(\theta_0,\theta_1)=0$, ($\lambda$ is a multiplier) and we can treat the two parameters as independent, while we differentiate.

Using subscripts to denote derivatives with respect to parameters (one subscript first derivative, two subscripts second derivative), the determinant of the Hessian of the correct extended log-likelihood will be

$$D_H(L^e) = [L^*_{00}+\lambda g_{00}][L^*_{11}+\lambda g_{11}] - [L^*_{01}+\lambda g_{01}]^2 = D_H(L) \tag{1}$$

What is the OP doing instead?

He considers the wrong likelihood $L^*(\theta_0,\theta_1)$ "ignoring" the relation between the two parameters, and without taking into account the constraint $g(\theta_0,\theta_1)$. He then proceeds with differentiation and obtains

$$D_H(L^*) = L^*_{00}L^*_{11} - [L^*_{01}]^2 \tag{2}$$

It is evident that $(2)$ is not in general equal to $(1)$.

But if $g_{00}=g_{11}=g_{00}=0$, then

$$(1) \rightarrow D_H(L^e) = L^*_{00}L^*_{11} - [L^*_{01}]^2 = D_H(L^*) = D_H(L)$$

So if the relation between the actual parameter and the redundant parameter is such that the second partial derivatives of the implicit function that links them are all zero, then the approach that is fundamentally wrong, ends up "correct".

For the Bernoulli case, we indeed have

$$g(\theta_0,\theta_1) = \theta_0 + \theta_1 -1 \Rightarrow g_{00}=g_{11}=g_{01}=0$$

ADDENDUM
To respond to @Khashaa question and show the mechanics here, we consider a likelihood specified with a redundant parameter, but also under a constraint that links the redundant parameter with the true one. What we do with log-likelihoods is maximize them -so here we have a case of constrained maximization. Assume a sample of size $n$,:

$$\max L_n^*(\theta_0, \theta_1) = \ln \theta_0\sum_{i=1}^nx_i + \left(n-\sum_{i=1}^nx_i\right)\ln\theta_1,\;\; s.t. \;\; \theta_1 = 1-\theta_0$$

This problem has a Langrangean (what informally I called "correct extended likelihood" above),

$$L^e = \ln \theta_0\sum_{i=1}^nx_i + \left(n-\sum_{i=1}^nx_i\right)\ln\theta_1 + \lambda(\theta_1 - 1+\theta_0)$$

The first-order conditions for a maximum are

$$ \frac {\sum_{i=1}^nx_i}{\theta_0} + \lambda = 0,\;\;\; \frac {n-\sum_{i=1}^nx_i}{\theta_1} +\lambda_0 =0$$

for which we obtain the relation

$$\frac {\sum_{i=1}^nx_i}{\theta_0} = \frac {n-\sum_{i=1}^nx_i}{\theta_1} \Rightarrow \theta_1\sum_{i=1}^nx_i = \left(n-\sum_{i=1}^nx_i\right)\theta_0$$

using the constraint under which the above are valid, $\theta_1 = 1-\theta_0$ we obtain

$$ (1-\theta_0)\sum_{i=1}^nx_i = \left(n-\sum_{i=1}^nx_i\right)\theta_0 $$

$$\Rightarrow \sum_{i=1}^nx_i = n\theta_0 \Rightarrow \hat \theta_0 = \frac 1n\sum_{i=1}^nx_i$$

as we should.

Moreover, since the constraint is linear in all the parameters, its second derivatives will be zero. This is reflected in the fact that in the first-derivatives of the lagrangean, the multiplier $\lambda$ "stands alone" and it will be eliminated when we will take second derivatives of the lagrangean. Which in turn will lead us to a Hessian whose determinant will equal the (one-dimensional) second derivative of the original one-parameter log-likelihood, after imposing also the constraint (which is what the OP does). Then taking the negative of the expected value in both cases, does not change this mathematical equivalence, and we arrive at the relation "one-dimensional Fisher Information = determinant of two-dimensional Fisher Information". Now given that the constraint is linear in all the parameters, the OP obtains the same result (at second-derivative level) without introducing the constraint with a multiplier in the function to be maximized, because at second derivative level, the presence/effect of the constraint disappears in such a case.

All these have to do with calculus, not with statistical concepts.