Let $(Y_i, X_i)_{1\leq i \leq n}$ be i.i.d. such that $Y_i = (Y_i^A, Y_i^B)' \in \mathbb{R}^2$ and $X_i = (X_i^A, X_i^B)' \in \mathbb{R}^2$. Suppose that
$$ Y_i^A = X_i^A \beta_A + \epsilon_i^A, $$ $$ Y_i^B = X_i^B \beta_B + \epsilon_i^B $$
where $\epsilon_i = (\epsilon_i^A, \epsilon_i^B)'$ are i.i.d. with distribution $N(0, \Sigma)$ independently of $(X_i)_{1\leq i\leq n}$.
Define
$$ \tilde{X} = \begin{bmatrix} X_A & 0 \\\ 0 & X_B \end{bmatrix} $$
With some derivation, we obtain the sample conditional likelihood function for the joint model,
$$ L_n(\beta) = -\sum_{i=1}^N \frac{1}{2} (Y - \tilde{X}\beta)'\Sigma^{-1}(Y-\tilde{X}\beta) - \log(2\pi) - \frac{1}{2} \log (\det \Sigma), $$
which allows us to derive Fisher information,
$$ \mathbb{E}\left[\tilde{X}\Sigma^{-1}\tilde{X}\right] = \mathbb{E}\left[\frac{1}{\det\Sigma}\begin{bmatrix} X_A^2 & c \\\ c & X_B^2 \end{bmatrix}\right] $$
where $c$ is some term and where we assume, for illustrative purposes, $\Sigma$ takes the form
$$ \Sigma = \begin{bmatrix} 1 & \gamma \\\ \gamma & 1 \end{bmatrix} $$
But then the asymptotic variance of the ML estimator for $\beta = (\beta_A, \beta_B)'$ is of the form
$$ \mathbb{E}\left[\det\Sigma \cdot \tilde{d} \begin{bmatrix} X_B^2 & \tilde{c} \\\ \tilde{c} & X_A^2 \end{bmatrix}\right] $$
since it is the inverse of Fisher information under some regularity conditions, where $\tilde{c}$ and $\tilde{d} are once again some terms.
My question is that I thought that (1, 1) of the asymptotic covariance matrix captures information about the variance of the first component of the estimand, and (2, 2) captures the second, and so on. Fisher information also describes the same thing, where (1, 1) of the Fisher matrix describes the curvature of the joint likelihood with respect to the first component of the estimand. But taking the inverse of a 2x2 matrix requires flipping (1, 1) and (2, 2), which means that now, apparently the asymptotic variance of the first component of the estimand, $\beta_A$, is for some reason described as a function of $X_B$. This doesn't make sense to me (but I could also have made an error somewhere).
