For the linear model, general linear models which allow for a more general covariance structure $V(\theta)_{N\times N}=(I_{N}+\theta A_{N\times N})(I_{N}+\theta A_{N\times N})^{'}$ ,where $A_{N\times N}$ is not a function of $\theta$, $I$ is an identity matrix. $Y_{N\times1} = X\beta + \epsilon$, $\epsilon_{N\times 1} ∼ MVN(0, V(\theta ))$ are linear models. The mean is then $E(Y)=X\beta$ and the variance is $V(\theta)$, where $\theta$ is a scalar.

Based on this model assumption, I am trying to find a score function and fisher information. First, I derived a log-likelihood function $$\ell(\beta,\theta)=-\frac{n}{2}log(2\pi)-\frac{1}{2}log|V(\theta)| -\frac{1}{2}(Y-X\beta)^{'}V(\theta)^{-1}(Y-X\beta).$$

Let $U(\theta)=\frac{d}{d\theta}V(\theta)$. The score equations then are

$$S_{\beta}(\beta,\theta)=X^{'}V(\theta)^{-1}(Y-X\beta)=0$$ $$S_{\theta}(\beta,\theta)=-\frac{1}{2}tr(V(\theta)^{-1}U(\theta))+\frac{1}{2}(Y-X\beta)^{'}V(\theta)^{-1}U(\theta)V(\theta)^{-1}(Y-X\beta)=0.$$

Now based on the score function of $\theta$, I want to derive a fisher information function to apply fisher scoring algorithm. Following the derivative of the rule of matrix, what I found is

\begin{eqnarray} I_{\theta}(\beta,\theta)&=&\frac{1}{2}(-V(\theta)^{-1}U(\theta)V(\theta)^{-1}U(\theta)+V(\theta)^{-1}\frac{d U(\theta)}{d\theta})^{'}\\ & &+\frac{1}{2}(Y-X\beta)^{'}V(\theta)^{-1}\frac{dU(\theta)}{d\theta}V(\theta)^{-1}U(\theta)V(\theta)^{-1}(Y-X\beta)\\ & &-\frac{1}{2}(Y-X\beta)^{'}V(\theta)^{-1}\frac{dU(\theta)}{d\theta}V(\theta)^{-1}(Y-X\beta)\\ & &+\frac{1}{2}(Y-X\beta)^{'}V(\theta)^{-1}U(\theta)V(\theta)^{-1}\frac{dU(\theta)}{d\theta}V(\theta)^{-1}(Y-X\beta)\\ \end{eqnarray}

The first term of the fisher information is derivative of the trace function with respect to $\theta$, while the other terms are derivative the second term. Considering $\theta$ is a scalar, if the above fisher information is correct, the dimension of the first term in the fisher information is $N\times N$, while that of other three terms is $1\times 1$. My understanding is that since the information matrix is the derivative of the scalar score function with respect to the scalar, the corresponding information matrix should be a scalar.

Did I make a mistake somewhere? Or is there any easier way to derive the fisher information in this case? Thanks in advance.

  • $\begingroup$ If I follow your notation correctly, your score function $S_\theta$ is real-valued. Consequently its derivative with respect to its second argument (the scalar $\theta$) must be a number, not an $N\times N$ matrix. $\endgroup$ – whuber Jun 13 '16 at 19:14
  • $\begingroup$ indeed, you forgot the trace in the first term. The Fisher information of $\theta$ should be a scalar $\endgroup$ – Alex Monras Jun 13 '16 at 19:15
  • $\begingroup$ @AlexMonras Yes. The fisher information should be a scalar. Should I keep the trace function after taking the derivative with respect to $\theta$? I followed the derivative rule like $\frac{d tr(F(x))}{dx}=f(x)^{T}$, $f(x)$ is the derivative of $F(x)$ with respect to $x$. $\endgroup$ – Sue Jun 13 '16 at 19:18
  • $\begingroup$ @whuber Yes. I think so. So I am thinking there is something wrong in the first component of the fisher information. $\endgroup$ – Sue Jun 13 '16 at 19:20
  • 1
    $\begingroup$ well it's actually $\frac{d}{dx}\mathrm{tr}[F(x)]=\mathrm{tr}[\frac{dF(x)}{dx}]$, as follows from the linearity of the derivative $\endgroup$ – Alex Monras Jun 13 '16 at 19:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.