Let $\mathbf Y$ be $N\times K$ response matrix and $\mathbf X$ be $N\times (p+1)$ design matrix (including the intercept), consider the multivariate linear regression,

$$ \mathbf Y = \mathbf{XB+E}\,, $$

where $\mathbf B$ is the $(p+1)\times K$ matrix of parameters and $\mathbf E$ is the matrix of errors.

Suppose the covariance matrices $\mathbf\Sigma_i$ of the error are different for each observation, $i=1,\ldots, N$, then solve $\mathbf B$ by minimizing the multivariate weighted criterion,

$$ \mathrm{RSS}(\mathbf{B};\mathbf{\Sigma})=\sum_{i=1}^N(y_i-\mathbf B^Tx_i)^T\mathbf{\Sigma}_i^{-1}(y_i-\mathbf B^Tx_i)\,, $$

where $y_i$ is the vector of $K$ responses for observation $i$, and $x_i$ is the $i$-th observation with $p+1$ elements (including the intercept).

I tried to differentiate it as follows,

$$ \frac{\partial \mathrm{RSS}(\mathbf{B};\mathbf{\Sigma})}{\partial \mathbf B} = -2\sum_{i=1}^Nx_i(y_i-\mathbf B^Tx_i)^T\mathbf{\Sigma}_i^{-1} = -2\left(\sum_{i=1}^Nx_iy_i^T\mathbf{\Sigma}_i^{-1}-\sum_{i=1}^Nx_ix_i^T\mathbf B^T\mathbf{\Sigma}_i^{-1}\right)\,,\tag{*} $$

but I failed to continue since it seems that we cannot decouple the coefficient matrix with the covariance matrices.

I am wondering if there is an analytical solution for such a problem. If not, how can we get the numerical solution? I am also confused about which algorithm can be employed to solve such matrix multiplication equation $(*)$.

This question is adapted from Exercise 3.11 of The Elements of Statistical Learning.

  • 1
    $\begingroup$ If you have a gradient, you can do gradient ascent on the log likelihood $\endgroup$ – Firebug Jul 5 at 14:18
  • $\begingroup$ @Firebug, If B is a vector, I think I can use gradient descent or other similar methods to solve it, but here B is a matrix. I don't have a clear idea. $\endgroup$ – ya wei Jul 5 at 14:42
  • 1
    $\begingroup$ It doesn't change anything actually. Even though it's easier to represent coefficients as a matrix, on the jacobian formulation you'll have them as a long vector. It's the same, for example, when using gradient methods on neural networks. $\endgroup$ – Firebug Jul 5 at 19:29
  • $\begingroup$ @Firebug, I got your point, thanks for your hint! $\endgroup$ – ya wei Jul 6 at 0:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.