# Multivariate Linear Regressions with heterogeneous covariance matrices

Let $$\mathbf Y$$ be $$N\times K$$ response matrix and $$\mathbf X$$ be $$N\times (p+1)$$ design matrix (including the intercept), consider the multivariate linear regression,

$$\mathbf Y = \mathbf{XB+E}\,,$$

where $$\mathbf B$$ is the $$(p+1)\times K$$ matrix of parameters and $$\mathbf E$$ is the matrix of errors.

Suppose the covariance matrices $$\mathbf\Sigma_i$$ of the error are different for each observation, $$i=1,\ldots, N$$, then solve $$\mathbf B$$ by minimizing the multivariate weighted criterion,

$$\mathrm{RSS}(\mathbf{B};\mathbf{\Sigma})=\sum_{i=1}^N(y_i-\mathbf B^Tx_i)^T\mathbf{\Sigma}_i^{-1}(y_i-\mathbf B^Tx_i)\,,$$

where $$y_i$$ is the vector of $$K$$ responses for observation $$i$$, and $$x_i$$ is the $$i$$-th observation with $$p+1$$ elements (including the intercept).

I tried to differentiate it as follows,

$$\frac{\partial \mathrm{RSS}(\mathbf{B};\mathbf{\Sigma})}{\partial \mathbf B} = -2\sum_{i=1}^Nx_i(y_i-\mathbf B^Tx_i)^T\mathbf{\Sigma}_i^{-1} = -2\left(\sum_{i=1}^Nx_iy_i^T\mathbf{\Sigma}_i^{-1}-\sum_{i=1}^Nx_ix_i^T\mathbf B^T\mathbf{\Sigma}_i^{-1}\right)\,,\tag{*}$$

but I failed to continue since it seems that we cannot decouple the coefficient matrix with the covariance matrices.

I am wondering if there is an analytical solution for such a problem. If not, how can we get the numerical solution? I am also confused about which algorithm can be employed to solve such matrix multiplication equation $$(*)$$.

This question is adapted from Exercise 3.11 of The Elements of Statistical Learning.

• If you have a gradient, you can do gradient ascent on the log likelihood – Firebug Jul 5 at 14:18
• @Firebug, If B is a vector, I think I can use gradient descent or other similar methods to solve it, but here B is a matrix. I don't have a clear idea. – ya wei Jul 5 at 14:42
• It doesn't change anything actually. Even though it's easier to represent coefficients as a matrix, on the jacobian formulation you'll have them as a long vector. It's the same, for example, when using gradient methods on neural networks. – Firebug Jul 5 at 19:29
• @Firebug, I got your point, thanks for your hint! – ya wei Jul 6 at 0:56