Maximum Likelihood for Multivariate Regression (ML)

TL;DR: How do you perform likelihood maximization for a multivariate regression in the context of ML?

Background: For univariate regression we can view datapoints $$(x, y)$$ as being sampled from a conditional probability

$$p(y|x)$$

We assume the conditional probability is given by

$$p(y|x) = \mathcal{N}(y | F(x; \theta), \sigma),$$

that is, a normal distribution where the mean is given by a general function $$F$$ (e.g. a neural network), and the standard deviation is $$\sigma$$ (to be determined).

Then for a given set of samples $$(x_i, y_i)$$, $$i=1...N$$, we can choose the optimal set of parameters, $$\theta^*$$, by maximizing the likelihood, or as you commonly do in ML, minimizing the negative log likelihood.

For a univariate normal distribution, minimizing negative log likelihood can be done in two steps, it turns out:

1. Finding the optimal $$\theta$$ by minimizing the sum of squares error between $$y_i$$ and $$F(x_i; \theta)$$. Typically, done via gradient descent.
2. Estimating $$\sigma$$ by solving d(likelihood)/d$$\sigma$$ = 0 directly.

For multivariate regression, $$x, y$$ become vectors of some dimensionality $$>1$$ and the negative log likelihood is

$$\sum_i \left(\vec{y}_i - \vec{F}(x_i; \theta)\right)^T\Sigma^{-1}\left(\vec{y}_i - \vec{F}(x_i; \theta)\right) + \text{Term dependent on } \Sigma$$

$$\Sigma$$ is the covariance matrix.

Maximum likelihood is no longer simply minimizing sum of squares, as far as I can see, due to the $$\Sigma^{-1}$$.

What is the correct way to maximize likelihood when dealing with an arbitrary $$F(x; \theta)$$ (e.g. a neural network)? How do you estimate $$\Sigma$$?

I could minimize $$\sum_i \left(\vec{y}_i - \vec{F}(x_i; \theta)\right)^T\Sigma^{-1}\left(\vec{y}_i - \vec{F}(x_i; \theta)\right)$$ with some reasonable guess for $$\Sigma$$, but as far as I can see, there is no guarantee that that process will not yield a different $$\Sigma$$. Perhaps an iterative algorithm, but I'd like a derivation of that.

I have tried to read relevant passages in both of Bishop's books, googling, asking chatgpt, and reading my tea leaves, but everywhere only the univariate case is dealt with or it's assumed that $$\Sigma$$ is diagonal.

• Try looking up "Seemingly Unrelated Regression" (en.wikipedia.org/wiki/Seemingly_unrelated_regressions), it may answer your question. Commented Apr 24 at 19:50
• @jbowman, thanks. I don't fully understand the explanation of the model on wikipedia. It seems like x_i only influences y_i - that doesn't seem to apply here. Commented Apr 26 at 7:36
• The variables can be the same across equations, they just can't ALL be the same, because in that case the solution collapses to the same solution as equation-by-equation solutions. In any case, the point was about the estimation of $\Sigma$, and how it's done, which is mentioned briefly in the link. Commented Apr 26 at 15:27

The loglikelihood is $$-\frac{1}{2}\log\det\Sigma - \frac{1}{2}(y-F(\theta))^T\Sigma^{-1}(y-F(\theta))$$

One common way to maximise it (eg, for linear mixed models) is to alternate between estimating $$\theta$$ for fixed $$\Sigma$$ using just the second term, and maximising a profile likelihood for $$\Sigma$$ after profiling out $$\theta$$ [and if you write $$\Sigma=\sigma^2V$$, after profiling out $$\sigma^2$$ as well. There's a detailed derivation of the linear mixed model case here, with some consideration of nonlinear mixed models (ie, nonlinear $$F$$) as well