Maximum Likelihood for Multivariate Regression (ML)

Question

TL;DR: How do you perform likelihood maximization for a multivariate regression in the context of ML?

Background: For univariate regression we can view datapoints $(x, y)$ as being sampled from a conditional probability

$$ p(y|x) $$

We assume the conditional probability is given by

$$ p(y|x) = \mathcal{N}(y | F(x; \theta), \sigma), $$

that is, a normal distribution where the mean is given by a general function $F$ (e.g. a neural network), and the standard deviation is $\sigma$ (to be determined).

Then for a given set of samples $(x_i, y_i)$, $i=1...N$, we can choose the optimal set of parameters, $\theta^*$, by maximizing the likelihood, or as you commonly do in ML, minimizing the negative log likelihood.

For a univariate normal distribution, minimizing negative log likelihood can be done in two steps, it turns out:

1. Finding the optimal $\theta$ by minimizing the sum of squares error between $y_i$ and $F(x_i; \theta)$. Typically, done via gradient descent.
2. Estimating $\sigma$ by solving d(likelihood)/d$\sigma$ = 0 directly.

For multivariate regression, $x, y$ become vectors of some dimensionality $>1$ and the negative log likelihood is

$$ \sum_i \left(\vec{y}_i - \vec{F}(x_i; \theta)\right)^T\Sigma^{-1}\left(\vec{y}_i - \vec{F}(x_i; \theta)\right) + \text{Term dependent on } \Sigma $$

$\Sigma$ is the covariance matrix.

Maximum likelihood is no longer simply minimizing sum of squares, as far as I can see, due to the $\Sigma^{-1}$.

What is the correct way to maximize likelihood when dealing with an arbitrary $F(x; \theta)$ (e.g. a neural network)? How do you estimate $\Sigma$?

I could minimize $\sum_i \left(\vec{y}_i - \vec{F}(x_i; \theta)\right)^T\Sigma^{-1}\left(\vec{y}_i - \vec{F}(x_i; \theta)\right)$ with some reasonable guess for $\Sigma$, but as far as I can see, there is no guarantee that that process will not yield a different $\Sigma$. Perhaps an iterative algorithm, but I'd like a derivation of that.

I have tried to read relevant passages in both of Bishop's books, googling, asking chatgpt, and reading my tea leaves, but everywhere only the univariate case is dealt with or it's assumed that $\Sigma$ is diagonal.

Answer 1

The loglikelihood is $$-\frac{1}{2}\log\det\Sigma - \frac{1}{2}(y-F(\theta))^T\Sigma^{-1}(y-F(\theta))$$

One common way to maximise it (eg, for linear mixed models) is to alternate between estimating $\theta$ for fixed $\Sigma$ using just the second term, and maximising a profile likelihood for $\Sigma$ after profiling out $\theta$ [and if you write $\Sigma=\sigma^2V$, after profiling out $\sigma^2$ as well. There's a detailed derivation of the linear mixed model case here, with some consideration of nonlinear mixed models (ie, nonlinear $F$) as well