Least squares objective function for maximum a posteriori (MAP) estimate

Question

I want to fit a model to a dataset by using an optimization procedure (i.e. scipy's least_square). My original MLE/least-squares objective function is:

$$ \underset{\theta}{\text{argmin}}\left( \frac{1}{2} \sum_n[f(x_n|\theta) - y_n]^2 \right)$$

As a next step, I want to incorporate a prior distribution on the $\theta$ parameters, which are two and I've decided to model as a bivariate normal distribution.

$$ P(\theta) \sim N(\mu_{\theta}, \Sigma_{\theta}) $$

Note that $\theta$ and $\mu_{\theta}$ are a $1\times2$ vector, while $\Sigma_{\theta}$ is a $2 \times 2$ covariance matrix (i.e. the 2 parameters have non-zero correlation).

I have already good estimates for $\mu_{\theta}$ and $\Sigma_{\theta}$, and I want to include this prior distribution as an additional regularization term in the objective function. In probability space, what I want to do is find the maximum a posteriori (MAP) estimate:

$$ \underset{\theta}{\text{argmax}}\left( P(\theta) P(X|\theta)\right)$$

My question is: what is the correct formulation for the least squares objective function that incorporates the prior distribution as a regularization term?

I tried to work the math, and the closest I could get is the first formula below (1). I also found on the internet formula (2).

$$ \begin{align} & 1) \space\space \underset{\theta}{\text{argmin}}\left( \frac{1}{2} \sum_n[f(x_n|\theta) - y_n]^2 + \frac{1}{2} (\theta - \mu_{\theta})^T \Sigma_{\theta}^{-1} (\theta - \mu_{\theta})\right) \\ & 2) \space\space \underset{\theta}{\text{argmin}}\left( \frac{1}{2} \sum_n[f(x_n|\theta) - y_n]^2 + \frac{\text{det} (\Sigma_{\theta})^{0.5}}{2} (\theta - \mu_{\theta})^T(\theta - \mu_{\theta}) \right) \end{align}$$

Is either of these correct? Any advice on how to correctly define this MAP objective function is appreciated.

Thanks!

Answer 1

If our objective is to find $ \underset{\theta}{\text{argmax}} \left(P(\theta|Y) \right)$, the correct formulation ends up being:

$$ P(\theta|Y) \propto P(Y|\theta)P(\theta) $$

$$ P(Y|\theta) \sim N(f(X|\theta), \sigma^2) \propto \exp \bigg(-\frac{1}{\sigma^2} \sum_n\big[y_i - f(x_i|\theta)\big]^2 \bigg)$$

$$ P(\theta) \sim N\big(\mu_{\theta}, \Sigma_{\theta}\big) \propto \exp \bigg(-(\theta - \mu_{\theta}) \Sigma_{\theta}^{-1} (\theta - \mu_{\theta})'\bigg)$$

Now we take negative logs and combine the 2 proportionalities:

$$ \underset{\theta}{\text{argmin}}\bigg(\frac{1}{\sigma^2} \sum_n\big[y_i - f(x_i|\theta)\big]^2 + (\theta - \mu_{\theta}) \Sigma_{\theta}^{-1} (\theta - \mu_{\theta})'\bigg)$$

Finally we express the likelihood error without the standard error, so that it looks like on least squares.

$$ \underset{\theta}{\text{argmin}}\bigg(\sum_n\big[y_i - f(x_i|\theta)\big]^2 + \sigma^2 (\theta - \mu_{\theta}) \Sigma_{\theta}^{-1} (\theta - \mu_{\theta})'\bigg)$$