# Least squares objective function for maximum a posteriori (MAP) estimate

I want to fit a model to a dataset by using an optimization procedure (i.e. scipy's least_square). My original MLE/least-squares objective function is:

$$\underset{\theta}{\text{argmin}}\left( \frac{1}{2} \sum_n[f(x_n|\theta) - y_n]^2 \right)$$

As a next step, I want to incorporate a prior distribution on the $$\theta$$ parameters, which are two and I've decided to model as a bivariate normal distribution.

$$P(\theta) \sim N(\mu_{\theta}, \Sigma_{\theta})$$

Note that $$\theta$$ and $$\mu_{\theta}$$ are a $$1\times2$$ vector, while $$\Sigma_{\theta}$$ is a $$2 \times 2$$ covariance matrix (i.e. the 2 parameters have non-zero correlation).

I have already good estimates for $$\mu_{\theta}$$ and $$\Sigma_{\theta}$$, and I want to include this prior distribution as an additional regularization term in the objective function. In probability space, what I want to do is find the maximum a posteriori (MAP) estimate:

$$\underset{\theta}{\text{argmax}}\left( P(\theta) P(X|\theta)\right)$$

My question is: what is the correct formulation for the least squares objective function that incorporates the prior distribution as a regularization term?

I tried to work the math, and the closest I could get is the first formula below (1). I also found on the internet formula (2).

\begin{align} & 1) \space\space \underset{\theta}{\text{argmin}}\left( \frac{1}{2} \sum_n[f(x_n|\theta) - y_n]^2 + \frac{1}{2} (\theta - \mu_{\theta})^T \Sigma_{\theta}^{-1} (\theta - \mu_{\theta})\right) \\ & 2) \space\space \underset{\theta}{\text{argmin}}\left( \frac{1}{2} \sum_n[f(x_n|\theta) - y_n]^2 + \frac{\text{det} (\Sigma_{\theta})^{0.5}}{2} (\theta - \mu_{\theta})^T(\theta - \mu_{\theta}) \right) \end{align}

Is either of these correct? Any advice on how to correctly define this MAP objective function is appreciated.

Thanks!

• What is $f$...?
– Tim
Feb 23 '18 at 7:44
• Hi @Tim, here $f(x|\theta)$ is a blackbox function that takes some data x and the model parameters, for which I've formulated a prior, and outputs an estimate of y. That is why I'm solving this problem via numerical optimization rather than a close form solution. Feb 24 '18 at 0:56

If our objective is to find $$\underset{\theta}{\text{argmax}} \left(P(\theta|Y) \right)$$, the correct formulation ends up being:

$$P(\theta|Y) \propto P(Y|\theta)P(\theta)$$

$$P(Y|\theta) \sim N(f(X|\theta), \sigma^2) \propto \exp \bigg(-\frac{1}{\sigma^2} \sum_n\big[y_i - f(x_i|\theta)\big]^2 \bigg)$$

$$P(\theta) \sim N\big(\mu_{\theta}, \Sigma_{\theta}\big) \propto \exp \bigg(-(\theta - \mu_{\theta}) \Sigma_{\theta}^{-1} (\theta - \mu_{\theta})'\bigg)$$

Now we take negative logs and combine the 2 proportionalities:

$$\underset{\theta}{\text{argmin}}\bigg(\frac{1}{\sigma^2} \sum_n\big[y_i - f(x_i|\theta)\big]^2 + (\theta - \mu_{\theta}) \Sigma_{\theta}^{-1} (\theta - \mu_{\theta})'\bigg)$$

Finally we express the likelihood error without the standard error, so that it looks like on least squares.

$$\underset{\theta}{\text{argmin}}\bigg(\sum_n\big[y_i - f(x_i|\theta)\big]^2 + \sigma^2 (\theta - \mu_{\theta}) \Sigma_{\theta}^{-1} (\theta - \mu_{\theta})'\bigg)$$