1
$\begingroup$

I want to fit a model to a dataset by using an optimization procedure (i.e. scipy's least_square). My original MLE/least-squares objective function is:

$$ \underset{\theta}{\text{argmin}}\left( \frac{1}{2} \sum_n[f(x_n|\theta) - y_n]^2 \right)$$

As a next step, I want to incorporate a prior distribution on the $\theta$ parameters, which are two and I've decided to model as a bivariate normal distribution.

$$ P(\theta) \sim N(\mu_{\theta}, \Sigma_{\theta}) $$

Note that $\theta$ and $\mu_{\theta}$ are a $1\times2$ vector, while $\Sigma_{\theta}$ is a $2 \times 2$ covariance matrix (i.e. the 2 parameters have non-zero correlation).

I have already good estimates for $\mu_{\theta}$ and $\Sigma_{\theta}$, and I want to include this prior distribution as an additional regularization term in the objective function. In probability space, what I want to do is find the maximum a posteriori (MAP) estimate:

$$ \underset{\theta}{\text{argmax}}\left( P(\theta) P(X|\theta)\right)$$

My question is: what is the correct formulation for the least squares objective function that incorporates the prior distribution as a regularization term?

I tried to work the math, and the closest I could get is the first formula below (1). I also found on the internet formula (2).

$$ \begin{align} & 1) \space\space \underset{\theta}{\text{argmin}}\left( \frac{1}{2} \sum_n[f(x_n|\theta) - y_n]^2 + \frac{1}{2} (\theta - \mu_{\theta})^T \Sigma_{\theta}^{-1} (\theta - \mu_{\theta})\right) \\ & 2) \space\space \underset{\theta}{\text{argmin}}\left( \frac{1}{2} \sum_n[f(x_n|\theta) - y_n]^2 + \frac{\text{det} (\Sigma_{\theta})^{0.5}}{2} (\theta - \mu_{\theta})^T(\theta - \mu_{\theta}) \right) \end{align}$$

Is either of these correct? Any advice on how to correctly define this MAP objective function is appreciated.

Thanks!

$\endgroup$
  • $\begingroup$ What is $f$...? $\endgroup$ – Tim Feb 23 '18 at 7:44
  • $\begingroup$ Hi @Tim, here $ f(x|\theta) $ is a blackbox function that takes some data x and the model parameters, for which I've formulated a prior, and outputs an estimate of y. That is why I'm solving this problem via numerical optimization rather than a close form solution. $\endgroup$ – masta-g3 Feb 24 '18 at 0:56
0
$\begingroup$

If our objective is to find $ \underset{\theta}{\text{argmax}} \left(P(\theta|Y) \right)$, the correct formulation ends up being:

$$ P(\theta|Y) \propto P(Y|\theta)P(\theta) $$

$$ P(Y|\theta) \sim N(f(X|\theta), \sigma^2) \propto \exp \bigg(-\frac{1}{\sigma^2} \sum_n\big[y_i - f(x_i|\theta)\big]^2 \bigg)$$

$$ P(\theta) \sim N\big(\mu_{\theta}, \Sigma_{\theta}\big) \propto \exp \bigg(-(\theta - \mu_{\theta}) \Sigma_{\theta}^{-1} (\theta - \mu_{\theta})'\bigg)$$

Now we take negative logs and combine the 2 proportionalities:

$$ \underset{\theta}{\text{argmin}}\bigg(\frac{1}{\sigma^2} \sum_n\big[y_i - f(x_i|\theta)\big]^2 + (\theta - \mu_{\theta}) \Sigma_{\theta}^{-1} (\theta - \mu_{\theta})'\bigg)$$

Finally we express the likelihood error without the standard error, so that it looks like on least squares.

$$ \underset{\theta}{\text{argmin}}\bigg(\sum_n\big[y_i - f(x_i|\theta)\big]^2 + \sigma^2 (\theta - \mu_{\theta}) \Sigma_{\theta}^{-1} (\theta - \mu_{\theta})'\bigg)$$

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.