Often when I'm looking at bayesian analyses, the influence of the prior is chosen via cross validation. For example, suppose $X$ and $Y$ represent some real valued data that I want perform a bayesian linear regression on to estimate $Y$ from $X$. This can be formulated as:

${\hat a} = argmin_a \ \sum_i \|y_i-ax_i\|^2_2 + \lambda \| a^Ta \|$

The sum of squares loss imposes the model $p(y|x,a) = \mathbb{N}(y|\mu_y + ax, \Sigma_{yy} - a^T\Sigma_{xx}a)$ and regularization an isotropic gaussian on $A$ that $p(a) = \mathbb{N}(a| 0, \lambda^{-1} I)$. So this is equivalent to the objective

$\hat{a} = argmax_{a} \ \log \ p({\bf y}, a|{\bf x}, \lambda)$

$ \ \ = argmax_{a} \ \log \ p(a| \lambda) \Pi_i \ p(y_i|x_i,a) $

In cross validation - you might initialize $\lambda$ to many different parameters to pick the one with the best out-of-sample error.

But I thought this just looked like a bit of a dirty hack. Would it be better to maximize the model evidence, i.e. marginalize out $a$ (resulting in a more complex optimization involving expectation-maximization)? This is, for the first step of choosing $\lambda$

$ {\hat \lambda} = argmax_{\lambda} \ \log \ \ p({\bf y}| {\bf x}, \lambda)$

$ \ \ = argmax_{\lambda} \ \log \ \int \ p({\bf y}, a| {\bf x}, \lambda)da$

$ \ \ = argmax_{\lambda} \ \log \ \int p(a| \lambda) \Pi_i \ p(y_i|x_i,a)da$

Once $\lambda$ has been selected via this procedure, choose $a$ by

${\hat a} = argmin_a \ \sum_i \|y_i-ax_i\|^2_2 + {\hat \lambda} \| a^Ta \|$

without the need for a cross-validation step as $\lambda$ has already been chosen.

This has the benefit of accounting for the best $\lambda$ over all possible values of $a$, weighted by what values of $a$ are most probable. Cross validation is elegant in a sense of efficiently using the data but it feels unnecessarily frequentist.

My question is: is EM a valid technique for selecting $\lambda$ in this setting or does this suffer from any statistical shortcomings not present with cross validation?

  • $\begingroup$ Maybe I am missing something...why can't you just optimize for $a $ and $\lambda $ directly with penalized ML. $\endgroup$ Feb 2, 2016 at 2:00
  • $\begingroup$ For one because that's not a convex procedure, so pretty difficult to do practically speaking (unless you have a known procedure in mind you can point me toward). Also, I'm not sure what implications that would have on the out-of-sample error for any new data. $\endgroup$
    – user27886
    Feb 2, 2016 at 2:08
  • $\begingroup$ Okay. Well in either case I don't understand the last equation you have. If you integrate over $a $ then how do you choose a specific $\hat a $ to maximize.....you are integrating over all possible values of $a $ right? I think you can maximize the likelihood in respect to $a $ given a value for $\lambda $ (maximization step). Then take the average of the squared elements of $\hat a $ to approximate a value for $\lambda $ (the expectation step). $\endgroup$ Feb 2, 2016 at 2:56
  • $\begingroup$ My thought is that you integrate over $a$ just for choosing $\lambda$ and running EM. Once you have the $\lambda$, you go back to the cost where $a$ is not integrated out and find $\hat a$ via argmax of the L2 cost. I see that my post does not adequately explain this, and is misleading/wrong. I'll update it to make this more clear. $\endgroup$
    – user27886
    Feb 2, 2016 at 3:30
  • 1
    $\begingroup$ @user27886 to be a bit tongue in cheek, why not choose a completely informative prior that deterministically predicts the data? That will certainly minimize your error. $\endgroup$
    – AdamO
    Feb 2, 2016 at 4:24


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.