Cross validation or EM for selecting strength of the prior?

Often when I'm looking at bayesian analyses, the influence of the prior is chosen via cross validation. For example, suppose $$X$$ and $$Y$$ represent some real valued data that I want perform a bayesian linear regression on to estimate $$Y$$ from $$X$$. This can be formulated as:

$${\hat a} = argmin_a \ \sum_i \|y_i-ax_i\|^2_2 + \lambda \| a^Ta \|$$

The sum of squares loss imposes the model $$p(y|x,a) = \mathbb{N}(y|\mu_y + ax, \Sigma_{yy} - a^T\Sigma_{xx}a)$$ and regularization an isotropic gaussian on $$A$$ that $$p(a) = \mathbb{N}(a| 0, \lambda^{-1} I)$$. So this is equivalent to the objective

$$\hat{a} = argmax_{a} \ \log \ p({\bf y}, a|{\bf x}, \lambda)$$

$$\ \ = argmax_{a} \ \log \ p(a| \lambda) \Pi_i \ p(y_i|x_i,a)$$

In cross validation - you might initialize $$\lambda$$ to many different parameters to pick the one with the best out-of-sample error.

But I thought this just looked like a bit of a dirty hack. Would it be better to maximize the model evidence, i.e. marginalize out $$a$$ (resulting in a more complex optimization involving expectation-maximization)? This is, for the first step of choosing $$\lambda$$

$${\hat \lambda} = argmax_{\lambda} \ \log \ \ p({\bf y}| {\bf x}, \lambda)$$

$$\ \ = argmax_{\lambda} \ \log \ \int \ p({\bf y}, a| {\bf x}, \lambda)da$$

$$\ \ = argmax_{\lambda} \ \log \ \int p(a| \lambda) \Pi_i \ p(y_i|x_i,a)da$$

Once $$\lambda$$ has been selected via this procedure, choose $$a$$ by

$${\hat a} = argmin_a \ \sum_i \|y_i-ax_i\|^2_2 + {\hat \lambda} \| a^Ta \|$$

without the need for a cross-validation step as $$\lambda$$ has already been chosen.

This has the benefit of accounting for the best $$\lambda$$ over all possible values of $$a$$, weighted by what values of $$a$$ are most probable. Cross validation is elegant in a sense of efficiently using the data but it feels unnecessarily frequentist.

My question is: is EM a valid technique for selecting $$\lambda$$ in this setting or does this suffer from any statistical shortcomings not present with cross validation?

• Maybe I am missing something...why can't you just optimize for $a$ and $\lambda$ directly with penalized ML. Feb 2, 2016 at 2:00
• For one because that's not a convex procedure, so pretty difficult to do practically speaking (unless you have a known procedure in mind you can point me toward). Also, I'm not sure what implications that would have on the out-of-sample error for any new data. Feb 2, 2016 at 2:08
• Okay. Well in either case I don't understand the last equation you have. If you integrate over $a$ then how do you choose a specific $\hat a$ to maximize.....you are integrating over all possible values of $a$ right? I think you can maximize the likelihood in respect to $a$ given a value for $\lambda$ (maximization step). Then take the average of the squared elements of $\hat a$ to approximate a value for $\lambda$ (the expectation step). Feb 2, 2016 at 2:56
• My thought is that you integrate over $a$ just for choosing $\lambda$ and running EM. Once you have the $\lambda$, you go back to the cost where $a$ is not integrated out and find $\hat a$ via argmax of the L2 cost. I see that my post does not adequately explain this, and is misleading/wrong. I'll update it to make this more clear. Feb 2, 2016 at 3:30
• @user27886 to be a bit tongue in cheek, why not choose a completely informative prior that deterministically predicts the data? That will certainly minimize your error. Feb 2, 2016 at 4:24