Given a linear regression

\begin{align} y_i = \mathbf{x}_i^T \mathbf{b} \qquad i = 1,..,N \end{align}

or in matricial form:

\begin{align} \mathbf{y} = \mathbf{X}^T \mathbf{b} \end{align}

  • MLE Estimation:

The Maximum Likelihood Estimates of the coefficients, with regularization, are:

\begin{align} \hat{\mathbf{b}} = (\mathbf{X}\mathbf{X}^T + \lambda\mathbf{I})^{-1}\mathbf{X}^T\mathbf{y} \end{align}

  • Bayesian Estimation:

If we want a Bayesian estimation, we put a prior distribution over the coefficients, often like:

\begin{align} \mathbf{b} \sim \mathcal{N}(0, \lambda\mathbf{I}) \end{align}

My question is, does it make sense to use the MLE estimators to set up the hyperparameters? For instance:

\begin{align} \mathbf{b} \sim \mathcal{N}(\hat{\mathbf{b}}, \mathbf{\Sigma_\hat{b}} ) \end{align}

what are the drawbacks of setting the hyperparameters like that?

It is been noted in the comments that this may cause overfitting. Does it changes something if we put add an hyperprior and then use the MLEs for the hyper-hyper-parameters of the hyperprior?

Edit (to clarify what I mean by using MLE for the hyper-hyper-parameters):

For instance, something like:

\begin{align} \mathbf{b}_k &\sim \mathcal{N}(\mathbf{b}_{0k}, \mathbf{\Sigma} )\\ \mathbf{b}_{0k} &\sim \mathcal{N}(\hat{\mathbf{b}}, \mathbf{\Sigma_\hat{b}} ) \end{align}

I'm thinking on some sort of mixture of regressions, where the mean coefficients of a given component $k$, $\mathbf{b}_{0k}$, come from a base distribution that generates the means of all the components.

  • 4
    $\begingroup$ Having hyperparameters depending on the data means the prior is no longer a prior since it depends on the data. One of the drawbacks of this feature is overfitting. $\endgroup$ – Xi'an May 29 '16 at 15:30
  • 1
    $\begingroup$ Totally agree that it's very questionable to claim after seeing the data that you were pretty sure that you were going to see exactly this data. I guess, if using a previous analysis as a prior for a new analysis this may make sense, although I would be tempted to down-weight the prior information or to use a mixture with a non-informative component (additionally some kind of between occasion/experiment/situation/dataset variability may be needed to properly reflect the variation between datasets). $\endgroup$ – Björn May 29 '16 at 17:26
  • $\begingroup$ Thanks @Xi'an and Björn. Would it change something if we add another level of hyperpriors and then use the MLE for this hyper-hyper-parameters? (I edited the question) I'm guessing the answer will be "less overfitting, but still some [even if we added 100 layers]" :) $\endgroup$ – alberto May 29 '16 at 20:57
  • 2
    $\begingroup$ It is not clear what do you mean by using MLE for hyper-hyper-parameters - could you provide an example? As for Xi'an's comment, check this recent thread for an example of why we do not "optimize" priors given the data: stats.stackexchange.com/questions/214664/… $\endgroup$ – Tim May 29 '16 at 21:00
  • $\begingroup$ Thanks Tim, I edited the comment, hoping that it's not overly complicated. I appreciate the link. Even if it does not directly answer the question about the MLE in the hyper-hyper-parameters (except that I gues the circularity would also apply) I liked the mention of empirical bayes, which made me wonder why one can't accuse EB of being circular reasoning. $\endgroup$ – alberto May 29 '16 at 21:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.