Given a linear regression
\begin{align} y_i = \mathbf{x}_i^T \mathbf{b} \qquad i = 1,..,N \end{align}
or in matricial form:
\begin{align} \mathbf{y} = \mathbf{X}^T \mathbf{b} \end{align}
- MLE Estimation:
The Maximum Likelihood Estimates of the coefficients, with regularization, are:
\begin{align} \hat{\mathbf{b}} = (\mathbf{X}\mathbf{X}^T + \lambda\mathbf{I})^{-1}\mathbf{X}^T\mathbf{y} \end{align}
- Bayesian Estimation:
If we want a Bayesian estimation, we put a prior distribution over the coefficients, often like:
\begin{align} \mathbf{b} \sim \mathcal{N}(0, \lambda\mathbf{I}) \end{align}
My question is, does it make sense to use the MLE estimators to set up the hyperparameters? For instance:
\begin{align} \mathbf{b} \sim \mathcal{N}(\hat{\mathbf{b}}, \mathbf{\Sigma_\hat{b}} ) \end{align}
what are the drawbacks of setting the hyperparameters like that?
It is been noted in the comments that this may cause overfitting. Does it changes something if we put add an hyperprior and then use the MLEs for the hyper-hyper-parameters of the hyperprior?
Edit (to clarify what I mean by using MLE for the hyper-hyper-parameters):
For instance, something like:
\begin{align} \mathbf{b}_k &\sim \mathcal{N}(\mathbf{b}_{0k}, \mathbf{\Sigma} )\\ \mathbf{b}_{0k} &\sim \mathcal{N}(\hat{\mathbf{b}}, \mathbf{\Sigma_\hat{b}} ) \end{align}
I'm thinking on some sort of mixture of regressions, where the mean coefficients of a given component $k$, $\mathbf{b}_{0k}$, come from a base distribution that generates the means of all the components.
