Suppose I'm fitting a complicated (e.g. neural network) model's parameters $\theta$ to some data $D$, and I'm trying to tune hyperparameters (e.g. number of layers, size of layers) $\eta$.
Normally I would solve $\max_\theta p(D|\theta,\eta)$, at least approximately, for a fixed $\eta$ and then do a grid search or ad-hoc search by re-fitting for different values of $\eta$.
\begin{align*} \hat{\theta}(D, \eta) &= \mathrm{argmax}_\theta p(D|\theta, \eta) \\ \hat{\eta} &= \mathrm{argmax}_\eta p(D'|\hat{\theta}(D,\eta),\eta) \end{align*}
Couldn't I instead just try out random parameters $\theta$ for each $\eta$ and see how they do on average?
$$ \int p(D | \theta, \eta) p (\theta| \eta) \mathrm{d}\theta = \int p(D, \theta | \eta) \mathrm{d}\theta = p(D|\eta)$$
I could pick some priors $p(\theta|\eta)$ and then draw a bunch of random samples $\{\theta_1(\eta), \theta_2(\eta), \dots \theta_n(\eta) \}$ and solve $$ \hat{\eta} = \mathrm{argmax}_\eta \frac{1}{n} \sum_i p(D|\theta_i(\eta), \eta)$$
I guess it's a type of Monte Carlo Empirical Bayes.
What gave me the idea is the following quote for Goodfellow, Bengio and Courville's "Deep Learning":
Saxe et al. (2011) showed that layers consisting of convolution followed by pooling naturally become frequency selective and translation invariant when assigned random weights. They argue that this provides an inexpensive way to choose the architecture of a convolutional network: first, evaluate the performance of several convolutional network architectures by training only the last layer; then take the best of these architectures and train the entire architecture using a more expensive approach.
I guess they fit some of the paramaters here, but I think it's related?
My questions are:
Is this a legitimate way of doing model selection/hyperparameter tuning
- In theory?
- In practice?
[monte-carlo]
and replaced[empirical-bayes]
with[bayesian]
(see my answer for details). If you don't agree with the edit, feel free to revert it, or edit it again by yourself. $\endgroup$