# What is wrong with treating everything as a hyperparameter?

I've seen a number of questions like this asking whether certain parameters can be treated as a hyperparameter. Why can't we just treat everything as a hyperparameter? I understand that this is an extreme stance, but the way I see it, hyperparameters (along with the parameters) ultimately determines the set of all possible models that we can fit the data against. For example, if we set a low learning rate, the set of models it can explore can be a superset of that when we set a high learning rate.

• "It's turtles all the way down."
– whuber
Jul 7 at 14:19
• You can treat everything as a hyperparameter! Auto-sklearn is an example of this approach. The potential downside (compared to a simple approach with no hyperparameters, like linear regression) is longer training time (searching all those hyperparameters), less efficient use of the data, more risk of overfitting.
– Paul
Jul 7 at 18:20
• This can be a good idea if efficient methods exist to calculate out of sample error, as demonstrated by recent results in Gaussian Process kernel parameter optimization such as arxiv.org/abs/2105.11535 Jul 7 at 19:01

#### Treating "everything" as a hyperparameter leads to an infinite regress of priors

In principle, you can take any constant in a distribution that has an allowable range, and you can then treat it like a conditioning random variable. Consequently, in principle you can always have more hyperparameters in your analysis if you want to. But you have to stop somewhere.

Treating a formerly fixed quantity in a prior distribution as a hyperparameter means that you are changing your prior distribution. To see this, suppose you have a prior for $$\theta$$ using some constant $$\phi$$. If you treat $$\phi$$ as a hyperparameter with density $$f$$ then you get the following change in your (marginal) prior for your parameter:

$$\begin{matrix} & & & \text{Prior} \\[6pt] \text{Known constant } \phi & & & \pi(\theta|\phi) \\[6pt] \text{Hyperparameter } \phi & & & \pi(\theta) = \int \pi(\theta|\phi) f(\phi) d \phi \\[6pt] \end{matrix}$$

Every time we take a fixed quantity in the prior and treat it as a hyperparameter, we change the (marginal) prior. Usually this change makes the prior become more diffuse, because of the additional uncertainty in relation to a quantity it depends on. If we were to try to "treat everything as a hyperparameter" that would just mean that we would construct an infinite regress of prior distributions, as we take more and more quantities and assign them a hyperprior, thereby changing the (marginal) prior. You would never get to a point where you have exhausted all quantities that could be generalised to hyperparameters, so you would never get to an endpoint giving you a prior distribution to use in your analysis.

• NB: this applies to the Bayesian meaning of the term hyperparameter, but not the ML meaning.
– Paul
Jul 7 at 18:22

I'd think about this from a practical perspective, for example a simple supervised classification task.

For this, one would normally chose a model to start with based on some heuristic about data size, shape, and quality. Said model will be parameterized, with our aim being to learn a good set of parameters to predict the class of novel examples drawn from the same distribution as the training data.

As you say, it would be perfectly possible to learn the entire set of parameters for said model using some kind of hyperparameter optimization framework. But this would be an incredibly inefficient way of training your classifier, as you treat the entire function as a black box. The classifier you've chosen will probably come with its own optimization function that aims to produce the lowest possible error on a training set, usually using some kind of feedback mechanism to update the parameters based on the quality of the predictions it is producing.

Your choice of model was a prior you imposed, but that model probably has parameters that can be used to define it but that can't be learned by the standard training algorithm for that model. Example: the number of trees in a random forest. So we need some mechanism to chose these 'hyperparameters', which are hopefully few in number. Of course the space of hyperparameters is essentially infinite so we come to some reasonable balance based on a compute/time budget, and evaluate various settings of the parameters (probably using cross validation) to find a good model for the task at hand.