# What is wrong with treating everything as a hyperparameter?

I've seen a number of questions like this asking whether certain parameters can be treated as a hyperparameter. Why can't we just treat everything as a hyperparameter? I understand that this is an extreme stance, but the way I see it, hyperparameters (along with the parameters) ultimately determines the set of all possible models that we can fit the data against. For example, if we set a low learning rate, the set of models it can explore can be a superset of that when we set a high learning rate.

• "It's turtles all the way down."
– whuber
Jul 7 at 14:19
• You can treat everything as a hyperparameter! Auto-sklearn is an example of this approach. The potential downside (compared to a simple approach with no hyperparameters, like linear regression) is longer training time (searching all those hyperparameters), less efficient use of the data, more risk of overfitting.
– Paul
Jul 7 at 18:20
• This can be a good idea if efficient methods exist to calculate out of sample error, as demonstrated by recent results in Gaussian Process kernel parameter optimization such as arxiv.org/abs/2105.11535 Jul 7 at 19:01

#### Treating "everything" as a hyperparameter leads to an infinite regress of priors

In principle, you can take any constant in a distribution that has an allowable range, and you can then treat it like a conditioning random variable. Consequently, in principle you can always have more hyperparameters in your analysis if you want to. But you have to stop somewhere.

Treating a formerly fixed quantity in a prior distribution as a hyperparameter means that you are changing your prior distribution. To see this, suppose you have a prior for $$\theta$$ using some constant $$\phi$$. If you treat $$\phi$$ as a hyperparameter with density $$f$$ then you get the following change in your (marginal) prior for your parameter:

$$\begin{matrix} & & & \text{Prior} \\[6pt] \text{Known constant } \phi & & & \pi(\theta|\phi) \\[6pt] \text{Hyperparameter } \phi & & & \pi(\theta) = \int \pi(\theta|\phi) f(\phi) d \phi \\[6pt] \end{matrix}$$

Every time we take a fixed quantity in the prior and treat it as a hyperparameter, we change the (marginal) prior. Usually this change makes the prior become more diffuse, because of the additional uncertainty in relation to a quantity it depends on. If we were to try to "treat everything as a hyperparameter" that would just mean that we would construct an infinite regress of prior distributions, as we take more and more quantities and assign them a hyperprior, thereby changing the (marginal) prior. You would never get to a point where you have exhausted all quantities that could be generalised to hyperparameters, so you would never get to an endpoint giving you a prior distribution to use in your analysis.

• NB: this applies to the Bayesian meaning of the term hyperparameter, but not the ML meaning.
– Paul
Jul 7 at 18:22