Why don't we just learn the hyper parameters?

Question

I was implementing a pretty popular paper "EXPLAINING AND HARNESSING ADVERSARIAL EXAMPLES" and in the paper, it trains an adversarial objective function

J''(θ) = αJ(θ) + (1 − α)J'(θ).

It treats α as a hyperparameter. α can be 0.1, 0.2, 0.3, etc.

Regardless of this specific paper, I'm wondering, why don't we just include α into our parameters and learn the best α?

What is the disadvantage to do so? Is it because of overfitting? If so, why does learning just 1 more parameter cause so much overfitting?

Answer 1

Lets see how the first order condition would look like if we plug the hyperparameter $\alpha$ and try to learn it the same way as $\theta$ from data: $$\frac \partial{\partial\alpha} J''(\theta) = \frac \partial{\partial\alpha}\alpha J(\theta) + \frac \partial{\partial\alpha}(1 − \alpha)J'(\theta)\\ = J(\theta) − J'(\theta) = 0$$ Hence, $$J(\theta) = J'(\theta)$$

When this hyperparameter is optimiized, then it will cause both J and J' become the same function, i.e. equal weights. You'll end up with a trivial solution.

If you want a more generic philosophizing then consider this: hyperparameters are usually not tangled with data. What do I mean? In a neural network or even a simple regression your model parameters will be in some ways interacting directly with data: $$y_L=X_L\beta_L$$ $$a_L=\sigma(y_L)$$ $$X_{L+1}=a_L$$ and so on down the layers. You see how $\beta_L$ get tangled in your data. So, when you take a derivative over any $\beta$ of the objective function you get data points entering the result in non obvious ways in matrix, Hessians, cross products etc.

However, if you try to estimate the first order conditions over the hyperparameters, you don'y get this effect. The derivatives of hyperparameters often operate the entire chunks of your model, without shuffling its parts like derivatives over parameters. That's why optimizing hyperparameters often leads to trivial solutions like the one I gave you for the specific paper. Optimizing hyperparameters doesn't distress your data set and make it uncomfortable enough to produce something interesting.

Answer 2

"Why don't we just learn the hyper parameters?"

It's a great question! I'll try to provide a more general answer. The TL;DR answer is that you can definitely learn hyperparameters, just not from the same data. Read on for a slightly more detailed reply.

---

A hyperparameter typically corresponds to a setting of the learning algorithm, rather than one of its parameters. In the context of deep learning, for example, this is exemplified by the difference between something like the number of neurons in a particular layer (a hyperparameter) and the weight of a particular edge (a regular, learnable parameter).

Why is there a difference in the first place? The typical case for making a parameter a hyperparameter is that it is just not appropriate to learn that parameter from the training set. For example, since it's always easier to lower the training error by adding more neurons, making the number of neurons in a layer a regular parameter would always encourage very large networks, which is something we know for a fact is not always desirable (because of overfitting).

To your question, it's not that we don't learn the hyper-parameters at all. Setting aside the computational challenges for a minute, it's very much possible to learn good values for the hyperparameters, and there are even cases where this is imperative for good performance; all the discussion in the first paragraph suggests is that by definition, you can't use the same data for this task.

Using another split of the data (thus creating three disjoint parts: the training set, the validation set, and the test set, what you could do in theory is the following nested-optimization procedure: in the outer-loop, you try to find the values for the hyperparameters that minimize the validation loss; and in the inner-loop, you try to find the values for the regular parameters that minimize the training loss.

This is possible in theory, but very expensive computationally: every step of the outer loop requires solving (till completion, or somewhere close to that) the inner-loop, which is typically computationally-heavy. What further complicates things is that the outer-problem is not easy: for one, the search space is very big.

There are many approaches to overcome this by simplifying the setup above (grid search, random search or model-based hyper-parameter optimization), but explaining these is well beyond the scope of your question. As the article you've referenced also demonstrates, the fact that this is a costly procedure often means that researchers simply skip it altogether, or try very few setting manually, eventually settling on the best one (again, according to the validation set). To your original question though, I argue that - while very simplistic and contrived - this is still a form of "learning".

Answer 3

Hyperparameters like the one in that paper are often used to balance multiple terms in the loss function. If you made them learnable, the optimization process would simply learn to allocate all of the weight towards terms which are easier to optimize at the cost of terms which are harder to optimize, which defeats the point of balancing the terms.

Another way to see it is that the loss function is a surrogate for an actual objective that is difficult to define or optimize, such as "generate output images should look realistic" or "should be resistant to adversarial examples". In that case, the true goal isn't "find the hyperparameters to minimize the surrogate loss", it's "find the hyperparameters such that when we run SGD on the rest of the parameters to optimize the surrogate, we get good performance on the true objective".

Answer 4

Since you asked "regardless of the paper", I would like to take a simpler example: Penalised linear regression (Ridge/Lasso).

For those cases, I can think of two reasons why: But first, note that there are two functions here: (F1) The loss function, which is an analytic function of the hyper-parameter and the data (in the paper you linked, it's $\tilde{J}$; and (F2) an estimate of the generalisation error, which depends on the optimum solution to (F1) and the hyper-parameter you picked in (F1).

Caveat: A cursory glance at the paper reveals that the authors train a neural network classifier for the MNIST dataset. It doesn't explicitly say how to pick the hyper-parameter $\alpha$, but I would have picked one $\alpha$ that minimises the validation error of the best model.

1. The objective function for optimising the hyper-parameter is an expression that is a proxy for generalisation error. This expression is hard to write down as a simple analytic function that can be differentiated, but it can be easily evaluated at some point by simply solving the underlying optimisation problem.

2. Evaluating the function (F2) requires you to solve an optimisation problem, which could be expensive. So, even if you can approximate the gradient for F2 to do gradient descent, it would be expensive and slow. In such cases, doing a grid-search is often "good enough."

Having said that, there are techniques to optimise black-box objective functions (such as F2) by assuming some smoothness structure due to their dependence on the hyper-parameter. As an example, you can see this post that shows how a Lasso model's performance varies with its hyper-parameter $\lambda$:

(Image taken from this post: https://stats.stackexchange.com/a/26607/54725)

Some references:

• Workshop on Bayesian Optimisation for Black Box Functions: https://bayesopt.github.io/
• Yelp's MOE: https://github.com/Yelp/MOE
• Google's Vizier: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf