6
$\begingroup$

I was watching Andrew Ng's lecture on the difference between parameter vs hyperparameter, https://www.youtube.com/watch?v=VTE2KlfoO3Q&ab_channel=Deeplearning.ai, and a question came to me.

Is there really that much of a distinction between hyperparameter vs parameter?

For example, weight is often regarded as a parameter as opposed to a hyperparameter. But recent papers have found that random search of the weight can obtain a good result, and beats state-of-the-art optimization methods https://arxiv.org/abs/1803.07055 Is this not the same method for hyperparameter tuning?

Simultaneously, there are papers that tunes the learning rates, optimizers and other so-called "hyperparameters" associated with a model. https://arxiv.org/abs/1606.04474

Then there are methods that directly learns the hyperparameters through gradient based methods. https://arxiv.org/abs/1903.03088

Another inspiration is adaptive control (huge field, spanning 5 decades now), the so-called "hyperparameters" associated with the controller is always learned.

$\endgroup$
3
$\begingroup$

That's a great question - I'm not sure what the best way to answer this, but in a statistical framework, I believe the differences are a bit more clearly cut. I'll be curious to see how others answer this from a purer ML/DL perspective.

I think one way in which they differ is that parameters (at last from a statistical standpoint) are something on which you can make inference on, whereas a hyper-parameter is an element of the algorithm that is tuned to optimize it.

For a concrete example, say you are running a LASSO-type penalty for a linear regression model. The $\beta$ weights/coefficients are parameters as one can make a decision on the estimated values and determine relevance or directionality (i.e., check which coefficients are not 0 in a LASSO procedure, or which "protect agaisnt" vs. "increase" risk). Using the same LASSO example, the $\alpha$ weight on a penalty function can be considered a hyper parameter, since the actual value of the $\alpha$ would not provide any insight into the model/post-hoc analysis.

This is a bit of a "statistical" perspective of a difference b/w what is a parameter vs. a hyper-parameter, though that's one option with how to differentiate. With non-parametric algorithms, decision trees, and neural networks, this is where I think there are more gray areas.

$\endgroup$
2
  • 1
    $\begingroup$ Even in the "statistical" perspective, the distinction may not always be clear, except in a relative sense. $\endgroup$
    – GeoMatt22
    Oct 2 '20 at 4:59
  • $\begingroup$ Bayesian priors and hyperpriors are a whole other monster I didn't even want to touch on haha. It's a very gray area, but excellent point. +1 $\endgroup$ Oct 2 '20 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.