1
$\begingroup$

This question already has an answer here:

I find the meaning of hyperparameters not always clear. The hyperparameters are defined as "the parameters of the prior". Suppose that one has prior information about a certain parameter $\theta$. More precisely, one supposes that $\theta \sim N(\mu_0, \sigma_0^2)$. In that case, the hyperparameters are given by $\mu_0$ and $\sigma_0^2$.

But, what in case of a non-informative prior? What in case of a uniform prior? For instance, suppose we have the following model. We have data $y_1,\ldots,y_n$ where the likelihood is given conditionally on a parameter $\lambda$. Thus: $$y_i \ | \ \lambda \sim N\left(\mu,\frac{\sigma^2}{\sqrt{\lambda}}\right), \qquad \lambda \sim \Gamma(\nu/2,\nu/2)$$ with the non-informative prior $g(\mu, \sigma^2) \sim 1/\sigma$. So we have in fact three parameters $\lambda, \mu, \sigma^2$. What are the hyperparameters in this case?

$\endgroup$

marked as duplicate by Sycorax, Tim, Greenparker, gung, John Jun 14 '16 at 22:24

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 7
    $\begingroup$ Possible duplicate of What exactly is a hyperparameter? See also: (2) stats.stackexchange.com/questions/208225/… and (3) stats.stackexchange.com/questions/149098/… $\endgroup$ – Sycorax Jun 14 '16 at 14:46
  • 2
    $\begingroup$ I don't see why there is a question here. Why would the characterization of a prior (such as "non-informative") have any effect at all on the meaning or definition of a hyperparameter? $\endgroup$ – whuber Jun 14 '16 at 14:48
  • $\begingroup$ If your prior has any parameters, then those parameters are hyperparameters... $\endgroup$ – Tim Jun 14 '16 at 14:49
  • $\begingroup$ Well, in the most basic case when you have prior information and where the likelihood is given ... I know that the hyperparameters are the parameters of the prior. My confusion arises when there are multiple levels of parameters. $\endgroup$ – Cavents Jun 14 '16 at 16:19
  • 1
    $\begingroup$ The uniform distribution on $(a,b)$ has two parameters: $a$ and $b$. The improper, uniform prior over $-\infty, \infty$ would have no parameters (there's nothing left to specify). $\endgroup$ – Matthew Gunn Jun 14 '16 at 19:36
1
$\begingroup$

In your example:

  • if $\nu$ is known there are no hyperparameters. A model does not require to have hyperparameters.
  • if $\nu$ is unknown and you set a prior on it, then $\nu$ would become an hyperparameter.

Nevertheless, I do not understand why $\mu$ and $\sigma$ do not appear in you conditioning and I guess that you did that because there are what is often call a nuisance parameter (meaning that they appear in your modeling but are not of interest ($\lambda$ seems to be your parameter of interest)) which is (from a terminology perspective) different from an hyperparameter. So:

  • $\lambda$ is your parameter of interest,
  • $\mu$ and $\sigma$ are nuisance parameters.

Then from a practical perspective, in most (but not all) cases, there is to IMHO no difference between nuisance parameters and hyperparameters (in most cases, hyperparameters are "nuisance hyperparameters"), both are here to be marginalized out.

For a more broad discussion about the definition of an hyperparameter see What exactly is a hyperparameter?.

$\endgroup$
  • $\begingroup$ Thanks. If $v$ is unknown then we put a prior, and you say '$v$ is a hyperparameter'. But suppose that $v \sim N(\mu_0,\tau_0^2)$ then what are $\mu_0$ and $\tau_0^2$? $\endgroup$ – Cavents Jun 14 '16 at 16:18
  • $\begingroup$ If $\mu_0$ and $\tau_0$ are unknown they are hyperparameters too (think of them as "hyperhyperparameters" or level 2 hyperparameters). Is it okay to you ? Maybe you should also look at stats.stackexchange.com/questions/200329/… that could help your understanding $\endgroup$ – peuhp Jun 14 '16 at 16:20
  • $\begingroup$ Allright, yes that is clear. In this particular case, the value of $v$ was known explicitly so $v$ is the only occurring hyper parameter while $\lambda, \mu$ and $\sigma^2$ are the parameters. $\endgroup$ – Cavents Jun 14 '16 at 16:23

Not the answer you're looking for? Browse other questions tagged or ask your own question.