1
$\begingroup$

I find the meaning of hyperparameters not always clear. The hyperparameters are defined as "the parameters of the prior". Suppose that one has prior information about a certain parameter $\theta$. More precisely, one supposes that $\theta \sim N(\mu_0, \sigma_0^2)$. In that case, the hyperparameters are given by $\mu_0$ and $\sigma_0^2$.

But, what in case of a non-informative prior? What in case of a uniform prior? For instance, suppose we have the following model. We have data $y_1,\ldots,y_n$ where the likelihood is given conditionally on a parameter $\lambda$. Thus: $$y_i \ | \ \lambda \sim N\left(\mu,\frac{\sigma^2}{\sqrt{\lambda}}\right), \qquad \lambda \sim \Gamma(\nu/2,\nu/2)$$ with the non-informative prior $g(\mu, \sigma^2) \sim 1/\sigma$. So we have in fact three parameters $\lambda, \mu, \sigma^2$. What are the hyperparameters in this case?

Improve this question
$\endgroup$
5
  • 7
    $\begingroup$ Possible duplicate of What exactly is a hyperparameter? See also: (2) stats.stackexchange.com/questions/208225/… and (3) stats.stackexchange.com/questions/149098/… $\endgroup$
    – Sycorax ♦
    Jun 14, 2016 at 14:46
  • 2
    $\begingroup$ I don't see why there is a question here. Why would the characterization of a prior (such as "non-informative") have any effect at all on the meaning or definition of a hyperparameter? $\endgroup$
    – whuber ♦
    Jun 14, 2016 at 14:48
  • $\begingroup$ If your prior has any parameters, then those parameters are hyperparameters... $\endgroup$
    – Tim ♦
    Jun 14, 2016 at 14:49
  • $\begingroup$ Well, in the most basic case when you have prior information and where the likelihood is given ... I know that the hyperparameters are the parameters of the prior. My confusion arises when there are multiple levels of parameters. $\endgroup$
    – Cavents
    Jun 14, 2016 at 16:19
  • 1
    $\begingroup$ The uniform distribution on $(a,b)$ has two parameters: $a$ and $b$. The improper, uniform prior over $-\infty, \infty$ would have no parameters (there's nothing left to specify). $\endgroup$
    – Matthew Gunn
    Jun 14, 2016 at 19:36

1 Answer 1

Reset to default
1
$\begingroup$

In your example:

  • if $\nu$ is known there are no hyperparameters. A model does not require to have hyperparameters.
  • if $\nu$ is unknown and you set a prior on it, then $\nu$ would become an hyperparameter.

Nevertheless, I do not understand why $\mu$ and $\sigma$ do not appear in you conditioning and I guess that you did that because there are what is often call a nuisance parameter (meaning that they appear in your modeling but are not of interest ($\lambda$ seems to be your parameter of interest)) which is (from a terminology perspective) different from an hyperparameter. So:

  • $\lambda$ is your parameter of interest,
  • $\mu$ and $\sigma$ are nuisance parameters.

Then from a practical perspective, in most (but not all) cases, there is to IMHO no difference between nuisance parameters and hyperparameters (in most cases, hyperparameters are "nuisance hyperparameters"), both are here to be marginalized out.

For a more broad discussion about the definition of an hyperparameter see What exactly is a hyperparameter?.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks. If $v$ is unknown then we put a prior, and you say '$v$ is a hyperparameter'. But suppose that $v \sim N(\mu_0,\tau_0^2)$ then what are $\mu_0$ and $\tau_0^2$? $\endgroup$
    – Cavents
    Jun 14, 2016 at 16:18
  • $\begingroup$ If $\mu_0$ and $\tau_0$ are unknown they are hyperparameters too (think of them as "hyperhyperparameters" or level 2 hyperparameters). Is it okay to you ? Maybe you should also look at stats.stackexchange.com/questions/200329/… that could help your understanding $\endgroup$
    – beuhbbb
    Jun 14, 2016 at 16:20
  • $\begingroup$ Allright, yes that is clear. In this particular case, the value of $v$ was known explicitly so $v$ is the only occurring hyper parameter while $\lambda, \mu$ and $\sigma^2$ are the parameters. $\endgroup$
    – Cavents
    Jun 14, 2016 at 16:23

Not the answer you're looking for? Browse other questions tagged or ask your own question.