Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.
2 added 4 characters in body
source | link

An exponential family has a density of the form

$$ f(y\vert \mu, \phi) = h(y,\phi)\exp \left(\frac{b(\mu)T(y) - A(\mu)}{d(\phi)}\right) $$

Above, $\phi$ is the dispersion parameter. Typically you look at models of the form

$$ f(y_i\vert \mu_i, \phi) = h(y_i,\phi)\exp \left(\frac{b(\mu_i)T(y_i) - A(\mu_i)}{d(\phi)}\right) $$

where $\mu_i = \vec{x}^\top_i\vec{\beta}$ where $\vec{x}_i$ are your covariates and $\vec{\beta}$ are slopes. Thus, all observations share the dispersion parameter. Now to

Basically I am a bit confused by the meaning of these three concepts: the variance of y (the response), the dispersion parameter and the deviance.

...

Would someone please specify the meaning of the three terms? :)

then the dispersion parameter, $\phi$, is used as the variance is

$$\text{Var}(y_i\vert\mu_i,\phi)=A''(\mu_i)d(\phi) = V(\mu_i)d(\phi), \qquad V(\mu)=A''(\mu_i)$$

so the dispersion parameter is a parameter used to scale the variance function $V$. Notice that it is used to scale the observations', $i = 1,\dots,n$, variances by the same factor $d(\phi)$.

The variance of $y_i$ is $\text{Var}(y_i\vert\mu_i;\phi)$. It is the variance you expect given a $\mu_i$ and dispersion parameter $\phi$.

The deviance of $y_i$deviance, as Carl writes, is a generalization of the idea of using the sum of squares of residuals. The deviance is given by

$$D = 2\phi\sum_{i=1}^n\left(\log f(y_i\vert \tilde\mu_i) - \log f(y_i\vert \mu_i) \right)$$

were $\tilde\mu_i$ are the linear predictors in a saturated model. To see that this is a generalization of the sum of squares of residuals, we focus on the distribution of canonical form where $b(\mu) = \mu$ and $T(y) = y$. In this case

$$D = 2\sum_{i=1}^n\left(y_i(\tilde\mu_i - \mu_i) - A(\tilde\mu_i) + A(\mu_i)\right)$$

A linear model has

$$A(\mu) = \mu^2/2,\quad \tilde\mu_i = y_i$$

so

$$\begin{split} D &= 2\sum_{i=1}^n\left(y_i(\tilde\mu_i - \mu_i) - A(\tilde\mu_i) + A(\mu_i)\right) \\ &= 2\sum_{i=1}^n\left(y_i(y_i - \mu_i) - \frac{y_i^2}{2} + \frac{\mu_i^2}{2}\right) \\ &= \sum_{i=1}^n\left(y_i - \mu_i\right)^2 \end{split}$$

which is the usual residual sum of squares.

An exponential family has a density of the form

$$ f(y\vert \mu, \phi) = h(y,\phi)\exp \left(\frac{b(\mu)T(y) - A(\mu)}{d(\phi)}\right) $$

Above, $\phi$ is the dispersion parameter. Typically you look at models of the form

$$ f(y_i\vert \mu_i, \phi) = h(y_i,\phi)\exp \left(\frac{b(\mu_i)T(y_i) - A(\mu_i)}{d(\phi)}\right) $$

where $\mu_i = \vec{x}^\top_i\vec{\beta}$ where $\vec{x}_i$ are your covariates and $\vec{\beta}$ are slopes. Thus, all observations share the dispersion parameter. Now to

Basically I am a bit confused by the meaning of these three concepts: the variance of y (the response), the dispersion parameter and the deviance.

...

Would someone please specify the meaning of the three terms? :)

then the dispersion parameter, $\phi$, is used as the variance is

$$\text{Var}(y_i\vert\mu_i,\phi)=A''(\mu_i)d(\phi) = V(\mu_i)d(\phi), \qquad V(\mu)=A''(\mu_i)$$

so the dispersion parameter is a parameter used to scale the variance function $V$. Notice that it is used to scale the observations', $i = 1,\dots,n$, variances by the same factor $d(\phi)$.

The variance of $y_i$ is $\text{Var}(y_i\vert\mu_i;\phi)$. It is the variance you expect given a $\mu_i$ and dispersion parameter $\phi$.

The deviance of $y_i$ as Carl writes is a generalization of the idea of using the sum of squares of residuals. The deviance is given by

$$D = 2\phi\sum_{i=1}^n\left(\log f(y_i\vert \tilde\mu_i) - \log f(y_i\vert \mu_i) \right)$$

were $\tilde\mu_i$ are the linear predictors in a saturated model. To see that this is a generalization of the sum of squares of residuals, we focus on the distribution of canonical form where $b(\mu) = \mu$ and $T(y) = y$. In this case

$$D = 2\sum_{i=1}^n\left(y_i(\tilde\mu_i - \mu_i) - A(\tilde\mu_i) + A(\mu_i)\right)$$

A linear model has

$$A(\mu) = \mu^2/2,\quad \tilde\mu_i = y_i$$

so

$$\begin{split} D &= 2\sum_{i=1}^n\left(y_i(\tilde\mu_i - \mu_i) - A(\tilde\mu_i) + A(\mu_i)\right) \\ &= 2\sum_{i=1}^n\left(y_i(y_i - \mu_i) - \frac{y_i^2}{2} + \frac{\mu_i^2}{2}\right) \\ &= \sum_{i=1}^n\left(y_i - \mu_i\right)^2 \end{split}$$

which is the usual residual sum of squares.

An exponential family has a density of the form

$$ f(y\vert \mu, \phi) = h(y,\phi)\exp \left(\frac{b(\mu)T(y) - A(\mu)}{d(\phi)}\right) $$

Above, $\phi$ is the dispersion parameter. Typically you look at models of the form

$$ f(y_i\vert \mu_i, \phi) = h(y_i,\phi)\exp \left(\frac{b(\mu_i)T(y_i) - A(\mu_i)}{d(\phi)}\right) $$

where $\mu_i = \vec{x}^\top_i\vec{\beta}$ where $\vec{x}_i$ are your covariates and $\vec{\beta}$ are slopes. Thus, all observations share the dispersion parameter. Now to

Basically I am a bit confused by the meaning of these three concepts: the variance of y (the response), the dispersion parameter and the deviance.

...

Would someone please specify the meaning of the three terms? :)

then the dispersion parameter, $\phi$, is used as the variance is

$$\text{Var}(y_i\vert\mu_i,\phi)=A''(\mu_i)d(\phi) = V(\mu_i)d(\phi), \qquad V(\mu)=A''(\mu_i)$$

so the dispersion parameter is a parameter used to scale the variance function $V$. Notice that it is used to scale the observations', $i = 1,\dots,n$, variances by the same factor $d(\phi)$.

The variance of $y_i$ is $\text{Var}(y_i\vert\mu_i;\phi)$. It is the variance you expect given a $\mu_i$ and dispersion parameter $\phi$.

The deviance, as Carl writes, is a generalization of the idea of using the sum of squares of residuals. The deviance is given by

$$D = 2\phi\sum_{i=1}^n\left(\log f(y_i\vert \tilde\mu_i) - \log f(y_i\vert \mu_i) \right)$$

were $\tilde\mu_i$ are the linear predictors in a saturated model. To see that this is a generalization of the sum of squares of residuals, we focus on the distribution of canonical form where $b(\mu) = \mu$ and $T(y) = y$. In this case

$$D = 2\sum_{i=1}^n\left(y_i(\tilde\mu_i - \mu_i) - A(\tilde\mu_i) + A(\mu_i)\right)$$

A linear model has

$$A(\mu) = \mu^2/2,\quad \tilde\mu_i = y_i$$

so

$$\begin{split} D &= 2\sum_{i=1}^n\left(y_i(\tilde\mu_i - \mu_i) - A(\tilde\mu_i) + A(\mu_i)\right) \\ &= 2\sum_{i=1}^n\left(y_i(y_i - \mu_i) - \frac{y_i^2}{2} + \frac{\mu_i^2}{2}\right) \\ &= \sum_{i=1}^n\left(y_i - \mu_i\right)^2 \end{split}$$

which is the usual residual sum of squares.

1
source | link

An exponential family has a density of the form

$$ f(y\vert \mu, \phi) = h(y,\phi)\exp \left(\frac{b(\mu)T(y) - A(\mu)}{d(\phi)}\right) $$

Above, $\phi$ is the dispersion parameter. Typically you look at models of the form

$$ f(y_i\vert \mu_i, \phi) = h(y_i,\phi)\exp \left(\frac{b(\mu_i)T(y_i) - A(\mu_i)}{d(\phi)}\right) $$

where $\mu_i = \vec{x}^\top_i\vec{\beta}$ where $\vec{x}_i$ are your covariates and $\vec{\beta}$ are slopes. Thus, all observations share the dispersion parameter. Now to

Basically I am a bit confused by the meaning of these three concepts: the variance of y (the response), the dispersion parameter and the deviance.

...

Would someone please specify the meaning of the three terms? :)

then the dispersion parameter, $\phi$, is used as the variance is

$$\text{Var}(y_i\vert\mu_i,\phi)=A''(\mu_i)d(\phi) = V(\mu_i)d(\phi), \qquad V(\mu)=A''(\mu_i)$$

so the dispersion parameter is a parameter used to scale the variance function $V$. Notice that it is used to scale the observations', $i = 1,\dots,n$, variances by the same factor $d(\phi)$.

The variance of $y_i$ is $\text{Var}(y_i\vert\mu_i;\phi)$. It is the variance you expect given a $\mu_i$ and dispersion parameter $\phi$.

The deviance of $y_i$ as Carl writes is a generalization of the idea of using the sum of squares of residuals. The deviance is given by

$$D = 2\phi\sum_{i=1}^n\left(\log f(y_i\vert \tilde\mu_i) - \log f(y_i\vert \mu_i) \right)$$

were $\tilde\mu_i$ are the linear predictors in a saturated model. To see that this is a generalization of the sum of squares of residuals, we focus on the distribution of canonical form where $b(\mu) = \mu$ and $T(y) = y$. In this case

$$D = 2\sum_{i=1}^n\left(y_i(\tilde\mu_i - \mu_i) - A(\tilde\mu_i) + A(\mu_i)\right)$$

A linear model has

$$A(\mu) = \mu^2/2,\quad \tilde\mu_i = y_i$$

so

$$\begin{split} D &= 2\sum_{i=1}^n\left(y_i(\tilde\mu_i - \mu_i) - A(\tilde\mu_i) + A(\mu_i)\right) \\ &= 2\sum_{i=1}^n\left(y_i(y_i - \mu_i) - \frac{y_i^2}{2} + \frac{\mu_i^2}{2}\right) \\ &= \sum_{i=1}^n\left(y_i - \mu_i\right)^2 \end{split}$$

which is the usual residual sum of squares.