I'm trying to follow Andrew NG cs course on supervised learning. He defines the exponential family as: $$ p(y;\eta) = b(y)exp(\eta T(y) -a(\eta)) $$ and then continues to say that "our goal is to predict the expected value of $T(y)$ given $x$", and that "the canonical response function is $g(\eta) = E(T(y);\eta)$"

Why is that? why do we try to predict the mean of some $T(y)$ we don't care anything about instead of trying to predict $E(y)$?

In a Canadian stat course, I found another definition of the exponential family:

$$ f_{Y}(y;\theta)=exp(yb(\theta) + c(\theta) + d(y)) $$ and there they clearly say: "In a GLM, the relationship between a function of $E[Y]$ and the parameters is linear."

So who's right here? what are we trying to model and what's the intuition behind it? is $T(y)$ an important concept to understand - and if so what's the intuition behind it as well?


1 Answer 1


Your first definition is the general definition of an exponential family, the same as in wikipedia. Using this for iid sampling $X_1, \dotsc, X_n$ from a normal distribution $(\mu, \sigma^2)$, we get $$ T(x_1, \dotsc,x_n)=(\sum x_i^2, n\bar{x}) $$ that is, the sufficient statistics. And yes, this is an important general concept! See wikipedia.

This is too general to use with generalized linear models (glm's) when the following definition is used (corresponding to your second definition, but augmented) $$ f(y_i; \theta_i, \phi)= \exp[A_i \{y_i \theta_i -\gamma(\theta_i)\}/\phi + \tau(y_i, \phi/A_i)] $$ (cited here from Venables & Ripley MASS, 4th edition). For the normal distribution above we get $\phi=\sigma^2$, $\theta=\mu, \gamma(\theta)=\theta^2/2$.

In this restricted formulation, the glm general theory is only about modeling the observable $Y_i$ itself, and the canonical parameter $\theta=\mu$ (in this case). How to estimate the scale parameter $\phi$ is outside this general framework, and must be treated on its own in each special case.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.