Section 2.1 of "Exegeses on linear models" (p. 3) describes a local linear model based on a second-order Taylor expansion. How would I go about fitting the model?

I see that the first two terms reduce to linear main effects, quadratic main effects, and pairwise interactions. Those alone I know I can fit with OLS or Gaussian maximum likelihood after centering the data around some $x_0$. But I don't know how to handle the model when the other two terms are included.

Notably, Venables' examples do not make explicit use of this model. Is it even possible to simultaneously identify $\sigma$, $\gamma$, and $\delta$? Or is there another reason why I shouldn't fit this fairly general model? Am I missing Venables' point?

Edit: in light of Glen_b's answer this is probably moving the goalposts a little, but ideally an answer would describe how to fit that particular functional form, or why it's impossible or otherwise a bad idea to try. And for reference here it is: $$ Y = \beta_0 + \sum_i \beta_i (X_i - x_{0i}) + \sum_{i,j} \beta_{i,j} (X_i - x_{0i})(X_j - x_{0j}) + \left( \sigma + \sum_i \gamma_i (X_i - x_{0i}) \right) Z + \delta Z^2 $$ where $i$ and $j$ index predictors $X$, $x_0$ is some point for centering the data, and $Z$ is a standard Gaussian error term


The extra terms are interpreted as variance heterogeneity and skewness.

One might either take these relatively generically or try to treat them more directly and specifically.

i) Generically (i.e. taken as clues on how to expand the model), there are obvious choices (like GLMs, say) which deal with variance heterogeneity and skewness.

ii) If you took the terms more specifically, then you might consider, say a ML approach (I don't know the extent to which one might encounter difficulties, but at least in-principle it looks like it might be doable, or a very similar model might be).

However, to my eyes the text makes it quite clear that it's mainly the first sense, ("variance heterogeneity" and "skewness" in a general sense) rather than the second sense (the specific form of terms listed in the expansion) that's intended, and it explains why:

there is no guarantee that the heuristic remains particularly cogent and at some point practical considerations and experience with the context have to take over, but it is worth considering what it is this simple idea is telling us to look for in extending the linear model

So I believe the intent is "extend such models" further by "including variance heterogeneity and skewness" but not 'fit this specific model' ... because "practical considerations" and "experience" would take over from the precise form of what is merely a "heuristic" - i.e. the expansion is a means to see what general kind of extension "it tells us to look for".

Indeed, since it's just an approximation, I see no particular reason why you'd want to take it literally; it doesn't even have the advantage of simplicity. I'd perhaps consider such specific models if I had a good reason, but in general "practical considerations and experience" (of the subject-matter context) play the far greater part in the modelling choices than the precise form of a second-order expansion.

I have sometimes used a model with a term something like the term in $Z$ and at least occasionally pondered a model with a term something like the one in $Z^2$, but usually prefer other models with notionally similar features ... to use my models in a "more structured and context dependent way".

| cite | improve this answer | |
  • $\begingroup$ Ah, I didn't quite realize what he meant by that sentence. So what he's saying is that, for a linear regression model to be roughly equivalent to a second-order approximation, it should also model heteroskedasticity and skewness in addition to quadratic and interaction terms. $\endgroup$ – shadowtalker Feb 2 '15 at 0:29
  • 1
    $\begingroup$ I think you're still being a tad overly specific. I think he's talking about an extended model more generally (not necessarily 'a linear regression model' per se - we might be able to retain 'linear model', though) and he's taking those terms as suggesting aspects which models may need to incorporate without being prescriptive about what form the model ultimately takes. Take a look at his discussion in sections 3 and 4, for example, but don't interpret the models being discussed there as the only possibilities for responding to those issues (they're explicitly examples, not exhaustive lists) $\endgroup$ – Glen_b Feb 2 '15 at 0:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.