I have been studying Statistics recently, using a few introductory texts.

My issue is these texts only seem to provide analysis methods that are suitable to linear relationships: Pearson r correlation coefficients etc. Additionally all the Statistical methods presented seem to be based on the normal Gaussian distribution.

My question is what tools, methods and techniques are applicable to the analysis of non-linear, non-Gaussian statistics, with the related question of what resources (particularly books/textbooks) can I use to become acquainted with these procedures?


I will list some of the models/methods so that you can more easily locate information on them (whether here, on Wikipedia and other such sources, via google, in the titles of books and articles and so on) on your own. This is not a complete list but covers a number of the more common approaches. I'll also mention some references and a few online resources at the end.

  1. simple methods for fitting curved relationships within a regression framework - e.g. polynomial regression, though trigonometric and other models may be used.

    Polynomial regression example (from here - pink is the fitted values at each $x$):
    enter image description here

    Trigonometric regression (link above): enter image description here

  2. transformation of $\mathbf{x}$ and $y$ variables may be used; one common example is the use of Box-Cox family of power transformations. Where $\mathbf{x}$ is transformed alone, this is essentially a form of the previous case. When both sides are transformed, it can deal with non-normality, heteroskedasticity and curved relationships, though balancing up all three at once can sometimes be tricky. It's often the case that making relationships more homoskedastic can improve the normality (though usually not completely). When taking the interpretation back to the original scale, care needs to be taken.

    Here's an example of the result of transforming a relationship that's non-linear and strongly heteroskedastic to one where linear regression is more suitable (from here):
    enter image description here

  3. nonlinear regression - $y = g(\mathbf{x})+\epsilon$

    ($\mathbf{x}$ is a vector of regressors) used for a nonlinear functional relationship ($g$) with either an assumption of Gaussian errors ($\epsilon$) or simply a square-error loss function. Widely used in the physical sciences (physics, chemistry and so on), though may be found in a variety of other contexts.

    An example (from here black is data, pink is fitted model, $\text{E}(y) = \beta_0 + \beta_1 e^{-\beta_2 x}$):
    enter image description here

    A second example (from here):
    enter image description here

  4. Generalized Linear models (GLM)

    Used for both Gaussian and non-Gaussian $Y$ (within the exponential family) for functions where $g[E(Y|\mathbf{X=x})]$ is linear in $\mathbf{x}$ and the variance is a function of the mean. Very widely used.

    Example of a fit of a binomial GLM to mortality data ($E(d_x)=E_xq_x$, where $q_x = \frac{Ab^x}{1+Ab^x}$):
    enter image description here

  5. There are more general models fitted via maximum likelihood estimation, method of moments (MoM or sometimes MME) or in other ways. In general some model is specified, some loss function is minimized, or some other fitting criterion specified (as in MoM), in order to produce estimates; a variety of optimization or root-finding methods are applied.

  6. Smoothing techniques, including regression splines, smoothing splines (or penalized versions of either), kernel smoothing/local linear regression/local polynomial regression (including LOESS and other local regression methods), and so on. Also under the umbrella of nonparametric regression, though that includes some other techniques not of direct relevance to your question.

    See also Generalized Additive Models.

    Example (weighted natural cubic spline fit to log-transformed data):
    enter image description here

There's a plethora of information on almost all of these topics here on CV.

The book "(An R and S-PLUS) Companion to Applied Regression" (Fox or more recently "An R Companion to Applied Regression" Fox & Weisberg) has a number of relevant chapters (including coverage of transformaiton and GLMs, (and there are additional online chapters here, especially the first couple of chapters at that link which cover nonlinear regression and nonparametric regression) that you may find useful.

Also see Fox, Applied Regression, Linear Models, and Related Methods.

Some of the models/methods are discussed in Elements of Statistical Learning (Hastie, Tibshirani and Friedman). The 10th printing of the second edition is available online, but its not especially introductory (though parts are pretty straightforward) and its focus is different from what you're likely after. Nevertheless you may find some of its chapters of value.


I'd add that the distinction you sense is less than it seems. "Nonlinear statistics" is not really a coherent or standard label, not least because it is defined negatively rather than positively.

  1. On linear: the big positive is that many nonlinearities are coped with by models that are linear in the parameters. You don't have to jump from what you meet in an introductory course straight to nonlinear least-squares. The latter is sometimes a good servant, but often an awkward master.

  2. On normal: only lousy books and courses assume or even imply that it is a case or "normal or die". Modern statistics is cool about data being exponential, gamma, lognormal, binomial, Poisson, or yet more ornery.

  3. The easiest bridge between introductory and slightly more advanced statistics is provided by the idea of a transformation, that some transformation often makes the nonlinear (more nearly) linear and the non-normal (more nearly) normal.

  4. An excellent bridge between introductory and the above is (some variations of taste and judgement here should be expected from statistical people) through the idea of generalized linear models. I would start with an account such as Dobson and Barnett which conveys well the idea that what you know can be extended.


The best answer to this may depend on your discipline, but I like Bruce Hansen's Econometrics textbook (free, online): http://www.ssc.wisc.edu/~bhansen/econometrics/Econometrics.pdf

See Section 9.1 for nonlinear least squares -- these results are really not very different from the linear case.

You might also be interested in some nonparametric techniques. The same guy also has good lecture notes on nonparametrics. Here is a link: http://www.ssc.wisc.edu/~bhansen/718/718.htm


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.