Various recent efforts of mine on modelling some data through logistic regression have been... not successful. While there is still more data to look at, I've been wanting to explore nonlinear dependencies in the data too.

Are there techniques out there that have better fitting capabilities than logistic regression (or linear regressions in general) that still retain easy interpretability and explanatory power (i.e. are not neural networks or other 'black box' techniques)?

My initial intuition is to apply transforms to the individual variables and combinations thereof. For the variables $ x_1...x_n $, possibilities include taking $log(x_i)$, $x_i^m$, $\sqrt[m]x_i $ or $x_i * x_j$ etc. Third order relationships (i.e. $x_i *x_j *x_k$) would be omitted as they aren't easy to visualise or graph.

Needless to say, even for a smallish number of variables, the number of these transformed variables could easily balloon to unwieldy sizes, and the number of combinations of these variables to systematically try would be similarly huge. Furthermore, there would be issues of multicolinearity between the various transforms of a given variable $x_i$.

EDIT: I'm now considering using orthogonal polynomials for the continuous variables and logical combinations ( $x_i & x_j* ) for the categorical variables of different types. I feel like I'm reinventing the wheel here, yet still haven't found useful information on doing this sort of stuff sensibly. Any help would be greatly appreciated.


3 Answers 3


I think it is worth considering the use of generalised additive models (GAMs). GAMs are able to encapsulate non-linear relations between the response variable and the outcome variables and are straight-forward to explain. They are well-understood and widely used within the Statistics community. In totally informal manner: GAMs are practically GLMs with a out-of-the-box, semi-automated basis expansion module strapped in. No need to define quirky $x^{\frac{1}{4}}, \sin(2\pi x)$, etc. transformations, the best non-linear relation will be automatically selected. GAM are great to visualise the (potentially) varying influence of $x$ on the outcome $y$.

There is an abundance of good resources on using GAMs online (e.g. here and here), in print (e.g. here and here) and CV has literally dozens of insightful questions and answers on generalized additive models. R has two extremely good GAM packages gam and mgcv. I would suggest you start with mgcv as a matter of convenience.

I would suggest you also look at the FAT/ML (Fairness, Accountability, and Transparency in Machine Learning) initiative. It has some great novel ideas. In relation with GAMs I would point you to the 2017 invited talk by Rich Caruana on "Friends Don’t Let Friends Deploy Black-Box Models"; it shows an application of an extension of GAMs (called GA2M) that is used instead of standard ML techniques (random forests) and gives results of similar accuracy but also being fully interpretable.

  • $\begingroup$ This is a good answer, but even simpler to use is regression splines. See for instance stats.stackexchange.com/questions/212867/… or stats.stackexchange.com/questions/206073/… for examples with code. $\endgroup$ Jun 22, 2018 at 11:36
  • $\begingroup$ @kjetilbhalvorsen: Thank you sifu! In general I would tend to think a linear model with ns as a special case of a GAM. (Yes, obviously the estimation is different and all...) I am uncertain if using regression splines will be simpler at this use case. It would open up questions like: "how many degrees of freedom/basis I use", "how do I visualise the spline itself", "what about the combined SE", etc. while I think that with mgcv::gam(..., method=REML) user life is simplified. The final answers would be functionally equivalent for most cases anyway. $\endgroup$
    – usεr11852
    Jun 22, 2018 at 17:54

I will spin the question to terminology that I think clarifies the issue at hand - how can we increase model flexibility while maintaining interpretability and not substantially increasing model variance (in the sense of bias/variance trade off)?

Applying many transformations to features & interactions is a reasonable approach to increase the flexibility of the model. Regularized regression, for example the LASSO or elastic net, can be applied on top of the synthesized feature set to fit a comparatively flexible model while simultaneously performing a feature selection & shrinking model variability. In addition, regularization will shrink or outright remove model parameters for correlated features. This makes interpretation of individual effects in the fitted model more reliable by correctly attributing effect sizes.

However, this approach requires the practitioner to determine the functional form of the model. For example, how does one decide which transformations to apply? Which features to interact and their degree? Depending on the context, like domain knowledge or data size & sparsity, this may not be practical. In such cases, we'd prefer even more flexible methods like neural nets or tree ensembles, without sacrificing interpretability, which neural nets and tree ensembles typically do.

One such model is RuleFit (initial publication and short explanation). The idea is to fit arbitrary shapes in the data (flexibility) using decision rules and then build a model using a small subset of rules (variability). This is accomplished using a tree ensemble for rule generation followed by a LASSO for rule selection. Tree depth is used to control the degree of interaction between features. Note that the end result is still a linear model which enjoys the typical interpretability. The final rule set in the fitted model may provide additional insights by identifying splits in the data where the response varies largely.

A final note - while interpretable models are desirable, it is possible to perform many types of diagnostics & inference on "black box" models using model agnostic methods.

  • $\begingroup$ Thanks, I will read up on the things you linked. One of my goals is to be able to explain to my non-statistical coworkers the influence individual variables have on the behaviour as a whole. Pure linear regression is great for this, as it reduces the answer to a single number, "this feature x has a effect on the overall result". When introducing polynomials, the relationship becomes a one-dimensional function rather than a constant, which is still fine, "The influence of this feature is strong at lower values of the output, but weaker at higher values". Two-dimensional graphs is pushing it. $\endgroup$
    – Ingolifs
    Jun 19, 2018 at 22:45
  • $\begingroup$ I think you made an error here: "In such cases, we'd like even more flexible methods, like neural nets or tree ensembles, which do not sacrifice interpretability, generally unlike neural nets or tree ensembles." $\endgroup$
    – Ingolifs
    Jun 19, 2018 at 22:46
  • 1
    $\begingroup$ @Ingolifs That sentence is confusing. I meant "In such cases, we'd like even more flexible methods (like neural nets) which do not sacrifice interpretability (unlike neural nets). I'll amend. $\endgroup$
    – khol
    Jun 19, 2018 at 23:16

As a engineer from the physics field, I understand how this matter can be crucial.

If you need multi-dimensional "smooth" fits, gaussian process regression usually works quite well. The analysis of your data can then be done through projection plots (for 2 or maybe 3 input parameters at the same time, at most). You don't get any analytic expression of your data, but still can link it to your knowledge of the problem.

Another good solution that I heard of (but never tried by myself) is the global sensitivity analysis using Sobol indices. This method is based on the decomposition of your problem in first, second, and so on, orders, where the $n$-th order corresponds to combinations of subsets of n input parameters (among the total number $N$).

Finally, there is a supervised learning technique that I like because it's a white box approach: classification and regression trees. You get a binary tree which can be easily interpreted, because it works directly on input parameters. Combined influences of several input parameters can be found by studying the deeper layers of the tree.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.