General approaches and techniques for developing good explanatory models for nonlinear data

Question

Various recent efforts of mine on modelling some data through logistic regression have been... not successful. While there is still more data to look at, I've been wanting to explore nonlinear dependencies in the data too.

Are there techniques out there that have better fitting capabilities than logistic regression (or linear regressions in general) that still retain easy interpretability and explanatory power (i.e. are not neural networks or other 'black box' techniques)?

My initial intuition is to apply transforms to the individual variables and combinations thereof. For the variables $ x_1...x_n $, possibilities include taking $log(x_i)$, $x_i^m$, $\sqrt[m]x_i $ or $x_i * x_j$ etc. Third order relationships (i.e. $x_i *x_j *x_k$) would be omitted as they aren't easy to visualise or graph.

Needless to say, even for a smallish number of variables, the number of these transformed variables could easily balloon to unwieldy sizes, and the number of combinations of these variables to systematically try would be similarly huge. Furthermore, there would be issues of multicolinearity between the various transforms of a given variable $x_i$.

EDIT: I'm now considering using orthogonal polynomials for the continuous variables and logical combinations ( $x_i & x_j* ) for the categorical variables of different types. I feel like I'm reinventing the wheel here, yet still haven't found useful information on doing this sort of stuff sensibly. Any help would be greatly appreciated.

Answer 1

I think it is worth considering the use of generalised additive models (GAMs). GAMs are able to encapsulate non-linear relations between the response variable and the outcome variables and are straight-forward to explain. They are well-understood and widely used within the Statistics community. In totally informal manner: GAMs are practically GLMs with a out-of-the-box, semi-automated basis expansion module strapped in. No need to define quirky $x^{\frac{1}{4}}, \sin(2\pi x)$, etc. transformations, the best non-linear relation will be automatically selected. GAM are great to visualise the (potentially) varying influence of $x$ on the outcome $y$.

There is an abundance of good resources on using GAMs online (e.g. here and here), in print (e.g. here and here) and CV has literally dozens of insightful questions and answers on generalized additive models. R has two extremely good GAM packages gam and mgcv. I would suggest you start with mgcv as a matter of convenience.

I would suggest you also look at the FAT/ML (Fairness, Accountability, and Transparency in Machine Learning) initiative. It has some great novel ideas. In relation with GAMs I would point you to the 2017 invited talk by Rich Caruana on "Friends Don’t Let Friends Deploy Black-Box Models"; it shows an application of an extension of GAMs (called GA2M) that is used instead of standard ML techniques (random forests) and gives results of similar accuracy but also being fully interpretable.

Answer 2

I will spin the question to terminology that I think clarifies the issue at hand - how can we increase model flexibility while maintaining interpretability and not substantially increasing model variance (in the sense of bias/variance trade off)?

Applying many transformations to features & interactions is a reasonable approach to increase the flexibility of the model. Regularized regression, for example the LASSO or elastic net, can be applied on top of the synthesized feature set to fit a comparatively flexible model while simultaneously performing a feature selection & shrinking model variability. In addition, regularization will shrink or outright remove model parameters for correlated features. This makes interpretation of individual effects in the fitted model more reliable by correctly attributing effect sizes.

However, this approach requires the practitioner to determine the functional form of the model. For example, how does one decide which transformations to apply? Which features to interact and their degree? Depending on the context, like domain knowledge or data size & sparsity, this may not be practical. In such cases, we'd prefer even more flexible methods like neural nets or tree ensembles, without sacrificing interpretability, which neural nets and tree ensembles typically do.

One such model is RuleFit (initial publication and short explanation). The idea is to fit arbitrary shapes in the data (flexibility) using decision rules and then build a model using a small subset of rules (variability). This is accomplished using a tree ensemble for rule generation followed by a LASSO for rule selection. Tree depth is used to control the degree of interaction between features. Note that the end result is still a linear model which enjoys the typical interpretability. The final rule set in the fitted model may provide additional insights by identifying splits in the data where the response varies largely.

A final note - while interpretable models are desirable, it is possible to perform many types of diagnostics & inference on "black box" models using model agnostic methods.

Answer 3

As a engineer from the physics field, I understand how this matter can be crucial.

If you need multi-dimensional "smooth" fits, gaussian process regression usually works quite well. The analysis of your data can then be done through projection plots (for 2 or maybe 3 input parameters at the same time, at most). You don't get any analytic expression of your data, but still can link it to your knowledge of the problem.

Another good solution that I heard of (but never tried by myself) is the global sensitivity analysis using Sobol indices. This method is based on the decomposition of your problem in first, second, and so on, orders, where the $n$-th order corresponds to combinations of subsets of n input parameters (among the total number $N$).

Finally, there is a supervised learning technique that I like because it's a white box approach: classification and regression trees. You get a binary tree which can be easily interpreted, because it works directly on input parameters. Combined influences of several input parameters can be found by studying the deeper layers of the tree.