Hi I'm studying regression techniques.

My data has 15 features and 60 million examples (regression task).

When I tried many known regression techniques (gradient boosted tree, Decision tree regression, AdaBoostRegressor etc) linear regression performed great.

Scored almost best among those algorithms.

What can be the reason for this? Because my data has so many examples so DT based method can fit well.

  • regularized linear regression ridge, lasso performed worse

Can anyone tell me about other performing-well regression algorithms?

  • Is Factorization Machine and Support vector regression good regression technique to try?
  • 2
    $\begingroup$ This has a lot more to do with your data than the algorithm. The structure of a linear regression is just a good fit for your data. $\endgroup$ Commented Jun 21, 2017 at 4:41
  • $\begingroup$ thank you for answering @MatthewDrury. by observing these characteristics, I'm trying to find characteristics of my data. It's clearly has small features and lot of examples. and work best on plain neural network regression. and by the fact non-parametric models such as gradient boosting works slightly worse than parametric regression(assuming the shape of function), can I say my data can't give a lot of insights to unknown data regardless of how many example I have? I'm having trouble with deducting characteristic of my data from result. $\endgroup$ Commented Jun 21, 2017 at 5:25
  • $\begingroup$ Work first with multiple linear rebression and then, study residual plots and such to really understand the fit. Then you can see in what ways the fit is bad. Don't just throw the data at different algorithms, work hard to understand the fits. $\endgroup$ Commented Jun 21, 2017 at 8:26
  • $\begingroup$ @kjetilbhalvorsen thanks for reply. I have 15 independent variable. so How can I plot or get insight from residual fit. can you help me? $\endgroup$ Commented Jun 21, 2017 at 9:32

4 Answers 4


You should not just throw the data at different algorithms and look at the quality of the predictions. You need to understand your data better, and the way of going about that is to first, visualize your data (the marginal distributions). Even if you are only interested finally in the predictions, you will be in a better position to make better models if you understand the data better. So, first, try to understand the data (and simple models fitted to the data) better, and then you are in a much better position to create more complex, and hopefully better, models.

Then, fit linear regression models, with your 15 variables as precictors (later you can look at possible interactions). Then, calculate the residuals from that fit, that is, $$ r_i = Y_i - \hat{Y}_i, \qquad i=1, 2,\dots, n $$ If the model is adecuate, that is, it was able to extract the signal (structure) from the data, then the residuals should show no patterns. Box, Hunter & Hunter: "Statistics for Experimenters" (which you should have a look at, its one of the best ever books on statistics) compares this with an analogy from chemistry: The model is a "filter" designed to catch impurities from water (the data). What is left, which passed through the filter, should then be "clean" and analysis of it (residuals analysis) can show that, when it does not contain impurities (structure). See Checking residuals for normality in generalised linear models

To know what to check for you need to understand the assumptions behind linear regression, see What is a complete list of the usual assumptions for linear regression?

One usual assumption is homoskedasticity, that is, constant variance. To check that, plot the residuals $r_i$ against the predicted values, $\hat{Y}_i$. To understand this procedure see: Why are residual plots constructed using the residuals vs the predicted values?.

Other assumptions is linearity. To check those, plot the residuals against each of the predictors in the model. If you see any curvity in those plots, that is evidence against linearity. If you find non-linearity, either you can try some transformations or (more modern approch) include that non-linear predictor in the model in a non-linear way, maybe using splines (you have 60 million examples so should be quite feasible!).

Then you need to check for possible interactions. The above ideas can be used also for variables not in the fitted model. Since you fit a model without interactions, that include interaction variables, like the product $x_i \cdot z_i$ for two variables $x$, $z$. So plot the residuals against all this interaction variables. A blog post with many example plots is http://docs.statwing.com/interpreting-residual-plots-to-improve-your-regression/

A book-length treatment is R Dennis Cook & Sanford Weisberg: "Residuals and influence in regression", Chapman & Hall. A more modern book-length treatment is Frank Harrell: "Regression modeling Strategies".

And, coming bact to the question in the title: "Can Tree-based regression perform worse than plain linear regression?" Yes, of course it can. Tree-based models has as regression function a very complex step function. If the data truly comes from (behave as simulated from) a linear model, then step functions can be a bad approximation. And, as shown by examples in the other answer, tree-based models might extrapolate badly outside the range of the observed predictors. You could also try randomforrest and see how much better that is than a single tree.

  • 3
    $\begingroup$ Just to clarify: When you say "marginal distributions," you could also say looking at the univariate distributions of each variable, correct? They are "marginal" in the sense that the distributions would appear on the margins of a scatterplot or something. $\endgroup$
    – Mark White
    Commented Jun 21, 2017 at 13:05
  • $\begingroup$ Another question: You say "You should not just throw the data at different algorithms and look at the quality of the predictions." My question is: Why? If you are checking accuracy on a test data, that is. If we are more interested in prediction, then we don't need to worry about Type I error or anything like that that would be an issue if we were interested in statistical significance and multiple testing. $\endgroup$
    – Mark White
    Commented Jun 21, 2017 at 13:08
  • $\begingroup$ Even if you are only interested finally in the predictions, you will be in a better position to make a better models if you understand the data better. So, first, try to understand the data (and simple models fitted at the data) better, and then you are in a much better position to create more complex, and hopefully better, models. $\endgroup$ Commented Jun 21, 2017 at 13:10

Peter Ellis has a very simple example

Image uploaded from linked site

where linear regression performs better than regression trees, extrapolating beyond the observed values in the sample.

In this image the black points are the observed values, and the colored points are the predicted values. The actual data are generated according to a simple line with some noise, so linear regression and the neural network do a good job of extrapolating beyond the observed data. The tree based models do not.

Now, with 60 million data points you might not be worried about this. (The future always manages to surprise me though!) But it is an intuitive illustration as to one situation in which trees will fail.

  • $\begingroup$ thank you for intuitive answer. even though I have many data points, considering the characteristics of my data, I think it fails to extrapolate! $\endgroup$ Commented Jun 22, 2017 at 5:17
  • $\begingroup$ since NN based model performs better than linear regression. $\endgroup$ Commented Jun 22, 2017 at 5:17
  • $\begingroup$ one more question. Is 'hard to extrapolate' the common problem of non parameteric regression techniques? $\endgroup$ Commented Jun 22, 2017 at 7:34
  • $\begingroup$ Non-parametric is a wide net. To extrapolate you need to identify some underlying continuous function. Tree models are more like identifying many little steps, hence they do not follow the line outside the domain of the observed sample in this example. $\endgroup$
    – Andy W
    Commented Jun 22, 2017 at 13:05

It is a well known fact that trees are ill-suited to model truly linear relationships. Here is an illustration (Fig 8.7) from the ISLR book: Fig. 8.7

Top Row: A two-dimensional classification example in which the true decision boundary is linear, and is indicated by the shaded regions. A classical approach that assumes a linear boundary (left) will outperform a decision tree that performs splits parallel to the axes (right).

So if your dependent variable depends on the regressors in a more or less linear fashion, you would expect that "linear regression performs great".


Any decision-tree based approach (CART, C5.0, random forests, Boosted regression trees etc.) identify homogeneous areas in your data and assign the mean value of the data contained in that region to the corresponding 'leave'. So, they are granular and then, they must show a series of steps in the outputs. Those based on 'forests' do not show that phenomenon pronouncedly but it is still there. The aggregation of a large number of trees nuances it. When a given value is outside the original range the datum is assigned to the ‘leave’ that includes the extreme condition found in the training dataset and the output is consequently the mean value of the values contained in that leave. Thus, no extrapolation is possible. By the way ANNs are poor extrapolators. You can check: Pichaid Varoonchotikul - Flood Forecasting using Artificial Neural and Hettiarachchi et al. The extrapolation of artificial neural networks for the modelling of rainfall—runoff relationships they are very illustrative and are easy to find in the net! Good luck!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.