Skip to main content
21 votes

Why do we use Linear Models when tree based models often work better than linear models?

Many excellent answers already. I would add a few more aspects. You do not say what you mean by "outperform", as in "tree models often outperform linear models". You presumably ...
Stephan Kolassa's user avatar
16 votes

Why do we use Linear Models when tree based models often work better than linear models?

For most problems it is easy to beat a tree model with a regression model. That's because tree models allow for all possible interactions among predictors and these are seldom needed. Interactions ...
Frank Harrell's user avatar
14 votes

Why do we use Linear Models when tree based models often work better than linear models?

Linear regression is also existent outside machine learning where it requires much less data. Why does Machine Learning need a lot of data while one can do statistical inference with a small set of ...
Sextus Empiricus's user avatar
13 votes

Why do we use Linear Models when tree based models often work better than linear models?

In addition to the excellent answers given already, a regression model yields parameter estimates, which tree models do not. If all you are interested in is prediction (which, as I understand it, is ...
Peter Flom's user avatar
  • 124k
9 votes

Why do we use Linear Models when tree based models often work better than linear models?

I share your observation that on well-behaved tabular data (quite many rows, not too many columns), a well-built boosted trees model usually beats a random forest, and a random forest usually beats a ...
Michael M's user avatar
  • 11.9k
8 votes

Is ROC curve unique?

Maybe I don't understand your notation, but I feel you are mixing up true probabilities, model outputs (which might be probability estimates) and events. given an event E, model output $f(\theta)$ (...
seanv507's user avatar
  • 7,075
7 votes
Accepted

Can a ML classifier's prediction be understood as a probability?

That would be desirable, but it is not guaranteed to make as much sense as we might like. First, you could make an argument that any predicted $p(\mathcal C_k|\mathbf x_i)\in[0,1]$ is a probability in ...
Dave's user avatar
  • 64.3k
6 votes
Accepted

Regression model with multiple rows per user to predict death

Since the response is mortality, the most appropriate method is survival analysis that explains time to death, a strictly positive value. It appears that the difference between the first ...
DrJerryTAO's user avatar
  • 1,774
5 votes

What is the best way to meaningfully compare the feature importances between parametric (Logit) and non-parametric models (RFs/XGB)?

This is tricky, because it depends in part on (i) what you mean by importance, and (ii) what you want out of such a comparison. To the first point, there are many ways to convert the somewhat vague ...
mkt's user avatar
  • 18.8k
5 votes
Accepted

How to interpret the results of a classifier when train/test method gives much better results than cross validated one?

What does these varying scores represent, particularly the low scores of cross validation? Together, they represent the fact that error estimtes based on a small number of tested cases are highly ...
cbeleites unhappy with SX's user avatar
4 votes
Accepted

How to interpret pairplots()

A pair plot is really just a scatter plot between all combinations of two variables in your dataset. This means you can essentially ignore one of the two triangles (the lower half is the same as the ...
PBulls's user avatar
  • 5,133
4 votes
Accepted

What is happening behind the scenes when we use CalibratedClassifierCV without prefit?

When we use cv=prefit, we will split the data into train, test and calibration sets, then fit a model using train sets, calibrate with calibration set and later use the calibrated model with the test ...
Ben Reiniger's user avatar
  • 4,723
3 votes

meaning of drop in OneHotEncoder

Implementation-wise, the drop keyword results in one category per column being dropped. Thus, out of ['Female', 'Male'], ...
chang_trenton's user avatar
3 votes

How reliable is ```train_test_split```? Is there a way to optimize it?

The aim of a train/test split is to estimate true model performance. To do this in an unbiased way, the test data needs to be independent of the training data. If rows are independent, then a random ...
Michael M's user avatar
  • 11.9k
3 votes

Scikit-learn : MultiOutputRegressor

They are completely different and unrelated things. MLPRegressor is a neural network model used for a regression problem. There are one or more target variables to ...
Tim's user avatar
  • 139k
3 votes

How to evaluate the results of a multilabel classifier using the predicted probabilities?

Good for you to look at the predicted probability values themselves. They contain useful information yet often get forgotten in favor of predicting the category with the highest probability, which can ...
Dave's user avatar
  • 64.3k
3 votes

AUC measure for Local outlier detection in python?

The cleanest way is to apply the logistic or sigmoid function; it works in the majority of the situation, including those when you don't have access to all the scores at once. ...
Rajesh's user avatar
  • 31
3 votes

Sklearn's LogisticRegression C hyperparameter issue

There are two components in the mentioned cost function: error and regularization. When you want to change the weight of one of the components by C times you can do one of the following: multiply ...
considerate_shrimp's user avatar
3 votes
Accepted

Am I finding redundant columns in my data using Factor Analysis

Your approach is correct, but it's important to consider the size of your dataset. The results and interpretations you obtain from your data can be greatly influenced by the number of samples you have....
Oscar Flores's user avatar
3 votes

Hierarchical clustering of a distance matrix with element weights

It sounds like this is an issue of geospatial projection. You’ll need to either transform your lat/lon into Cartesian coordinates, or modify your distance function to account for the areal distortion. ...
Randy Savage's user avatar
2 votes
Accepted

Implementations of Lasso in Python and R?

LASSO in Python With Python, in my experience, the most common implementation of LASSO (Least Absolute Shrinkage and Selection Operator) is provided by the ...
Robert Long's user avatar
  • 62.8k
2 votes

Scikitlearn: Why are hyperplane coefficients not available if kernel is not linear

Your intuition is on the right track. The usage of kernels is what makes SVMs so nice/convenient, since (very informally) it gives us the ability to build a classifier in some higher-dimensional space ...
chang_trenton's user avatar
2 votes

scikit learn: add lasso or ridge penalty only on subset of parameters

It is possible for ridge regression but not lasso as of scikit-learn 1.3.0. Compare the docs on the penalty weight alpha in ...
Jake Stevens-Haas's user avatar
2 votes

Representing Nested Models as an SKLearn Pipeline

Here is a hack to achieve what I am looking for. Specifically, add transform to the regression estimator directly. It is relatively lightweight but somewhat ...
chris's user avatar
  • 31
2 votes

Is my regularized logistic regression model overfit?

The sample size is not large enough to be able to reliably choose the shrinkage coefficient (penalty; regularization). The sample is barely large enough to estimate the overall prevalence of the ...
Frank Harrell's user avatar
2 votes

Is Linear Regression a good algorithm or even applicable with the distribution shown in the scatter plot I have shared in this question?

First, the linearity of linear regression is linearity in the coefficients, not in the predictors themselves or transformations of them. There's no need to restrict yourself to a single transformation ...
EdM's user avatar
  • 94.5k
2 votes

Finding the corners of noisy polygons

The following (using Mathematica) does not do what one would do "by eye" but that's because the data points don't fall perfectly on a desired number of line segments. This uses Mathematica's ...
JimB's user avatar
  • 3,940
1 vote

Why do we use Linear Models when tree based models often work better than linear models?

You can view xgboost as a stepwise linear regression adding nonlinear inputs. So it's more a question of how complicated you want to make the model
seanv507's user avatar
  • 7,075
1 vote

Scikitlearn: Why are hyperplane coefficients not available if kernel is not linear

What you're missing here is that the representation of kernels as the sums of nonlinear features is not known. What's known is that these kernel are likely to be representable as those new features, ...
Aksakal's user avatar
  • 61.7k
1 vote
Accepted

Why does the mean AURPC go down the more examples one uses?

I think the answer has to do with the probability of drawing a sample from x1=np.random.beta(a_neg, b_neg, n) being greater than a sample from ...
TheD's user avatar
  • 26

Only top scored, non community-wiki answers of a minimum length are eligible