9

@gunes has already given the answer, that random forests do work well with interactions. It may help to give a practical example, though. For that, we will need some toy data set.seed(2021) n = 5000 x1 <- rnorm(2*n) x2 <- runif(2*n) interaction <- x1 * x2 pseudointeraction <- sample(interaction) x3 <- rbeta(2*n, 2, 10) epsilon <- rt(2*n, df ...


8

Any monotonic injective transformation of the features won't change the model wrt how it splits the data. The reason is the same as for why scaling is unnecessary: the random forest looks for partitions, and partitions only depend on how the data are sorted. If there is an optimal split on some scale, then by the definition of monotonic injective, the same ...


7

Yes, tree based methods are good at detecting interactions, but not always. For example, using $x_1\lessgtr0$ and $x_2\lessgtr0$ in subsequent levels of the tree would be equivalent to using $x_1x_2\lessgtr0$ on one level. That said, since you set a max depth hyperparameter in random forests, adding promising interactions will decrease your overall depth and ...


6

Should predicting on your training set always throw out a very a high accuracy? Or would that indicate overfitting? No, not always. The reason some models perform better than others is due to the bias/variance tradeoff. Logistic regression is a linear model, and hence is high bias. A random forest is capable of learning non-linear effects (low boas) of ...


5

Yes, RFs' in-built OOB mse can be seen as an indicator for model performance. But you won't be able to compare its performance to different models (or different hyperparameters). Generally, you still want a "clean" hold-out set for validation. Train-test splits are often quite inaccurate for small data sets. Consider the bootstrap or repeated k-...


5

It's just a recommended default value. Leo Breiman observed that this value tends to work well on the classification problems that he worked on, but I'm not aware of any rigorous demonstration that this value must work best on all problems; indeed, the discovery that a different value works better on a specific problem seems to show that such a proof would ...


5

You could use multiple regression or random forest or both. Which will work best will depend a lot on your data and the associations among them.


4

There are several possible scenarios when one would think about calibrating probabilities: The model is misspecified or not optimally trained. That will be the case when non-linear relationships are modeled with a linear learner; or model is too rigid due to excessive regularization (model underfits); or to the contrary, the model is too flexible (overfit ...


4

A way to gauge, how useful a predictor $x_j$ is within a given model $M$ is by comparing the performance of the model $M$ with and without a predictor $x_j$ being included (say model $M^{-x_j}$). If we have multiple predictors though we are face with a situation we would have to create $p$ different $M^{-x_j}$ models going back and forth. The cost of this re-...


4

The first part of your conclusion is correct. Sklearn computes probabilities in the RFC by classifying subsamples and then averaging the proportion of observations in each class. This will bias probabilities that should be near 0 and 1 inward. This is simply because for a class to have a probability near 0 or 1 we need that many of the subsample classifiers ...


4

This is completely normal. The order of of the features for linear regression does not matter since addition commutes. $\beta_1 x_1 + \beta_2 x_2$ is the same as $\beta_2 x_2 + \beta_1 x_1$. This, in addition to the convex loss function, means that a single minimum exists and so it won't matter which order we arrange the features in. The reason you may ...


3

Let me copy and paste a warning message from sklearn permutation importance page Warning: Features that are deemed of low importance for a bad model (low cross-validation score) could be very important for a good model. Therefore it is always important to evaluate the predictive power of a model using a held-out set (or better with cross-validation) prior ...


3

Permutation importances would be suited for your goal, you can compute them as follows (example R code apended below): Fit a RF on a training dataset. Compute MSE on test observations. For each predictor variable, randomly permute the values of that variable in the test set and compute the MSE again. The difference between this MSE and the MSE computed ...


3

This would only work for generalized linear models (GLMs). With 'work' I mean that you get perfectly identical error estimates (e.g., train and test MSE) using the original predictors, and using a single predictor, computed by weighing each of the coefficients by their respective regression coefficients. Random forests are not linear models. Any deviations ...


3

Currently, this paper (doi:10.1177/0962280220946080) does a revision of previous algorithms, including those cited in previous answers. Further, that paper introduce the R library LongituRF, which allows to compute all those algorithms and the new ones.


3

There are some well developed algorithms that use trees to detect outliers. The key observation in these algorithms is that outliers correspond to short path lengths in fully developed trees. In other words, if you train a tree to leave only one sample at each leaf, you will notice that the path length (i.e. number of splits) of outliers is relatively small. ...


3

The two columns you see are the predicted probabilities for class 0 and class 1. The ROC result you have, the threshold is based on the positive probability. You can obtain the predicted label using a threshold of 0.53: ifelse(rf_prob_df[,2]>0.53,10) If the probability of 1 is 0.5 or say below 0.53, then the predicted class, with your new threshold, will ...


3

There are many fundamental issues you need to address first. Very important is whether you conceptualize this as a forced-choice classification task or as a continuous prediction task. "Classifier" denotes the former, and probability models are examples of the latter. See here. RF, since it can provide continuous outputs, is more in the ...


3

There is no advantage over using MSE or RMSE. Any model that has better $R^2$ than another model also will have better MSE and RMSE (assuming the same data). In that sense, all three are equivalent loss functions and measures of performance. A common reason for wanting to use $R^2$ over MSE or RMSE is the desire to say that $R^2=0.94$ means that $94\%$ of ...


3

The phrase "survival analysis" is usually taken to involve time-to-event data, not just event outcomes at a certain time. Otherwise, you're just throwing away potentially important time-to-event data. Also, survival analysis is well equipped to handle censored event times, for example individuals who still haven't experienced the event by the end ...


3

Random forest does this by default. Random forest predictions are the average of the terminal node values. The average of nonnegative values is also nonnegative.


3

No, that's not how feature importance works. The feature importance does not understand the classes as treatments. Feature importance essentially measures how well each feature can be used to construct a split that divides the data into the classes. The feature importance does not describe one class individually. You can verify this by fitting a random ...


3

The C-index is the fraction of pairs of comparable cases in which the observed event times are in the same order as the predicted. If survival curves can cross over time, then the order based on "predicted survival probabilities" can change depending on the choice of time point to calculate the probability. Here's an example of survival curves for ...


3

Yes, out-of-bag error is an estimate of the error rate (which is 1 - accuracy) that this training approach has for new data from the same distribution. This estimate is based on the predictions that you get for each data point by using only averaging those trees, for which the record was not in the training data. If you have a low number of trees the OOB ...


3

People measure predictive accuracy on the test set and not on the training set for good reasons. If both models deliver 65% on the test set, I'd say they're both of pretty much the same prediction quality, and the fact that the Random Forest (RF) has a much higher value on the training set wouldn't bother me much. It does say that the RF overfits the ...


2

If your test data has some levels that are an insignificant percentage of the population, but are just messing up your ability to make the model, here is an elegant way to remove the extraneous levels from the test set: uniquetrain <- unique( train$category) test <- test[test$category %in% uniquetrain,]


2

Random forest fits multi-dimensional staircase to your data. It produces sharp edges in predictions. If your data are of continuous nature, then probably there are better methods to fit them. That doesn't mean however that "you should avoid random forest to fit them" :-) I don't think there is any kind of data where you "should be worried"...


2

Answered in a comment by Ryan Bressler. RFs were introduced to deal with issues of overfitting in normal decision trees but this doesn't happen on every data set. More then the number of independent variables in depends on the number of possible splits that can be made with each variable and other things that can bias greedy decision tree learning and cause ...


2

Instead of random forest, you can also use tree-boosting for the fixed effects part in a model with random effects. The GPBoost library with Python and R packages builds on LightGBM and allows for combining tree-boosting and mixed effects models. Simply speaking it is an extension of linear mixed effects models where the fixed-effects are learned using tree-...


2

Q&A Should sample_weight and class_weight be used together simultaneously If your goal is to weight your classes because they are imbalanced, you can use either. Using class_weight="balanced is the same as sample_weight=[n_samples]. I tested it with an unbalanced set in kaggle. I estimated the "sample_weight" based on what was given in ...


Only top voted, non community-wiki answers of a minimum length are eligible