2

Doing simple bagging (mtry = 5) will give you RF predictive performance (RMSE = 14.3) somewhat close to the linear model (RMSE = 11.4). Modern machine learning methods are data hungry. You probably need a bigger dataset to see the RF outperform good ol' linear regression. df <- structure( list( A = c(50L, 50L, 50L, 50L, 50L, 60L, 60L, 60L, 60L,...


2

A way to gauge, how useful a predictor $x_j$ is within a given model $M$ is by comparing the performance of the model $M$ with and without a predictor $x_j$ being included (say model $M^{-x_j}$). If we have multiple predictors though we are face with a situation we would have to create $p$ different $M^{-x_j}$ models going back and forth. The cost of this re-...


2

Sample without replacement, i.e. we get a scrambled/"permuted" version of $x_j$.


2

You need to code Seasons, Holiday, and Functioning_day as factors, but your data.frame() is resulting in character vectors for those variables: > str(x.new) 'data.frame': 1 obs. of 14 variables: $ Rainfall : num 0 $ Snowfall : num 0 $ Day : num 31 $ Month : num 4 $ Year : num 2018 $ Functioning_Day: chr &...


1

This isn't surprising. A model with more features has a richer space of functions to approximate as compared to a function with fewer features. Feature selection, in my own opinion, is not about increasing performance. On the contrary, it is about finding a set of features which does good enough as compared to the model with the full set. I wrote a small ...


1

Whether or not the importances of gradient boosting are strongly influenced by randomness depends on the hyper-parameter configuration of the gradient boosting model. A gradient boosted model which uses random subsampling of features (or other randomized components) will estimate feature importances which vary to a greater or lesser degree upon repeated ...


1

The idea of your test set is to help you objectively evaluate your model's performance on unseen data. If you selected your hyperparameters with CV (on the training set) and you have only run the test set once in the end for a final evaluation, you can consider it to depict your model's performance. If you are OK with this performance, you can retrain your ...


1

Gradient boosting in R (xgboost) is widely used to instead of running and combining many random forest models. Your question is not only about how to run models in parallel but also about how to combine different models. One easy way of doing the combination is just majority vote, i.e., treat each forest as a tree and do another level of random forest. This ...


1

point 1 is extremely relevant and I think is the best approach. It could be modelled as 3 or 4 classes: 3 classes: "YES" CHEAP; "YES" EXPENSIVE; "NO" (expensive and cheap) 4 classes: "YES" CHEAP; "YES" EXPENSIVE; "NO" CHEAP; "NO" EXPENSIVE="MAYBE" CHEAP After some reading I ...


1

Also, in XGBoost the default measure of feature importance is average gain whereas it's total gain in sklearn. See, https://xgboost.readthedocs.io/en/latest/python/python_api.html#module-xgboost.sklearn importance_type (string, default "gain") – The feature importance type for the feature_importances_ property: either “gain”, “weight”, “cover”, “...


1

I want to add some thoughts on the problem at hand, so that the discussion may roll on. However, I propose something else to think about, so others may comment on this. When reading this post and the post in the highlighted link, we try to overcome the bias in the RF (possibly in the tails) and to correct biased output of the RF, by applying another method, ...


Only top voted, non community-wiki answers of a minimum length are eligible