2
$\begingroup$

I am trying to build a random forest regression model in R using all continuous panel data. There is a large amount of data relative to the number of predictors. Say 500,000 rows and about 40 predictors.

In this case, one of the variables is a t-stat of the correlation coefficient from another regression. However the value given only exists if the t-stat is considered significant (> than a critical value). If the t-stat was not, it comes through as 0.

Looking at the data shows about half of the 500,000 rows are 0, and the rest are populated. I am unsure as to what effect a variable like this would have on a Random Forest Regression, with so many 0 values. Are there usually issues with accuracy, or does it simply mess up importance measures? Perhaps in this case the variable should be categorical? (i.e. signficant or not). Some initial google searches hasn't yielded me anything yet.

$\endgroup$
0
$\begingroup$

The own model will determine if the variable is important or not and if the best cut is >0 and ==0. Take into account that random forest will start creating nodes using the variables that give most variance to the model, so if your variable appears as one of the most important I will also add it twice: one as a dummy 0 if not relevant and 1 as significative. With so many data and features I dont think it will create any issues of multicollinearity (being a non linnear model) or overfitting. I think It could be a good excercise of feature engineering.

I will also consider xgboost if you are using R

Best!

$\endgroup$
  • $\begingroup$ Thanks LocoGris. So do you mean have the variable appear as two columns? Where one column is the actual value (which will be 0 or its value), and the other column is a categorical variable of 0 or 1 depending on whether or not it is signficant? $\endgroup$ – AdamC Mar 14 '19 at 21:16
  • $\begingroup$ Yes, if you have enough data and trees, it is not a problem and you have both lots of data and can produce lots of trees. Also, library xgboost could work even better $\endgroup$ – LocoGris Mar 14 '19 at 21:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.