Why adding noisy predictors improves my random Forest prediction?

I am doing some tests with randomForest and I cannot understand the following results :

set.seed(12345)
library(caret);library(randomForest)
rmse_vec1<-rmse_vec2<-cor_vec1<-cor_vec2<-c()
for (i in 1:50)
{
df_train<-data.frame(x1=rnorm(1000),x2=rnorm(1000),x3=rnorm(1000),x4=rnorm(1000))
df_train$y<-jitter(df_train$x1+df_train$x3,amount=2) df_test<-data.frame(x1=rnorm(1000),x2=rnorm(1000),x3=rnorm(1000),x4=rnorm(1000)) df_test$y<-jitter(df_test$x1+df_test$x3,amount=2)

rf1<-randomForest(y~x1+x3,df_train) #First model
p1<-predict(rf1,newdata=df_test)

rf2<-randomForest(y~x1+x2+x3+x4,df_train) #Second model
p2<-predict(rf2,newdata=df_test)

cor_vec1<-c(cor_vec1,cor(p1,df_test$y)) cor_vec2<-c(cor_vec2,cor(p2,df_test$y))
rmse_vec1<-c(rmse_vec1,RMSE(p1,df_test$y)) rmse_vec2<-c(rmse_vec2,RMSE(p2,df_test$y))
}

length(which(cor_vec2>cor_vec1)) # 47
length(which(rmse_vec2<rmse_vec1)) # 42


In 94 % (for correlation) and 84 % (for RMSE) of tests, y ~ x1 + x2 + x3 + x4 is doing better than y ~ x1 + x3 even if y <- x1 + x3 + noise !

Could anyone explain me this result ?

1 Answer

probably because you're diluting your data with normally distributed (same mean and variance I assume) data which has 100% correlation to itself, which it turn improves your total correlation.

• Thank for your answer. Could you elaborate a little on this ? Because it is still unclear to me how it impacts my results. – MassCorr Sep 4 '18 at 14:04
• I just replaced every rnorm with runif in my code and it does not change anything. – MassCorr Sep 4 '18 at 14:07
• replace it with rnorm with different mean and variance for each variable, also maybe try increasing the amount of random variables – Gomunkul Sep 4 '18 at 14:08
• Indeed with different mean and var for each predictor y~x1+x3 is doing better than y~x1+x2+x3+x4. But I still dont understand clearly these results. – MassCorr Sep 4 '18 at 14:28