I ran several different models on a mini data set of about 100 observations with 90 features. When I tried OLS with backward selection the model is significant with many features significant (82 features selected). However, when I tried to use the same data on LASSO, all parameters shrank to 0 except intercept and the MSE is higher than the one by OLS. The same happened for random forest, I got negative % variance explained (please see below) and MSE much higher than OLS.

Is this a typical case of overfitting? If so, why is that the backward selection of OLS failed to address this?


step <- stepAIC(eye_lm, direction="both")

    Min       1Q   Median       3Q      Max 
    -0.63573 -0.14247 -0.01773  0.08343  0.99736 

                            Estimate Std. Error t value Pr(>|t|)    
  (Intercept)                   -6.754e+00  3.560e+00  -1.897 0.090287 .  
 chroma_C_0                    -1.903e+00  3.617e-01  -5.262 0.000519 ***
 chroma_C_1                    -1.322e+00  3.089e-01  -4.279 0.002052 ** 
 chroma_C_2                     2.824e-01  1.055e-01   2.676 0.025363 *  
 chroma_C_3                     1.127e+00  3.398e-01   3.317 0.008990 ** 
 chroma_C_4                     6.200e-01  2.002e-01   3.097 0.012781 *  
 Topic0                         1.938e+00  9.231e-01   2.100 0.065160 .  
 Topic1                         7.327e+00  1.206e+00   6.075 0.000185 ***

Residual standard error: 0.7481 on 9 degrees of freedom
Multiple R-squared:  0.9821,    Adjusted R-squared:  0.8209 
F-statistic: 6.093 on 81 and 9 DF,  p-value: 0.003054

LASSO: Y and X are the dependent and independent variables I used in OLS and I removed the intercept in the model matrix of X.

 cv.out <- cv.glmnet(x,y,alpha= 1,family="gaussian",type.measure = "mse")

Rnadon Forest: for OLS and random forest, I used the same dependent variables and independent variables. I only changed the nPerm and nTree from default values but even if I use default values, I still get negative variance explained.

 eye.rf = randomForest(Score ~syuzhet+ chroma_C_0 + chroma_C_1 + chroma_C_2 + chroma_C_3 + 
                    chroma_C_4 + chroma_C_5 + chroma_C_6 + chroma_C_7 + chroma_C_8 + 
                    chroma_C_9 + chroma_Q_0 + chroma_Q_1 + chroma_Q_2 + chroma_Q_3 + 
                    chroma_Q_4 + chroma_Q_5 + chroma_Q_6 + chroma_Q_7 + chroma_Q_8 + 
                    chroma_Q_9 + pitch_0 + pitch_1 + pitch_2 + pitch_3 + pitch_4 + 
                    pitch_5 + pitch_7 + pitch_9 + pitch_10 + pitch_11 + pitch_12 + 
                    pitch_13 + pitch_14 + pitch_15 + pitch_16 + pitch_17 + pitch_18 + 
                    pitch_20 + pitch_21 + pitch_22 + pitch_23 + pitch_24 + MFCC_0 + 
                    MFCC_3 + MFCC_4 + MFCC_5 + MFCC_6 + MFCC_7 + MFCC_8 + MFCC_10 + 
                    MFCC_11 + MFCC_12 + MFCC_13 + MFCC_14 + if2017 + industry + 
                    quarter + withCelebrities + withMusic + length.s.  + 
                    anger + anticipation + disgust + fear + joy + sadness + surprise + 
                    trust + Topic0 + Topic1, mtry = 25, 
                  nPerm = 10,
                  ntree = 6000,
                  data = ad)

 Type of random forest: regression
                 Number of trees: 6000
  No. of variables tried at each split: 25

      Mean of squared residuals: 3.552812
                % Var explained: -14.98
  • 1
    $\begingroup$ The predictions from your random forest have higher variance than the data itself? That points to using the model incorrectly. Can you supply more details on how you fit the lasso and random forest models? $\endgroup$ Commented Sep 4, 2018 at 1:53
  • $\begingroup$ I added more details $\endgroup$
    – lll
    Commented Sep 4, 2018 at 2:12
  • 1
    $\begingroup$ "I only changed the nPerm and nTree from default values but even if I use default values, I still get negative variance explained." What about mtry? That's the most important hyperparameter, because it controls the amount of correlation among trees; the default is $\lfloor\sqrt{\text{number of features}}\rfloor = 9$, but you've chosen 25 -- any particular reason? $\endgroup$
    – Sycorax
    Commented Sep 4, 2018 at 3:07
  • $\begingroup$ not for any particular reason.. i tried a bunch of different numbers of mtry from 2,5,default, up to 25 and all gave me negative variance explained. ... could it be possible that the data itself it just so sad that LASSO and random forest just cannot fit but why does OLS fit in this case then? that's what made me very confused $\endgroup$
    – lll
    Commented Sep 4, 2018 at 4:38
  • $\begingroup$ How does your backward selection make decisions? You provide little details there. $\endgroup$ Commented Sep 4, 2018 at 8:55

1 Answer 1


Your analysis would be much better if it involved a common cross-validation or validation schema. Nevertheless, you are on the right track: Since random forests are typically quite strong (especially in large p small n situation) even without tuning, getting a bad out-of-bag score is often a sign that something is fishy about other methods with much better results (e.g. OLS with backward elimination and an R-squared of 90%).

Let's demonstrate with a classic example based on completely random data. So everything above 0% R-squared (adjusted) would come from overfit.

# Random data
n <- 100
p <- 90
X <- data.frame(matrix(rnorm(n * (p + 1)), nrow = n, ncol = p + 1))

# Formula
form <- reformulate(colnames(X[-1]), response = "X1")

ols <- lm(form, data = X)
ols_after_be <- MASS::stepAIC(ols, direction = "both", trace = FALSE)
summary(ols_after_be) # Multiple R-squared:  0.8937,    Adjusted R-squared:  0.6812 

# RF
rf <- randomForest::randomForest(form, data = X)
rf # % Var explained: -8.01% (depends on run)

So we can clearly see that (adjusted) R-squared of OLS followed by variable selection is super promising 68% (yeah, let's publish!). On the other hand, the out-of-bag R-squared of the random forest is close to 0%.

Regarding the many comments below the OP:

  • OOB scores are not as "valid" as cross-validation scores, but if the observations are independent and not too few trees are run, they are often sufficiently reliable.

  • There are different defaults for "mtry" in a random forest in regression settings. One is $p/3$, another might be $\sqrt{p}$. In different real world situations, usually using values like this provides close to optimal cross-validation scores. The value chosen in the OP is somewhere in the middle, so it can't be too bad.

Take-home message: Stepwise variable selection is a very naive and bad solution to a large p small n problem. In this example, the original OLS (before any variable selection) yields a negative adjusted R-squared, which is completely fine. The problem is the variable selection part, not the OLS part.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.