when is it possible to have OLS fits better than random forest and LASSO?

Question

I ran several different models on a mini data set of about 100 observations with 90 features. When I tried OLS with backward selection the model is significant with many features significant (82 features selected). However, when I tried to use the same data on LASSO, all parameters shrank to 0 except intercept and the MSE is higher than the one by OLS. The same happened for random forest, I got negative % variance explained (please see below) and MSE much higher than OLS.

Is this a typical case of overfitting? If so, why is that the backward selection of OLS failed to address this?

OLS:

step <- stepAIC(eye_lm, direction="both")
step$anova

    Min       1Q   Median       3Q      Max 
    -0.63573 -0.14247 -0.01773  0.08343  0.99736 

  Coefficients:
                            Estimate Std. Error t value Pr(>|t|)    
  (Intercept)                   -6.754e+00  3.560e+00  -1.897 0.090287 .  
 chroma_C_0                    -1.903e+00  3.617e-01  -5.262 0.000519 ***
 chroma_C_1                    -1.322e+00  3.089e-01  -4.279 0.002052 ** 
 chroma_C_2                     2.824e-01  1.055e-01   2.676 0.025363 *  
 chroma_C_3                     1.127e+00  3.398e-01   3.317 0.008990 ** 
 chroma_C_4                     6.200e-01  2.002e-01   3.097 0.012781 *  
  ....
 Topic0                         1.938e+00  9.231e-01   2.100 0.065160 .  
 Topic1                         7.327e+00  1.206e+00   6.075 0.000185 ***

Residual standard error: 0.7481 on 9 degrees of freedom
Multiple R-squared:  0.9821,    Adjusted R-squared:  0.8209 
F-statistic: 6.093 on 81 and 9 DF,  p-value: 0.003054

LASSO: Y and X are the dependent and independent variables I used in OLS and I removed the intercept in the model matrix of X.

 cv.out <- cv.glmnet(x,y,alpha= 1,family="gaussian",type.measure = "mse")

Rnadon Forest: for OLS and random forest, I used the same dependent variables and independent variables. I only changed the nPerm and nTree from default values but even if I use default values, I still get negative variance explained.

 eye.rf = randomForest(Score ~syuzhet+ chroma_C_0 + chroma_C_1 + chroma_C_2 + chroma_C_3 + 
                    chroma_C_4 + chroma_C_5 + chroma_C_6 + chroma_C_7 + chroma_C_8 + 
                    chroma_C_9 + chroma_Q_0 + chroma_Q_1 + chroma_Q_2 + chroma_Q_3 + 
                    chroma_Q_4 + chroma_Q_5 + chroma_Q_6 + chroma_Q_7 + chroma_Q_8 + 
                    chroma_Q_9 + pitch_0 + pitch_1 + pitch_2 + pitch_3 + pitch_4 + 
                    pitch_5 + pitch_7 + pitch_9 + pitch_10 + pitch_11 + pitch_12 + 
                    pitch_13 + pitch_14 + pitch_15 + pitch_16 + pitch_17 + pitch_18 + 
                    pitch_20 + pitch_21 + pitch_22 + pitch_23 + pitch_24 + MFCC_0 + 
                    MFCC_3 + MFCC_4 + MFCC_5 + MFCC_6 + MFCC_7 + MFCC_8 + MFCC_10 + 
                    MFCC_11 + MFCC_12 + MFCC_13 + MFCC_14 + if2017 + industry + 
                    quarter + withCelebrities + withMusic + length.s.  + 
                    anger + anticipation + disgust + fear + joy + sadness + surprise + 
                    trust + Topic0 + Topic1, mtry = 25, 
                  nPerm = 10,
                  ntree = 6000,
                  data = ad)

 Type of random forest: regression
                 Number of trees: 6000
  No. of variables tried at each split: 25

      Mean of squared residuals: 3.552812
                % Var explained: -14.98

Answer 1

Your analysis would be much better if it involved a common cross-validation or validation schema. Nevertheless, you are on the right track: Since random forests are typically quite strong (especially in large p small n situation) even without tuning, getting a bad out-of-bag score is often a sign that something is fishy about other methods with much better results (e.g. OLS with backward elimination and an R-squared of 90%).

Let's demonstrate with a classic example based on completely random data. So everything above 0% R-squared (adjusted) would come from overfit.

# Random data
n <- 100
p <- 90
set.seed(3423)
X <- data.frame(matrix(rnorm(n * (p + 1)), nrow = n, ncol = p + 1))

# Formula
form <- reformulate(colnames(X[-1]), response = "X1")

# OLS
ols <- lm(form, data = X)
ols_after_be <- MASS::stepAIC(ols, direction = "both", trace = FALSE)
summary(ols_after_be) # Multiple R-squared:  0.8937,    Adjusted R-squared:  0.6812 

# RF
rf <- randomForest::randomForest(form, data = X)
rf # % Var explained: -8.01% (depends on run)

So we can clearly see that (adjusted) R-squared of OLS followed by variable selection is super promising 68% (yeah, let's publish!). On the other hand, the out-of-bag R-squared of the random forest is close to 0%.

Regarding the many comments below the OP:

• OOB scores are not as "valid" as cross-validation scores, but if the observations are independent and not too few trees are run, they are often sufficiently reliable.

• There are different defaults for "mtry" in a random forest in regression settings. One is $p/3$, another might be $\sqrt{p}$. In different real world situations, usually using values like this provides close to optimal cross-validation scores. The value chosen in the OP is somewhere in the middle, so it can't be too bad.

Take-home message: Stepwise variable selection is a very naive and bad solution to a large p small n problem. In this example, the original OLS (before any variable selection) yields a negative adjusted R-squared, which is completely fine. The problem is the variable selection part, not the OLS part.