I ran several different models on a mini data set of about 100 observations with 90 features. When I tried OLS with backward selection the model is significant with many features significant (82 features selected). However, when I tried to use the same data on LASSO, all parameters shrank to 0 except intercept and the MSE is higher than the one by OLS. The same happened for random forest, I got negative % variance explained (please see below) and MSE much higher than OLS.
Is this a typical case of overfitting? If so, why is that the backward selection of OLS failed to address this?
OLS:
step <- stepAIC(eye_lm, direction="both")
step$anova
Min 1Q Median 3Q Max
-0.63573 -0.14247 -0.01773 0.08343 0.99736
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -6.754e+00 3.560e+00 -1.897 0.090287 .
chroma_C_0 -1.903e+00 3.617e-01 -5.262 0.000519 ***
chroma_C_1 -1.322e+00 3.089e-01 -4.279 0.002052 **
chroma_C_2 2.824e-01 1.055e-01 2.676 0.025363 *
chroma_C_3 1.127e+00 3.398e-01 3.317 0.008990 **
chroma_C_4 6.200e-01 2.002e-01 3.097 0.012781 *
....
Topic0 1.938e+00 9.231e-01 2.100 0.065160 .
Topic1 7.327e+00 1.206e+00 6.075 0.000185 ***
Residual standard error: 0.7481 on 9 degrees of freedom
Multiple R-squared: 0.9821, Adjusted R-squared: 0.8209
F-statistic: 6.093 on 81 and 9 DF, p-value: 0.003054
LASSO: Y and X are the dependent and independent variables I used in OLS and I removed the intercept in the model matrix of X.
cv.out <- cv.glmnet(x,y,alpha= 1,family="gaussian",type.measure = "mse")
Rnadon Forest: for OLS and random forest, I used the same dependent variables and independent variables. I only changed the nPerm and nTree from default values but even if I use default values, I still get negative variance explained.
eye.rf = randomForest(Score ~syuzhet+ chroma_C_0 + chroma_C_1 + chroma_C_2 + chroma_C_3 +
chroma_C_4 + chroma_C_5 + chroma_C_6 + chroma_C_7 + chroma_C_8 +
chroma_C_9 + chroma_Q_0 + chroma_Q_1 + chroma_Q_2 + chroma_Q_3 +
chroma_Q_4 + chroma_Q_5 + chroma_Q_6 + chroma_Q_7 + chroma_Q_8 +
chroma_Q_9 + pitch_0 + pitch_1 + pitch_2 + pitch_3 + pitch_4 +
pitch_5 + pitch_7 + pitch_9 + pitch_10 + pitch_11 + pitch_12 +
pitch_13 + pitch_14 + pitch_15 + pitch_16 + pitch_17 + pitch_18 +
pitch_20 + pitch_21 + pitch_22 + pitch_23 + pitch_24 + MFCC_0 +
MFCC_3 + MFCC_4 + MFCC_5 + MFCC_6 + MFCC_7 + MFCC_8 + MFCC_10 +
MFCC_11 + MFCC_12 + MFCC_13 + MFCC_14 + if2017 + industry +
quarter + withCelebrities + withMusic + length.s. +
anger + anticipation + disgust + fear + joy + sadness + surprise +
trust + Topic0 + Topic1, mtry = 25,
nPerm = 10,
ntree = 6000,
data = ad)
Type of random forest: regression
Number of trees: 6000
No. of variables tried at each split: 25
Mean of squared residuals: 3.552812
% Var explained: -14.98
mtry
? That's the most important hyperparameter, because it controls the amount of correlation among trees; the default is $\lfloor\sqrt{\text{number of features}}\rfloor = 9$, but you've chosen 25 -- any particular reason? $\endgroup$ – Sycorax♦ Sep 4 '18 at 3:07