I ran several different models on a mini data set of about 100 observations with 90 features. When I tried OLS with backward selection the model is significant with many features significant (82 features selected). However, when I tried to use the same data on LASSO, all parameters shrank to 0 except intercept and the MSE is higher than the one by OLS. The same happened for random forest, I got negative % variance explained (please see below) and MSE much higher than OLS.
Is this a typical case of overfitting? If so, why is that the backward selection of OLS failed to address this?
step <- stepAIC(eye_lm, direction="both") step$anova Min 1Q Median 3Q Max -0.63573 -0.14247 -0.01773 0.08343 0.99736 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -6.754e+00 3.560e+00 -1.897 0.090287 . chroma_C_0 -1.903e+00 3.617e-01 -5.262 0.000519 *** chroma_C_1 -1.322e+00 3.089e-01 -4.279 0.002052 ** chroma_C_2 2.824e-01 1.055e-01 2.676 0.025363 * chroma_C_3 1.127e+00 3.398e-01 3.317 0.008990 ** chroma_C_4 6.200e-01 2.002e-01 3.097 0.012781 * .... Topic0 1.938e+00 9.231e-01 2.100 0.065160 . Topic1 7.327e+00 1.206e+00 6.075 0.000185 *** Residual standard error: 0.7481 on 9 degrees of freedom Multiple R-squared: 0.9821, Adjusted R-squared: 0.8209 F-statistic: 6.093 on 81 and 9 DF, p-value: 0.003054
LASSO: Y and X are the dependent and independent variables I used in OLS and I removed the intercept in the model matrix of X.
cv.out <- cv.glmnet(x,y,alpha= 1,family="gaussian",type.measure = "mse")
Rnadon Forest: for OLS and random forest, I used the same dependent variables and independent variables. I only changed the nPerm and nTree from default values but even if I use default values, I still get negative variance explained.
eye.rf = randomForest(Score ~syuzhet+ chroma_C_0 + chroma_C_1 + chroma_C_2 + chroma_C_3 + chroma_C_4 + chroma_C_5 + chroma_C_6 + chroma_C_7 + chroma_C_8 + chroma_C_9 + chroma_Q_0 + chroma_Q_1 + chroma_Q_2 + chroma_Q_3 + chroma_Q_4 + chroma_Q_5 + chroma_Q_6 + chroma_Q_7 + chroma_Q_8 + chroma_Q_9 + pitch_0 + pitch_1 + pitch_2 + pitch_3 + pitch_4 + pitch_5 + pitch_7 + pitch_9 + pitch_10 + pitch_11 + pitch_12 + pitch_13 + pitch_14 + pitch_15 + pitch_16 + pitch_17 + pitch_18 + pitch_20 + pitch_21 + pitch_22 + pitch_23 + pitch_24 + MFCC_0 + MFCC_3 + MFCC_4 + MFCC_5 + MFCC_6 + MFCC_7 + MFCC_8 + MFCC_10 + MFCC_11 + MFCC_12 + MFCC_13 + MFCC_14 + if2017 + industry + quarter + withCelebrities + withMusic + length.s. + anger + anticipation + disgust + fear + joy + sadness + surprise + trust + Topic0 + Topic1, mtry = 25, nPerm = 10, ntree = 6000, data = ad) Type of random forest: regression Number of trees: 6000 No. of variables tried at each split: 25 Mean of squared residuals: 3.552812 % Var explained: -14.98