Which is a better determinant of linear regression performance: the RMSE, R-squared or significance of coefficients?

Question

I have created 2 linear regression models on a data set and its extract (I removed some features from teh first data set for this second model as the number of samples was too small). None of them gives significant coefficients: Model1:

Residuals:
    Min      1Q  Median      3Q     Max 
-42.812 -25.919  -3.612  12.394  70.756 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept) 66.9098816 44.4508422   1.505    0.150
x1          0.0332949  0.4990344   0.067    0.948
x2         -4.1744552  6.7686418  -0.617    0.545
x3         -2.0553138 11.1224723  -0.185    0.855
x4         -4.0994890 15.9008763  -0.258    0.799
x5         -0.0068639  0.0798893  -0.086    0.932
x6         -0.1989496  2.2244522  -0.089    0.930
x7         -0.0103695  0.0459473  -0.226    0.824
x8          0.0009110  0.0010026   0.909    0.376
x9         -0.0001616  0.0006995  -0.231    0.820

Residual standard error: 38.12 on 18 degrees of freedom
Multiple R-squared:  0.1323,    Adjusted R-squared:  -0.3015 
F-statistic: 0.305 on 9 and 18 DF,  p-value: 0.9633

And for Model2 I have:

Residuals:
   Min     1Q Median     3Q    Max 
-47.48 -26.48  -7.77  16.99  75.94 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)  
(Intercept) 77.73117   36.14409   2.151   0.0428 *
x1          0.04839    0.37977   0.127   0.8998  
x2         -7.12185    5.57618  -1.277   0.2148  
x3         -2.23965   12.26234  -0.183   0.8567  
x4         -0.03269    0.06258  -0.522   0.6066  
x5         -0.01020    0.03985  -0.256   0.8004  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 35.42 on 22 degrees of freedom
Multiple R-squared:  0.08402,   Adjusted R-squared:  -0.1242 
F-statistic: 0.4036 on 5 and 22 DF,  p-value: 0.8411

I have 28 samples in my data set and I know it's better to use cross-validation. But I was wondering which of these models are better model in general? Can I rely on a model if the RMSE is small compared to the range (or mean) of the output variable but none of the coefficients of the variables is starred (significant)?

Answer 1

You are comparing two models with different numbers of parameters. For the comparison, all three quantities are not much meaningful because of the possibility of overfitting. Your model would get better and better if you put more variables but it would have poor prediction power with new data. This is the concept of overfitting.

When you compare two models with the different numbers of parameters, you need to compare the statistics which penalizes the number of parameters. And the adjusted R-squared is the quantity satisfying the goal. Higher adjusted R-squared is better as the ordinary R-squared. So the second model is better than the first model.

But the R-squared value is so poor already. Before you compare the two models, you better consider if the linear model is the right choice or if you forget any other important variable.