Can R-square be a criterion of simple linear models?

Question

For example, I constructed three simple linear models, say

Y ~ A

Y ~ B

Y ~ C

A, B and C are highly correlated

Now I have their R-squares and P-values, can I say that one with highest R-square and lowest P-value is better than others? If can't, what criterion should I use? My background is ecology BTW, I seldom got R-squares higher than 0.3

Answer 1

$R^2$ is a function of mean squared error (MSE), a value that we hope is small.

$$ R^2=\dfrac{ \sum_i\big( y_i-\hat y_i \big)^2 }{\sum_i\big( y_i-\bar y \big)^2}\\=\dfrac{ nMSE }{\sum_i\big( y_i-\bar y \big)^2} $$

Whichever model has the highest $R^2$ also has the lowest MSE. Consequently, there is a sense in which that model has the strongest performance, yes.

(Most likely, however, you would be able to use all three features in one regression model that beats any of the three simple linear regressions.)

Because of this setup where you have the same $y$ and same sample size (really, it’s the latter than matters), the highest $R^2$ will correspond to the lowest p-value on the slope parameter, as the p-value is a function of the MSE and the sample size.