Comparing two models in R I'm using R to develop regression models, and I need to compare two different models' performance. The question that arises is, "Is Model 1 statistically better than model 2?" and I don't seem to have a way to answer that question.
Background: Model 1 consists of Variable A regressed on the endpoint. Model 2 consists of Variables B, C, and D regressed on the endpoint. Both models are developed using lm - ordinary least squares, nothing too fancy here.
Given that these are not nested models, I cannot compare them using ANOVA.
I can look at the R2 of actual vs predicted for each model, and I see that Model 2 is better, but how do I determine if it is statistically significantly better?
I've also used the Concordance Correlation Coefficient, but again, I can't find a way to prove significance. The best I've come up with is that the rho for Model 2 is better than Model 1, but that rho value is within the 95% confidence limits of the rho for Model 1.
I should throw in there that my assessment of predicted vs actual has been on a 60 observation hold-out set (240 observations in the training set).
 A: You need to look at the adjusted $R^2$, not just $R^2$, since you are estimating models with a different number of parameters.  
That said, I would argue that "Is Model 1 statistically better than Model 2?" is too vague a statement—what is "better"? A lower residual sum of squares? Does $R^2$ really tell you which is "better"? What does the domain science say? Have you checked that both meet assumptions of OLS linear regression?  
Remember George Box's famous words "All models are wrong, but some are useful." Which model would be most useful in your circumstances?  In every field of science, simple models are often the most powerful for description, whereas a more complicated model may give better predictions (assuming due diligence in preventing overfit has been performed).      
I would use AIC to compare the candidate models, but there are no associated tests.
Perhaps consider the Cox test (in the lmtest package of R). There is nothing wrong with choosing the lower adjusted $R^2$, but again, this is heuristic in nature. If you wanted to really do it right, you should collect a testing set of data to validate the models and see which is more accurate in real world prediction.    
You should also consider collinearity. If B, C and/or D have significant correlation structure, the second model is invalidEDIT: as per Scortchi's comment, the model is still valid, but standard errors will increase drastically. I would still consider a correlation structure between B,C,D as evidence towards the first model having more utility under most use cases. /EDIT Furthermore, if A is collinear with B, C, and/or D, it gives reason to believe A is a latent variable being measured by B, C, and/or D; in which case the first model is probably a better choice. Would be interested to see if you have attempted a regression model with A as the dependent variable and B, C, and D as predictors. 
Overall, you should view model selection as an "art", rather than look for a process to give you a p-value on which is best.
