Help on R squared, Mean Squared Error (MSE), and/or RMSE as individual measures in regression model perfomance evaluation? Just a question on regression model evaluation statistics. Here we go.
I seem to be under the impression that $R^2$, MSE, and RMSE are all very closely related and essentially all play a part in determining the fit of a model, but somehow I am still confused about which one, then, I should be using as a final determinant of how good/bad my model is performing, or how is it that I can consider all three as separate indicators (with them being so similar)? 
It just seems like no matter what, the model with the $R^2$ closest to 1 also has the lowest MSE and RMSE values. I read that "sometimes" the $R^2$ doesn't determine the fit properly, and so the other metrics should be the determinants of fit, but if they all correlate (one goes up, the other goes up, and vice versa), is there even reason to separately consider them? (<- my bad if this seems redundant)
That's all I've got. Thanks for your time!
 A: Mean square error (MSE) and root mean square error (RMSE) obviously have a one-to-one relationship, as MSE can't be negative. Confusion can arise, however, from what MSE means in a particular context. As the Wikipedia page on mean squared error notes: 

In regression analysis, the term mean squared error is sometimes used to refer to the unbiased estimate of error variance: the residual sum of squares divided by the number of degrees of freedom. 

If that is what you have in mind for MSE then the standard $R^2$ reported for linear regression necessarily increases as MSE decreases, other things being equal, as it is defined as:
$$ R^2 \equiv 1 - {SS_{\rm res}\over SS_{\rm tot}}$$
where $SS_{\rm res}$ is the residual sum of squares and $SS_{\rm tot}$ is the total sum of squares.
Sometimes, however, MSE refers to the mean-squared error in predictions made on a new data set. It's quite possible for a regression model to overfit the data at hand by including too many predictors relative to the number of observations. In that case you could have very small MSE from your regression but large MSE when you apply the regression model to a new data set. The adjusted $R^2$ often reported by statistical software starts to take overfitting into account by adjusting for the numbers of predictors and observations.
One example of using MSE in this second context is to separate the data into a training set and a test set, develop the regression on the training set, and then evaluate MSE on the test set as a test of the quality of your regression model. That approach works best if you have very large numbers of observations. With smaller data sets, analogous approaches based on cross-validation or bootstrapping use MSE of predictions to serve a similar purpose.
