Random Forest Regression - R^2 score or MSE for Comparison I trained a Random Forest Model for Regression and till now I compared the R^2 Score between the different trained models, but as I have read a few articles that the R^2 Score might not be the best to compare the different models I thought about doing it with the RMSE of the model.
What are your thoughts about doing it this way and is there any better way of compare models against each other?
How can I test if my prediciton is close to the desired value, which number would be better than the R^2 Score?
Thank you,
R
 A: There is no real difference in discrimination between the mean squared error and the R-squared, when it is used on a test dataset as the R-Squared is just the MSE divided by the MSE of the prediction of the mean, which is a constant. So MSE and R-Squared are proportional to each other and the ranking will never be different between these two measures. 
This is not the case if the measures are averaged across different test dataset, as then the denominators of the R-Squared between the test sets are different. In this case, e.g. in 5-fold cross-validation, I would rather use the MSE. On the other hand, if the test datasets are not commensurable I would rather use the R-Squared, because it is rather comparable between inherently different datasets. 
I would also look at ranking measures like Spearman rho and Kendalls tau, as they are more robust against outliers than MSE and R-squared. Or at the median absolute error, or the median squared error. 
A: The r2_score is a statistical description of how your samples fit along a linear model. It only works well if there is a linear relationship between features and outcome (with few outliers). 
Instead you should calculate the mean-absolute-error or mean-squared-error using labeled samples from the testset, that your model did not see during training.
