MSE is 'scale dependent'. R-squared seems a better measure of fit for regressions. Are there others? Mean-Squared Error is scale dependent. For example if I have an MSE of 0.1 and multiply all of X and Y by 100, redo my regression and calculate MSE, I get an MSE of 1000.0. ((y_true-y_regr)^2 ---> 100^2*(y_true-y_regr)^2)
Whilst MSE is very useful/powerful and has its own meaning, the fact in and of itself that its value is large or small does not necessarily give meaning in 'goodness of fit' in and of itself. Thus it is integral but has a slightly different interpretation.
R-squared has its own pros/cons but seems a better measure and 'normalised' to the data itself. Are there other 'universal' (non-scale dependent etc.) measures of 'goodness of fit'?
Would correlations or even mutual information between y_true and the predicted regression y_regr be useful for how well a regression of any type, including neural networks etc., fits the data it is trying to predict?
 A: I would argue that $R^2$ is not a universal "grade" for a model where $R^2=90\%=0.9$ is an $\text{A}$ that makes us happy and $R^2=40\%=0.4$ is an $\text{F}$ that makes us sad. It might be that $R^2=0.4$ is excellent performance for a task or that $R^2=0.9$ is rather pedestrian for a different task.
In that sense, I do not believe there to be an easy loss function that grades your model quality. To say that, you must know the costs of making wrong predictions. If you make a prediction that misses the true value by $3$, put that in context. If that is three meters when you are trying to measure how far away another town is, then I'd say that's pretty good. If you're trying to measure how tall someone is, such performance does not sound so good.
Likewise, if you get $R^2 = 0.9$, put that in context. If you have a large reduction in variance, it could be because you have small errors, or it could be that the original data had a huge variance, so even reducing the variance to be merely "large" gives you a high $R^2$, even though you need to have "modest" errors to have a useful model.
A: RMSE tells us how far the model residuals are from zero on average, i.e. the average distance between the observed values and the predicate values. However, Willmott et. al. suggested that RMSE might be misleading to assess the model performance since RMSE is a function of the average error and  the distribution of squared errors. Chai recommended to use both RMSE and mean absolute error (MAE). It is better to report both metrics. By the way, $R^2$ is misleading as well since it increases for higher number of predictors. I would recommend to use adjusted $R^2$. This metric is kinda gold standard goodness of fit test. Attached articles will give u more explanations.
Reference:
http://www.jstor.org/stable/24869236
https://gmd.copernicus.org/articles/7/1247/2014/
