RMSE - where this evaluation metric came from? Does anyone know where this metric came from ?
Can someone bring article references or something like this? 
Im actually wondering if there's any mathematical concept or any way to demonstrate mathematically that this metric is consequence from any model. 
 A: RMSE arises from what is probably the most important model in statistics, linear regression. A linear regression model is fit with least squares, which means minimizing the mean square error (MSE) for the sample. Take the square root of MSE, so that it's on the same scale as the data and hence easier to interpret, and you get RMSE.
If that just makes you wonder why we fit linear regression models with least squares, here's how Wikipedia puts it:

The OLS [ordinary least squares] estimator is consistent when the regressors are exogenous and there is no perfect multicollinearity, and optimal in the class of linear unbiased estimators when the errors are homoscedastic and serially uncorrelated. Under these conditions, the method of OLS provides minimum-variance mean-unbiased estimation when the errors have finite variances. Under the additional assumption that the errors be normally distributed, OLS is the maximum likelihood estimator.

That's a lot of nice properties.
A: The method of least squares was first developed by French mathematician Adrien-Marie Legendre and published in 1806.
The broad question at the time was how to combine multiple observations of celestial bodies to estimate the parameters that governed their motion.


*

*More observations than parameters creates an overdetermined system of equations.

*An overdetermined system of equations generally cannot be solved. What instead should one do?

*Early work averaged observations together until there were as many observations as parameters, creating a solveable linear system. Legendre instead proposed minimizing the sum of square error.


Later, Gauss connected the method of least squares to his work on the normal distribution, showing that least squares is in a sense optimal with a linear equation and normally distributed errors.
For the full history, see Stephen Stigler's, The History of Statistics: Measurement of Uncertainty before 1900
