Does a scaled multi output neural network result in a scaled RMSE?

Question

I've scaled my data to be in the range 0-1. I've used this scaled data to train a deep neural network,

model.fit(xscaled,yscaled, verbose=0,epochs=180, batch_size=70, validation_split=0.2)

I've then used the sklearn rmse function to calculate rmse.

print('rmse',sqrt(metrics.mean_squared_error(y_test,predictions)))

which has given me a value of 0.097. The data set I'm using has 22 input values, with 3 outputs. Im struggling to understand the single rmse value, considering I have 3 outputs. Is the rmse scaled because I scaled the data? I.e. because the data is scaled from 0-1, and I have an rsme of 0.097, this results in an error of 9.7%? I understand that rsme measures depends on the range of data to how 'good' the model is, but considering I have a multi output model with different ranges for the 3 outputs im not sure how this works. Thank you.

Answer 1

First, RMSE is not equal to relative error so RMSE of 0.09 does not mean you made 9% error on average. (which is the mean relative error)

---

Then, as you mentioned, RMSE is scale-dependent. When dealing with multi-output regression, RMSE is computed for each output and then averaged. It is clear that you have to normalize outputs if you want global RMSE to be equally representative of each target's RMSE. You could also use relative RMSE : \begin{equation} RMSE\% = \frac{RMSE}{\bar{y}} \end{equation}

Wich represents RMSE relative to mean target value and enables you to compare and average errors on different targets.

However, I don't recommend averaging RMSE of the outputs unless it is done for models comparison purposes and averaging weights have been properly adjusted.