Is this the right way to evaluate MSE, RMSE, MAE, MAPE after creating a model on data passed through a standard scaler?

Question

Create the Scaler object

scaler = preprocessing.StandardScaler()

Fit your training data on the scaler object

scaled_X_train = scaler.fit_transform(training_data)

scaled_X_train = pd.DataFrame(scaled_X_train, columns = training_data.columns)

scaled_X_train.head()

Train your model here

Fit your testing data on the scaler object

scaled_X_test = scaler.fit_transform(testing_data)

scaled_X_test = pd.DataFrame(scaled_X_test, columns = testing_data.columns)

scaled_X_test.head()

Predict on test data using the trained model and the scaled test data

If y_test_actual is the scaled actual y &

If y_test_predicted is the scaled predicted y

Then would this be the right way to evaluate MSE, RMSE, MAE, MAPE

testing_mse = mean_squared_error(y_test_actual, y_test_predicted, squared=True)

testing_rmse = mean_squared_error(y_test_actual, y_test_predicted, squared=False)

testing_mae = mean_absolute_error(y_test_actual, y_test_predicted)

testing_mape = mean_absolute_percentage_error(y_test_actual, y_test_predicted)

Would these metrics be the same if the model was trained without performing any scaling?

The metrics would most likely be different. My question is how to get the right metrics when performing scaling of all features including the target variable ?

Answer 1

First of all, data scaling or normalization is useful for many models but there are cases where it's not needed. For example trees in general does not depend on scaling. The same should apply e.g. to gradient boosting methods, but there might be some special cases, see this link.

Regarding your question, metrics calculated on scaled data will be usually different. But when you use model which is not scale-dependent like the trees, the results (the performance) of the model should be in the same ballpark.

Note also the fact that some metrics are scale-dependent and some not. Typically squared and absolute based metrics (like MSE or MAE) will return different results. On the other percentage based errors (like MAPE) will not be influenced by scaling much (but depends on type of scaling, especially when you move "0").

I would recommend to try it by yourself. My recommendation would be following - use scaling for models that are scale-sensitive and report metrics in scaled/transformed space as in the original space. This might be easily achieved using inverse transform (see sklearn docs - you can do something like scaler.inverse_transform(y_test_predicted) and then calculate the metrics in the original space as well).

Advantage is that you can use scaled data for models which will work better in most cases and keep the original metrics as well (so you will see the real error for you data, e.g. what is true absolute error for temperature forecasting since its value is much useful than taking the one from scaled space).