So I'm trying to build a neural network to fit an approximating function to a data set.

The data set consists of (input,output) paris where the input features are discrete numbers [0,1,2,...,1027].

While the output is a single continous number between [0, 1027).

Now, If in preprocessing I normalize both my input features and outputs by dividing by the max value in the data set, my neural network which consists of a single hidden layer (relu) and a regression layer in the output learns well and I'm getting my MSE to drop to a nice level and I can plot and see that the NN fits well both the train set and the test set.

The problem is, when I de-normalize back the estimations to their [0,1027) range, the erros get multiplied by the max value (1027) and the overall results are now very poor.

Any tips on dealing with this kind of problem? how can I preform pre proccesing so my NN learn well, but not hurting performance when de-normalizing.



1 Answer 1


Any tips on dealing with this kind of problem? how can I preform pre proccesing so my NN learn well, but not hurting performance when de-normalizing.

You have done a good job of preprocessing the data by scaling it with Min and Max. You can also try subtracting the mean of the column and dividing with the standard deviation of the column. After preprocessing, do the train-test split of the data.

I want to point out that the conclusion that you have made that the performance is being hurt when de-normalising is not correct in the sense that the you are comparing same things (MSE for Normalized predictions vis-à-vis MSE for de normalized prediction) with very different scales.

The issues you have here are two:

A) Fine tuning your neural network. You can tune the hyperparameter of the neural network to achieve superior performance.

B) Coming up with an appropriate measure of accuracy. I would suggest a relative error measure. You can use Relative Squared Error(RSE). This measures the goodness of a prediction as compared to predicting the mean of the variable all the time. This measure can be used to compare models across data sets.

  • $\begingroup$ So basicly in (A) you are saying that I can train on my normalized data, and perform fine tuning at the end of training on the un-normalized original data? Do you have any refs on how to perform this procedure? In (B) do you mean that the RSE loss is obtained on the un-normalized data? meaning perform MSE on the un-normalized data and scale by sample distance from the mean? $\endgroup$
    – user3921
    Commented Jul 6, 2018 at 11:06
  • $\begingroup$ No. That's not what I stated. Fine tuning is a part of training the network. Must be done on either normalized or original data only. $\endgroup$
    – naive
    Commented Jul 6, 2018 at 11:12
  • $\begingroup$ Please consider accepting the answer if it served the purpose. Thanks! $\endgroup$
    – naive
    Commented Jul 7, 2018 at 5:22
  • $\begingroup$ Sorry, I don't quite get how and which fine tuning should I do, at which stage and on what $\endgroup$
    – user3921
    Commented Jul 8, 2018 at 11:04
  • $\begingroup$ I don't quite understand what do you mean by 'which fine tuning'. Fine tuning is the iterative procedure of finding the best possible combination of hyper parameters for the task at hand. $\endgroup$
    – naive
    Commented Jul 11, 2018 at 5:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.