normalise/denormalise data in in neuralnet prediction modell

Question

I built a neuralnet model to do predictions from measured data. I have a conceptual problem in understandinig how to normalise/denormalise my data in the "prediction-mode". Here is what I did: I used a dataset which I normalised with the min/max method: normalized = (x-min(x))/(max(x)-min(x)) The I splitted the dataset and used 80% of the data to train model and 20% to validate the model. This worked all fine. Only working with this initial dataset makes it also easy to "denormalise" the data to their initial values using this formula: denormalized = (normalized)*(max(x)-min(x))+min(x)

formulas are taken from here: De normalize predicted value

Now when I use the model to do predictions in realtime, I have difficulties in understanding how normalise/denormalise my data. I have every day new measured date from whcih I want to do a prediction. Do I have to use the min/max values that I gathered from my inital dataset, or do calculate every day new min/max values for the daily dataset to feed into the neuralnet model? I have tried both ways, and both results are not too good... When I use the min/max values from the training data, the results becomes a bit better (less noisy). For the model output I have to use the min/max values from the training, as I do not have any other data. Or is there a conceptual error in my thinking? Any help is welcome.. Thanks a lot in advance, Arne

Answer 1

Ideally you are supposed to save the min/max you used to normalize the training data; so that you can re-use it for test data/live data. DO NOT re-normalize the entire train + new data again because that is leakage.

Use the min/max of train data to normalize the test data

Answer 2

My best guess would be the following:

Since the "new" instances have not been there during the original normalization phase. One possible action is to include the new instances as bottom rows to your original dataset, re-normalize and then re-train your model. Doing so, you will adjust the relative location of the new instances with respect to the original dataset.

If your model still does not perform well in terms of prediction power, you might be experiencing an "overfitting" problem, which is a common problem in predictive models. One indication of overfitting is that your model does well on a training set but fails to generalize to unseen test records. If that's the case, you might want to look at how "complex" your neural network is and how many data records do you have.