0
$\begingroup$

I built a neuralnet model to do predictions from measured data. I have a conceptual problem in understandinig how to normalise/denormalise my data in the "prediction-mode". Here is what I did: I used a dataset which I normalised with the min/max method: normalized = (x-min(x))/(max(x)-min(x)) The I splitted the dataset and used 80% of the data to train model and 20% to validate the model. This worked all fine. Only working with this initial dataset makes it also easy to "denormalise" the data to their initial values using this formula: denormalized = (normalized)*(max(x)-min(x))+min(x)

formulas are taken from here: De normalize predicted value

Now when I use the model to do predictions in realtime, I have difficulties in understanding how normalise/denormalise my data. I have every day new measured date from whcih I want to do a prediction. Do I have to use the min/max values that I gathered from my inital dataset, or do calculate every day new min/max values for the daily dataset to feed into the neuralnet model? I have tried both ways, and both results are not too good... When I use the min/max values from the training data, the results becomes a bit better (less noisy). For the model output I have to use the min/max values from the training, as I do not have any other data. Or is there a conceptual error in my thinking? Any help is welcome.. Thanks a lot in advance, Arne

$\endgroup$

2 Answers 2

1
$\begingroup$

Ideally you are supposed to save the min/max you used to normalize the training data; so that you can re-use it for test data/live data. DO NOT re-normalize the entire train + new data again because that is leakage.

Use the min/max of train data to normalize the test data

$\endgroup$
0
$\begingroup$

My best guess would be the following:

Since the "new" instances have not been there during the original normalization phase. One possible action is to include the new instances as bottom rows to your original dataset, re-normalize and then re-train your model. Doing so, you will adjust the relative location of the new instances with respect to the original dataset.

If your model still does not perform well in terms of prediction power, you might be experiencing an "overfitting" problem, which is a common problem in predictive models. One indication of overfitting is that your model does well on a training set but fails to generalize to unseen test records. If that's the case, you might want to look at how "complex" your neural network is and how many data records do you have.

$\endgroup$
1
  • $\begingroup$ I agree, that including the new data and re-training the model would be the "clean" way to do it. But this would somehow contradict the idea of prediction, because I would have to validate the model every day. Overfitting could be a topic, I trained my model over the weekend with less neurons. I will have to check the results today. But can you (or anybody else) give advice what the right approach would be in general? Best regards, Arne $\endgroup$
    – user131308
    Sep 19, 2016 at 5:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.