1
$\begingroup$

This is probably a silly question, however, I'm using a neural network in R ("neuralnet" package) to predict a continuous variable from a series of 10 continuous predictors.

Using the model without standardizing the data leads to dreadful results, most of the times the training algorithm won't converge and when it converges the model has a very poor fit.

To avoid this problem I chose to standardize the training and testing set. This is working very good so far, MSE is lower than the one of a linear model and the net seems to perform good on the testing set. However, the output of the network set is standardized and therefore I face the following problems:

  • How should I get back the original value (i.e. not standardized)? For instance: the net outputs $z_1$ as a prediction for the test set, how do I get $y_1$? ($z_1$ is the standardized prediction for $y_1$.) Right now I am doing this $$ y_1 = z_1 \cdot \text{sd}(data) + \text{mean}(data) $$ where $data$ is the entire dataset. Needless to say that this approach seems not that great (although I get good results within the testing set).

  • Furthermore, if I had to predict new data, using the formula above would probably not be a good idea, right? What should I do?

$\endgroup$
  • 2
    $\begingroup$ Given that you are using z-score normalization when going from $y$ to $z$, then what you are doing (reconstructing $y$ from $z$) looks correct. I would also try min-max normalization in the same context, as in my experience it often gives better results. $\endgroup$ – jeff Sep 19 '15 at 22:18
  • 2
    $\begingroup$ A small note on MSE: it depends on the scale of the data, so you should not be comparing the MSE of a model for the original data to the MSE of a model for scaled data. $\endgroup$ – Richard Hardy Sep 20 '15 at 12:08
  • $\begingroup$ @halilpazarlama min-max normalization seems a good idea too, I'll try it out. My worries are that when testing on new data the model would generalize poorly because of the assumptions about mean and standard deviation. Essentially by doing this I'm assuming that they're fixed and exactly the value I'm giving them (mean(data), sd(data)) $\endgroup$ – mickkk Sep 20 '15 at 13:34
  • $\begingroup$ @RichardHardy I'm sorry I did not make it clear, MSE for the neural net is calculated scaling the output back to its original scale in order to compare apples to apples. This "scaling back" process is my main concern. However, even if I avoid the comparison, scaling back is needed for prediction. $\endgroup$ – mickkk Sep 20 '15 at 13:36
  • 1
    $\begingroup$ @mickk I think the same generalization risk exist in z-normalization too, since you infer the mean and std. from the training data, it might be the case where your test data has a different mean and std., and your test features might come off badly scaled. Besides, it always makes more sense to me to denote the "biggest" feature as 1 and the smallest feature as -1. Of course neither of them guarantee good generalization, it will always depend on your training data, i.e. how well it covers possible test samples. $\endgroup$ – jeff Sep 20 '15 at 13:46
2
$\begingroup$

How should I get back the original value (i.e. not standardized)?

Your method is correct; it's just algebra. $$\begin{aligned} z_1 &= \frac{ y_1 - \mu} {\sigma} \\ z_1 \sigma + \mu &= y_1 \end{aligned}$$

Furthermore, if I had to predict new data, using the formula above would probably not be a good idea, right? What should I do?

You just have to store the values of $\sigma$ and $\mu$ which were computed using the training data. Since the predictions of the model will be on the "$z$-scale," you can apply the same procedure to rescale the model predictions to the "$y$-scale".

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.