4
$\begingroup$

I am using an ANN for classification. My covariates are the relative lagged returns for gold, SPX and Oil.

When i do not scale my inputs between 0 and 1 I achieve a fast training time. however after scaling my training time increases dramatically, sometimes 40 fold.

My accuracy is also better when inputs are not scaled. I cant seem to find any reason for this online. Has anyone got any ideas?

|cite|improve this question
$\endgroup$
  • $\begingroup$ Is this also true for $z$-score scaling (0-mean, unit sd)? $\endgroup$ – Sycorax Aug 30 '18 at 17:36
  • $\begingroup$ No, this only appears to happen when scaling the relative returns between 0 and 1. I realize the relative returns are scaled and the model preforms well when using these. But when i scale all covariates between 1 and 0 my accuracy for the training set decreases and the time taken to train my model increases alot. $\endgroup$ – tonyf Aug 30 '18 at 17:56
  • $\begingroup$ How are you initializing the parameters? $\endgroup$ – user20160 Aug 30 '18 at 17:57
  • $\begingroup$ In terms of my layers? I have 21 neurons in my input layer, one in my output. i vary the number of perceptions in my hidden layer between 1 and 20. The training time for my unscaled data remains under 0.5 seconds throughout. When i used the scaled data, beyond 4 hidden layer perceptrons it will take anything between 5 and 20 seconds to train. Is there any explanation for this? $\endgroup$ – tonyf Aug 30 '18 at 18:04
  • 1
    $\begingroup$ Do you scale your learning rate appropriately? The magnitude of your gradient may be wildly different. $\endgroup$ – kbrose Aug 30 '18 at 20:35
2
$\begingroup$

We can find a reasonable explanation for this behavior in the Neural Network FAQ. TL;DR - try rescaling your data to lie in $[-1,1]$.

But standardizing input variables can have far more important effects on initialization of the weights than simply avoiding saturation. Assume we have an MLP with one hidden layer applied to a classification problem and are therefore interested in the hyperplanes defined by each hidden unit. Each hyperplane is the locus of points where the net-input to the hidden unit is zero and is thus the classification boundary generated by that hidden unit considered in isolation. The connection weights from the inputs to a hidden unit determine the orientation of the hyperplane. The bias determines the distance of the hyperplane from the origin. If the bias terms are all small random numbers, then all the hyperplanes will pass close to the origin. Hence, if the data are not centered at the origin, the hyperplane may fail to pass through the data cloud. If all the inputs have a small coefficient of variation, it is quite possible that all the initial hyperplanes will miss the data entirely. With such a poor initialization, local minima are very likely to occur. It is therefore important to center the inputs to get good random initializations. In particular, scaling the inputs to [-1,1] will work better than [0,1], although any scaling that sets to zero the mean or median or other measure of central tendency is likely to be as good, and robust estimators of location and scale (Iglewicz, 1983) will be even better for input variables with extreme outliers.

The key detail that makes me think that this is the answer is because you do not observe that it takes a long time to train the network when you use $z$-scores, which have negative and positive input values due to the mean-centering.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ This is correct, when i changed the scale to [-1,1] the training time is reduced dramatically. We see an the average training time decrease in some cases from 20 seconds to 0.8 seconds. However it is still double the time taken to train the network when using unscaled imputs. $\endgroup$ – tonyf Aug 31 '18 at 10:37
  • $\begingroup$ What range of values do you have for the unscaled inputs? $\endgroup$ – Sycorax Aug 31 '18 at 19:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.