I am using an ANN for classification. My covariates are the relative lagged returns for gold, SPX and Oil.
When i do not scale my inputs between 0 and 1 I achieve a fast training time. however after scaling my training time increases dramatically, sometimes 40 fold.
My accuracy is also better when inputs are not scaled. I cant seem to find any reason for this online. Has anyone got any ideas?
We can find a reasonable explanation for this behavior in the Neural Network FAQ. TL;DR - try rescaling your data to lie in $[-1,1]$.
But standardizing input variables can have far more important effects on initialization of the weights than simply avoiding saturation. Assume we have an MLP with one hidden layer applied to a classification problem and are therefore interested in the hyperplanes defined by each hidden unit. Each hyperplane is the locus of points where the net-input to the hidden unit is zero and is thus the classification boundary generated by that hidden unit considered in isolation. The connection weights from the inputs to a hidden unit determine the orientation of the hyperplane. The bias determines the distance of the hyperplane from the origin. If the bias terms are all small random numbers, then all the hyperplanes will pass close to the origin. Hence, if the data are not centered at the origin, the hyperplane may fail to pass through the data cloud. If all the inputs have a small coefficient of variation, it is quite possible that all the initial hyperplanes will miss the data entirely. With such a poor initialization, local minima are very likely to occur. It is therefore important to center the inputs to get good random initializations. In particular, scaling the inputs to [-1,1] will work better than [0,1], although any scaling that sets to zero the mean or median or other measure of central tendency is likely to be as good, and robust estimators of location and scale (Iglewicz, 1983) will be even better for input variables with extreme outliers.
The key detail that makes me think that this is the answer is because you do not observe that it takes a long time to train the network when you use $z$-scores, which have negative and positive input values due to the mean-centering.
