# Worse accuracy with input normalization (NNs)

I am training a neural network for audio classification. My inputs are "1-channel images" of size 60x130x1.

Surprisingly, I always get better accuracy when training the model with the original data, instead of with the normalized input data (mean = 0 , variance = 1).

This is how I normalize it:

mean = np.mean(X_train, axis = 0)
std = np.std(X_train, axis = 0)

X_train = (X_train-mean)/std
X_test = (X_test-mean)/std
X_val = (X_val-mean)/std


---------------------------------------------------------EDIT 1 ----------------------------------------------------------------

Some relevant values of my training data are:

Min and Max values (across training examples): 0.0 , 1954.4

Min and Max values of the mean (across training examples): 0.0023, 6.7611

Min and Max values of the std (across training examples): 0.0204 , 39.0361   ---------------------------------------------------------EDIT 1 ----------------------------------------------------------------

Does this makes any sense, or normalized inputs should always give better results? (The purple line corresponds to the normalized data) Training accuracy / per_epoch_per_minibatch  • In case you are working with scikit-learn, it has an automatic scaling class: from sklearn.preprocessing import StandardScaler, MinMaxScaler, MaxAbsScaler, RobustScaler; scaler = StandardScaler().fit(X_train); X_train_scaled = scaler.transform(X_train); X_test_scaled = scaler.transform(X_test); X_val = scaler.transform(X_test) It has the advantage that you can try different scaler and pick the one that works best for your data. Apr 27, 2018 at 11:44

why are you applying normalisation? Is it because you believe it to be a necessary step or is it because you have determined based on your data that it is appropriate?

Mean centring and scaling to unit variance is commonly useful, but not universally so and so you should think about the properties of your data.

Mean centring is rarely not useful, but may be less useful for highly skewed populations where subtracting the mean is not significantly accounting for large proportions of the variance in the dataset. Median or mode centering are less common solutions but may work if they reduce total variance more than the mean.

Unit variance is less useful if the data is all on the same dynamic range and noise is correlated with magnitude. In such scenarios scaling to unit variance will magnify the apparent magnitude of variance from a low amplitude signal should be retained as lower than the variance in a high amplitude signal.

I realise this link is specifically about PCA, but it discusses when unit scaling is and isn't helpful and the lessons are more generalisable. Note, to help interpret this link bear in mind that using variance scaled data creates a correlation matrix in the first step of PCA and non-scaled data a covariance matrix. PCA on correlation or covariance?

• thanks for the answer. I am applying it because I thought it was always a desirable thing to do, at least when working with ANNs. I gotta admit I am not super familiarized with these statistic measures. I will edit the post with some values regarding the data, to see what you think about it, in terms of how appropriate normalization might be. Apr 27, 2018 at 9:55
• Could you update with an image (heatmap) of your mean and std variables (I assume they are a 60x130x1 array) and also mean divided (array division not matrix) by std. This would quickly highlight if any pixels with low signal to noise are being unnecessarily inflated and so imputing noise into the model. Apr 27, 2018 at 14:42
• Yes, I'll do that. Check the updates Apr 27, 2018 at 15:21
• looks like your data is noisy even in the high amplitude regions (signal approx 1/4 of your standard deviation) but the dynamic range of your signal to noise ratio does not look huge. The first 5 rows seem to contribute the most amplitude, while the rows between 25 and 60 appear to have better signal to noise than 5-25. If you want to test if the signal to noise is interacting with the scaling you could try running holding out the pixels (or rows) that exhibit the weakest signal to noise ratio in the mean and see how the scaled vs unscaled comparison goes. Apr 27, 2018 at 15:59
• By dynamic range of your signal to noise ratio you mean the difference between max and min values of the mean? And where do you see that rows 25-60 have better signal to noise ratio than 5-25? In the mean/std plot? Apr 27, 2018 at 17:49