Let's say I have a dataset with 5 features. One row is (just to give you an idea of the range of data in each column) [200456, 76, 2, 1, 0, 9986] First I normalize columns with mean = 0 and variance = 1. Then I scale the columns of the data from 0 to 1. This gives me really good results, but in theory, I am not very convinced that this is a good method of normalization. I am working with multi-label classification of data if that helps. Kindly let me know if more information is required.
The part about normalizing across rows pops out at me. It's usual to normalize a feature (column) so that, having done this for each feature, the features will be on more comparable scales. Normalizing across rows probably won't make any physical sense, and I'm not sure I can see any situation where it would be justified. (Imagine mashing a person's height, weight, and blood pressure together.)
Even if you normalize only the columns, note: if you normalize all of your data and then split it into train/test, you will get unrealistically better test results than you should. Your training data represents the data you have before you deploy your model and your test data represents the data that comes in after deployment. By normalizing across this boundary, you are allowing data from the future (test set) to leak into the present (training set). This can't and won't happen in the real world.
Normalizing with 0 mean and 1 standard deviation is a common practice.
I'm assuming you've made a typo when mentioning normalizing across rows, if not, then you should only normalize each column (feature) independently of the other features. This should be repeated when pre-processing test data when checking model performance.The train sample and test sample means and sigma will likely be different, but given a large enough sample they should be very similar.
Lets assume you're normalizing only the columns. In this scenario, performing 2 different types of normalization on the same column as you have mentioned will result in the following:
- Lets assume you have a million records and you've normalized to mean = 0 and std. dev. = 1.
- If the population is truly normally distributed, you can safely assume 3 records will be near the value of 6. When you apply the second normalization of scaling down to a fixed range of 0 to 1, these outliers will reduce the value of the 2/3 of the population by 1/6 th.
- Depending upon whether such outliers do or do not appear in your training set, the data set would get scaled down arbitrarily. Most classification models will learn a very different separation boundary due to this effect of just a few records. The model performance on a test set will likely vary widely in such cases.
Hence, its not recommended to apply the 0 to 1 scaling after you've already normalized to mean 0 and std. dev. 1.