2
$\begingroup$

Let's say I have a dataset with 5 features. One row is (just to give you an idea of the range of data in each column) [200456, 76, 2, 1, 0, 9986] First I normalize columns with mean = 0 and variance = 1. Then I scale the columns of the data from 0 to 1. This gives me really good results, but in theory, I am not very convinced that this is a good method of normalization. I am working with multi-label classification of data if that helps. Kindly let me know if more information is required.

$\endgroup$
  • $\begingroup$ This is pretty counterintuitive. It sounds like a kind of proxy for double centering but double centering has never given good results for anybody I know of. What are these four values? How are they related? Or are they completely unrelated? Just to be clear, you're averaging these 4 values, finding their variance or std dev and then "standardizing" all 4 values to a mean of zero and variance or std dev of 1? How are you motivating or justifying this? Is the motivation simply that you can do it numerically? Or is it the case that you are asking this site for the motivation? $\endgroup$ – Mike Hunter Sep 19 '16 at 14:41
  • $\begingroup$ Hi DJhonson. As Sandeep S. Sandhu pointed out, I made a typo. The normalization are all over the columns. The values are complete unrelated. I have corrected my question. Thanks for pointing out. $\endgroup$ – Soumya Shubhra Ghosh Sep 20 '16 at 9:24
  • $\begingroup$ If you're scaling to (value - min) / (max - min) then "normalizing" (meaning, scaling to (value - mean) / SD) is irrelevant and redundant as a first step. You can just get there directly. I'd advise strongly that "normalizing" is an overloaded word even across statistical sciences, let alone quantitative fields. In a statistical context there is a high chance of confusing it with transformations that bring the data closer to a normal (Gaussian) distribution. I never trust the term normalizing unless I can see or sense the algebra used. $\endgroup$ – Nick Cox Sep 20 '16 at 9:31
2
$\begingroup$

The part about normalizing across rows pops out at me. It's usual to normalize a feature (column) so that, having done this for each feature, the features will be on more comparable scales. Normalizing across rows probably won't make any physical sense, and I'm not sure I can see any situation where it would be justified. (Imagine mashing a person's height, weight, and blood pressure together.)

Even if you normalize only the columns, note: if you normalize all of your data and then split it into train/test, you will get unrealistically better test results than you should. Your training data represents the data you have before you deploy your model and your test data represents the data that comes in after deployment. By normalizing across this boundary, you are allowing data from the future (test set) to leak into the present (training set). This can't and won't happen in the real world.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Normalizing with 0 mean and 1 standard deviation is a common practice.

I'm assuming you've made a typo when mentioning normalizing across rows, if not, then you should only normalize each column (feature) independently of the other features. This should be repeated when pre-processing test data when checking model performance.The train sample and test sample means and sigma will likely be different, but given a large enough sample they should be very similar.

Lets assume you're normalizing only the columns. In this scenario, performing 2 different types of normalization on the same column as you have mentioned will result in the following:

  1. Lets assume you have a million records and you've normalized to mean = 0 and std. dev. = 1.
  2. If the population is truly normally distributed, you can safely assume 3 records will be near the value of 6. When you apply the second normalization of scaling down to a fixed range of 0 to 1, these outliers will reduce the value of the 2/3 of the population by 1/6 th.
  3. Depending upon whether such outliers do or do not appear in your training set, the data set would get scaled down arbitrarily. Most classification models will learn a very different separation boundary due to this effect of just a few records. The model performance on a test set will likely vary widely in such cases.

Hence, its not recommended to apply the 0 to 1 scaling after you've already normalized to mean 0 and std. dev. 1.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.