I am pretty new in machine learning and hence facing a lot of confusion in data normalisation concepts. Someone pls clarify the following doubts :

1) While normalising a data matrix of m-samples x n-features, the normalisation should be done along m-samples or along the n-features i.e we normalise each row or each column here ?

2) We should normalise all the data together or normalisation should be done separately for each class.

3) Suppose I have a vector of 2-samples x n-features as shown

class_1   0.4432       27.19        0.6733   0.0828  0.4134  -0.6662  0.3381  0.0552   0.0    -0.01167    0.0    0.0    0.0    0.00056    0.0    0.0    0.03111    0.05778    0.01333    0.0    -0.02722    -0.02333    0.0    0.00222    -0.02722    0.0    0.0    0.0    10100   10010   11000   10100   10100   11000   11000   10001   11000   10001   10001   10100   10001   10001   10001 

class_2   0.7647       16.1073        0.7867   -0.2414  -0.2264  -0.8025  0.8649  -0.3524   -0.01944    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.04611    0.0    -0.01944    0.03556    -0.02722    0.03111    0.0    0.0    0.0    0.0    0.0    10010   10100   11000   10010   10001   10100   10010   10001   10001   11000   10001   10001   10001   11000   10001 

How to normalise in this case as you can see that the first few features have float values but there are some which are encoded binary features ( the 00110 types are some position informations) . If we normalise what will happend to such information ?

4) The training and testing data should be separated after normalisation or before normalisation and then normalised ?

5) What about the user sample ? once I train and prepare my classifier the user give a SINGLE input of 1-sample x n-features ? It is quite obvious that the values for each feature will differ in magnitude greatly as compared to normalised data used in training and testing. How to handle this thing ?


1 Answer 1


1.) The features, typically. If the features are real valued then the most common approach would be to represent each feature by the Standard score. Assuming each row corresponds to a sample and each column to a feature, $\mu_j$ and $\sigma_j$ are the mean and standard deviation of the $j^{th}$ column, and $x_{i,j}$ is the value of feature $j$ for sample $i$ you would compute the normalized value as $$ \hat{x}_{i,j} = \frac{x_{i,j} - \mu_j}{\sigma_j}. $$ That being said, there are plenty of other ways to do normalization and cases where normalizing rows makes sense. An example of the later would be if the samples are documents represented by word counts. Then computing $$ \hat{x}_{i,j} = \frac{x_{i,j}}{\sum_j x_{i,j}} $$ would give normalized histograms of word counts. This could be useful if your documents have very different length since it would keep features from having wildly different magnitude. This is just an example and there are probably better ways.

2.) All together. The former doesn't make any sense since you'll want to apply the same normalization at test time, when you assume you don't know the labels.

3.) I would start by standard scoring the real valued features. How to handle the binary features will depend on what they represent. If the binary features are encodings (like 0101 = 5) this is a bad idea (see this answer I gave for an explanation why) and you'll either want to convert them to the numerical values or encode them as categorical (1-of-$k$) variables. If they're just presence features (like $x_{ij} = 1$ if age of $i$ is greater than 50), then I would start by not doing anything to them.

4.) Typically before. Then normalize the test data using the same values you used to normalize the training data. For example, if you were using standard scores you wouldn't compute new means and standard deviations for the test data, you would just use the same ones you calculated from the training data. This is important.

5.) This is really the same as question 4. Use the same normalization you applied to the test set using the same parameters (means, standard deviations, whatever).

  • $\begingroup$ thanks alto... this was really helpful and for the first time someone clarified my doubts so nicely regarding mormalisation process... I have another small query.. the tf-idf method which you mentioned in the link.. can be used for any data where we are considering count of certain patterns or just for text ? $\endgroup$
    – Ashutosh
    Jan 21, 2014 at 6:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.