Should feature normalization be done for training , test set and target variables together? [duplicate]

Question

Consider a matrix X where the columns are the attributes and each row is an example. There is no order of occurrence of the examples. The target variables are continuous valued.

Does normalization to 0 mean and 1 standard deviation occur by considering the mean and standard deviation each column separately or for each row? Do we have a mean and sigma for each column? My confusion is that each feature is in a row representing and example. So feature normalization should be done for each example. But I may be wrong.

QUESTION 1) Should each row be normalized or each column wise? Is this the correct approach for calculating the mean and standard deviation for normalization (in MATLAB) and then apply it for normalization to zero mean and 1 std? I am calculating the mean and sigma for each attribute across all examples.

%X = data matrix rows are the number of examples, columns are the attributEs
for i = 1:size(X, 2),
  mu(i) = mean(X(:,i));
  sigma(i) = std(X(:,i));
end

Example:

 X= [1 2 4; 1 1 1; 3 2 2; 0 0 0]; %4 examples and 3 attributes
 %for each column there is a mu and sigma
mu =

    1.2500    1.2500    1.7500
sigma =

    1.2583    0.9574    1.7078

QUESTION 2) My dataset is composed of the input and targets (continuous valued). Should normalization of the entire dataset which contains the input and target be done altogether and then should the splitting of the normalized dataset into train, validation and test set and traintarget and test target be done?

Please correct me where wrong.

Answer 1

Question 1 You find the mean and standard deviation of the first column and perform the z-score transform in that column. Then you move to the next column and repeat the process.

Question 2 Elaborating on what Emma Jean wrote, consider the goal of machine learning. You want to make predictions about data that you truly haven’t ever seen. Indeed, speech recognition software ought to be able to make predictions about speech by people who haven’t been born yet (once they’re born and learn to talk, of course).

So you completely hide the out-of-sample data, since the real application of your model will not get to see the data on which it will be making predictions.

Getting back to question 1, if you want to develop a model on 100 observations (per column—each row is one multivariate observation) with 80 as development data and 20 for testing, you only calculate mean and standard deviation of those 80 observations, but you use those means and standard deviations to do the z-score transform of all 100 observations per column.

Answer 2

I've always taken the approach that you should treat your test set like you have never seen it before (i.e. you don't know what it looks like so you can't include it in your normalization!)

Normalize your training set only and apply the same scaler to your test set. This is easy to do in Python with scikit-learn methods but I am not sure how to implement this in MATLAB.