I am working on Machine Learning, particularly I have a dataset with 50+ columns and 100,000 rows. I need to get the data normalized with ranging to [0,1] (not with standardization) and I've split the dataset in a 80/20 percentatge for the training/test sets.

My question is: I must normalize first the training set and then normalize the test set with the means and deviations extracted from the training set normalization. How can I do that to each one of the columns? I mean, is there a defined method to get the (mean, deviation) tuples for every column in the training set in order to be able to normalize the test set with those values?


Obtain the mean values and standard deviations of the training set, whatever it may be, and apply those values to the test set. The basic assumption of any machine learning method is that all the data comes from the same distribution, and ideally you should apply the same data-dependent transformations to the test set.

If you normalize both together you have a better estimate of the normalization parameters but you would also have added a small information leak: there's no way your final model could've been based on parameters from unseen data.

In practice though there might be not much difference in performance, so you will see many people doing it.

How to do it in R:

##Let's create a 0.7 split from a dataset
data = iris
s = sample(1:nrow(data), 0.7*nrow(data))

train = data[s,]
test = data[-s,]

##Here we obtain the column means and sd, copied from the `mlr` tutorial
trainfun = function(data, target, args = list(center, scale)) {
  ## Identify numerical features
  cns = colnames(data)
  nums = setdiff(cns[sapply(data, is.numeric)], target)
  ## Extract numerical features from the data set and call scale
  x = as.matrix(data[, nums, drop = FALSE])
  x = scale(x, center = args$center, scale = args$scale)
  ## Store the scaling parameters in control
  ## These are needed to preprocess the data before prediction
  control = args
  if (is.logical(control$center) && control$center)
    control$center = attr(x, "scaled:center")
  if (is.logical(control$scale) && control$scale)
    control$scale = attr(x, "scaled:scale")
  ## Recombine the data
  data = data[, setdiff(cns, nums), drop = FALSE]
  data = cbind(data, as.data.frame(x))
  return(list(data = data, control = control))

predictfun = function(data, target, args, control) {
  ## Identify numerical features
  cns = colnames(data)
  nums = cns[sapply(data, is.numeric)]
  ## Extract numerical features from the data set and call scale
  x = as.matrix(data[, nums, drop = FALSE])
  x = scale(x, center = control$center, scale = control$scale)
  ## Recombine the data
  data = data[, setdiff(cns, nums), drop = FALSE]  
  data = cbind(data, as.data.frame(x))

train = trainfun(train, "Species", list(center = TRUE, scale = TRUE))

##So these are respectively your scaling parameters and your train data
control = train$control
train = train$data

##We apply it to the test set
test = predictfun(test, "Species", list(center = TRUE, scale = TRUE), control = control)

##Check the mean and sd of every columnn in train and test
cbind(train.mean = lapply(train[,-1], mean),train.sd = lapply(train[,-1], sd),test.mean = lapply(test[,-1], mean),test.sd = lapply(test[,-1], sd))
             train.mean    train.sd test.mean    test.sd  
Sepal.Length 2.917201e-16  1        0.09530032   0.9208225
Sepal.Width  2.444938e-16  1        -0.005609797 0.8804698
Petal.Length -1.120497e-16 1        0.0977674    0.9124194
Petal.Width  1.08351e-16   1        0.09497264   0.9345891

Also keep in mind there are many frameworks in R that can make that automatically, like caret and mlr. Also, if your algorithm includes internal normalization (center/scale) then this will be done without your interference.

  • 1
    $\begingroup$ Yes, but my question is technical: once I get the means and deviations for every column in the train set, how do I apply the value for a column to the respective column in the test set? $\endgroup$ – vandermies May 29 '16 at 8:31
  • $\begingroup$ @vandermies So are you asking how to do it in R, like a programming question? Then it'll be probably moved to the sister site SO, but I'll try to answer that nevertheless. $\endgroup$ – Firebug May 29 '16 at 13:55

Normalize the features in the entire data set in a single step, before splitting into folds for CV. You can't split data and transform within each fold prior to training/testing. Think of what would happen if you mean-zero standardized data in the training set separately from the testing set. The means and sd's would be different between the training and testing set, which would throw off your ML results.

The potential bias from normalizing outside a fold in CV depends on the data set, type of CV, total sample size, feature selection method, and sensitivity of the classifier to the scale and mean of the data. For example, with LOOCV, normalizing within each fold consisting of $n=1$ wont affect bias. 0.632+ and bootstrap-bias CV (see R. Kohavi) are also CV examples where the potential for bias varies. If features were selected using CV, then there will be a different effect. Regarding classifiers, linear discriminant analysis (LDA) uses the correlation matrix $\mathbf{R}$ from feature values within in a fold, which is scale independent - so this type of bias will be reduced.

An evaluation of 9 data sets, with 14 classifiers, and use of CV2, CV5, CV10, CV-1, CVB, with 10 re-partitionings, indicates almost no performance (accuracy) differences without and without mean zero-standardizing. Didn't assess normalization, but probably won't matter.

You could use random forests (RF), which bootstraps the training data, randomly selects features for node splitting during tree training, and drops the out-of-bag objects down trained trees for class prediction. RF won't be biased by this issue. You could also regularize an ANN classifier with fixed and variational dropout to attain an ARD-effect, which may overfit less than actual Bayesian $L^2$-based ARD.

  • $\begingroup$ Excuse me, but that's simply wrong. If you assume all data come from a single distribution, then the training average and standard deviation are your sample estimates. Your algorithm won't have access to the population true distributional values, you must use the sample estimates otherwise you incur optimistic bias on its generalization performance. $\endgroup$ – Firebug Nov 13 '17 at 10:17
  • $\begingroup$ Modified answer above. $\endgroup$ – JoleT Nov 14 '17 at 16:11
  • $\begingroup$ Excuse me twice, but all your points are underplaying the existance of the bias. The bias might be small (and it might not as well), but it still exists. Even in your LOO example, aggregating that single sample to estimate a mean and standard deviation on features is leakage already. This is independent of the resampling strategy or the learning algorithm selected. It's simply bad practice one could get away with large samples. $\endgroup$ – Firebug Nov 14 '17 at 16:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.