6
$\begingroup$

Suppose I have training and testing data and I want to train a classifier (e.g. SVM). Typically, features are normalized before classification to ensure some features aren't weighted more heavily than others.

Is there any risk that I will get a bad estimate of performance if I use different scaling parameters for my training and testing data? (This assumes I'm normalizing to the same range, like [0,1], for example.)

$\endgroup$
1
$\begingroup$

In general, this should be avoided. The basic assumption of learning algorithms is that all data come from the same distribution. Applying different normalization procedures (or with different parameters) on the training and test data violates this.

There are cases however where this may be appropriate. If it is known that the testing data follow a different distribution other methods can be applied. If the testing data (which are unlabelled) are available for training, techniques such as semi-supervised learning can be applied. In case the differences are due to measurement errors (such as batch effects, which are common on biological data), batch effect removal methods can be applied (see this paper for a comparison and references therein). In case unlabelled data are not available I don't see any way to handle it, unless prior knowledge is available.

$\endgroup$
0
$\begingroup$

I don't see any relation with underfitting or overfitting. It is more about applying the same type of transformation with different parameters. I've tried ZCAWhitening on MNIST data with various datasize m = 10k, 10k, 100. My intuition was that the same transformation with the same parameters(mu,sigma) must be applied. Otherwise, the data on training or test set get biased. We can think it like adding 1.0 to training set, while adding 1.2 to test set. Test set get biased by 0.2 at this condition. Should we expect to take similar results if we apply ZCAWhitening seperately? I say yes only if training and test set has sufficiently many examples. (If underlying variation is not much, small dataset may also show similar results because of their intrinsically similar structures) As number of examples increases, variation between each set decreases, so we can collect pretty similar mu, sigma, etc. Here are results:

m = 10k common normalization = 94.980% seperate normalization = 94.740%

m = 1k common normalization = 81.500% seperate normalization = 79.600%

m = 100 common normalization = 70.000% seperate normalization = 65.000%

It is clear that as number of samples increases difference between common and seperate normalization dicreases, as a result of diminishing sampling error.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.