Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I am experiencing a strange problem with varying the test set size.

This is mildly confusing to explain, but I'll do my best. I'm using octave to train an SVM on timeseries-like data and it's generally yielding good results.

However, occasionally when we've introduced "bad" features, the SVM will "give up" almost instantly during training and produce a model that effectively pins predictions on a single feature. In these cases, the train completes in ~1 second vs the normal 4-5 minutes. I've never really known what causes it, but I just presumed it couldn't find a useful model, which seemed reasonable.

The problem I'm facing now is that it appears something as simple as normalization can be the difference between a very effective model and a "give up instantly" result.

The specific case is this:

I have a dataset of say 5000 training examples. They are split into a 90/10 split for train/test and work well.

Similarly, if I split the data set 95/5, that also works fine (better, in fact for test performance, simply because it's smaller).

However, weirdly, if I split the dataset 90/5/5 and throw out the last 5%, the machine refuses to train. So all I'm doing is removing some of the test set, which is causing the entire process to fail.

To make things even more confusing, I found that normalization is the cause: If I normalize (zero-mean + scaling) then slice, everything works fine, however, if I slice, then normalize, the machine fails.

Basically, my conclusion is that the exclusion of certain training examples in the test set is causing normalization to (naturally) come up with differently values scaled that are for some reason problematic.

I have eyeballed the min/max/mean of the scaled values and none of them vary by more than a few percent, so I'm somewhat at a loss of what to pursue next.

Any guidance of how to go about diagnosing this or what may be happening would be most appreciated.

Cheers!

Update:

After some further work on this, I've realised that the machine "gives up" instantly if there is a feature with all values being zero (this can happen either on certain subsets of training data or just bad luck).

Although I admit a feature of all zeroes is of little value, is there a specific reason why this would cause the algorithm to behave as described?

Incidentally, we're using a Gaussian kernel if it makes a difference.

share|improve this question
    
Well, if removing part of the test set breaks the training, this is certainly a software bug; have you tried asking the authors? Or using other SVM implementation (R with kernlab package is equally easy to use as Octave)? –  mbq Jan 16 '13 at 13:42
3  
I doubt that this completely explains the phenomenon you observe, but you should not be using the test set for anything other than testing. You should not be using the test set to normalize the training set. The model you produce from the first $90\%$ should not depend on the test set. –  Douglas Zare Jan 16 '13 at 14:50
    
Interesting (and thanks for your response). A few questions on that - Surely you need to normalize your test set features by the same scaling factors as the training set? (otherwise, the numbers would be meaningless). So what your saying is that you should never use the entire dataset to normalize, only the training set? It's exactly this (scale then slice, vs slice then scale) that's causing the issues, so I'll definitely give that a try. –  Dr GF Jan 16 '13 at 23:04
    
I've worked out some additional details as to what triggers the problems. See Update section in original post. –  Dr GF Jan 17 '13 at 23:36
1  
I think what @DouglasZare was saying is that you should only use your test set for evaluating a model you built from doing cross-validation on the training data. He's right! –  Kyle. Jan 17 '13 at 23:48

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.