what would be a recommended division of train and test data for one class SVM?

Question

I have checked the following one class SVM classification in R that was posted in this thread:

https://stackoverflow.com/questions/27375517/one-class-classification-with-svm-in-r

In this program the author performs one class svm in the iris dataset. For what I see he chooses for the test dataset approximately the 33% of all the data, which is like 50 records belonging to the setosa class. The strange thing that I could notice is that for obtaining the predictions or testing its model, the author uses all the dataset or the 150 records of the iris data. The source code is the following:

library(e1071)
data(iris)
df <- iris

df <- subset(df ,  Species=='setosa')  #choose only one of the classes

x <- subset(df, select = -Species) #make x variables
y <- df$Species #make y variable(dependent)
model <- svm(x, y,type='one-classification') #train an one-classification model 


print(model)
summary(model) #print summary

# test on the whole set
pred <- predict(model, subset(iris, select=-Species)) #create predictions

I have a couple of doubts about this model and the example made:

1. In this case of the one class SVM which would be a suitable train and test division. I am a little bit doubtful if I could use the "golden-rule" of 80% for training and 20% for testing or can make like in the example to train with a subset of the data and then test with all the data?

2. For what I know this type of SVM could be considered as unsupervised learning, so I would not really have like a y label to test if my predictions are correct, but instead this model will detect outliers or a percentage of samples that do not belong to the class that I have, is this correct?

Thanks.

Answer 1

A one-classification SVM takes only the "normal" class of data and trains on that. It learns the boundaries of that class. Then, when fed new data, it attempts to classify the new data points as either "in" or "out" of the normal class. Because of this, the normal 80/20 split rule becomes a bit more complicated. You obviously do not train on any of the abnormal data by definition of a one-classification model. You also should withhold some normal data in order to make sure that it is properly classifying the normal data.

I can't find a specific example of the appropriate split percentage, but testing on your train data is definitely not appropriate. Personally, I would withhold 20% of the normal data and all of the abnormal data to test the model.

Answer 2

Things like separate train/test sets that make a lot of sense in machine learning on large data sets don't always translate well to smaller-scale studies. Unless you have many thousands of cases a split into train and test cases runs a risk of both losing power in training and losing sensitivity in testing. You throw away information by making some cases unavailable for training. With relatively few cases for testing your estimates of performance will necessarily be of low precision and might be highly dependent on the vagaries of the single train/test split.

For situations like your example with 150 cases it can make more sense to use the entire data set both for training and for evaluating the modeling performance, in a principled way that takes advantage of resampling. For example, simple 5-fold cross-validation is the equivalent of an 80/20 train/test split done 5 times; combining multiple runs with different random splits into 5 groups will improve performance. That's a good way to tune hyper-parameters for a model. In no one case are you testing on the training data, but the repetition and randomization takes advantage of all the data for both training and testing.

You can check modeling reliability by repeating all the modeling steps on multiple bootstrap samples of the data then checking on the full data set. That follows the bootstrap principle: the resamples are to the original data sample as the original data sample is to the underlying population. So a modeling approach based on bootstrap samples that works well on your full data sample might be expected to work well when you use your full data sample to model the underlying population.