SVM parameter dependence on number of samples

I need to do a grid search to optimize SVM parameters gamma, C and epsilon (svm from e1071 r package). The problem is that I have a fairly large data set, about 100000 rows and 40 variables.

I have concluded that I can probably survive grid search and cross validation on 40000 samples of data. But can optimized parameters for subset of 40000 data be used on the final model of 100000 or are the parameters dependent on sample size, and how?

Is any of the 3 parameters independent of other two, or at least in some stable range? For example, lets say I optimize just for gamma and C by holding epsilon constant. I know that C and epsilon influence the complexity of the model in different ways, and then I use best gamma from previous search, keep it constant and determine the other two arguments. Is this likely to give me a good result, or are all 3 parameters strongly depend on each other?

The $C$ parameter for most SVM implementations scales approximately linearly with the number of training patterns, so if you train with a subset, you will need to multiply $C$ by a factor of $N_s/N_t$ where $N_t$ is the number of training patterns and $N_s$ is the subset size for the parameter optimization. This is only an approximation though, so it may not give as good an estimate as performing model selection with the full training set.

• Correct me if I'm wrong, but C multiplies the loss term which grows linearly with N (sum over all samples of the hinge loss), so if you select C using N samples, and then train on 2N samples, C should be multiplied by 0.5, to get approximately the same regularization strength, so the factor should be Ns / Nt, no? – Itamar Katz Feb 19 '17 at 14:46

I have run into this problem many times and the best solution is to use the caret package from R which has a parallel processing component: http://caret.r-forge.r-project.org/parallel.html

You can then use your full dataset (which is very recommended) and run a large grid search that will be ran on different cores on your machine. This works for both a pc and mac so should work for whatever type of machine you have. There are many examples on how to do this on the web but the basic setup will look something like:

install.packages('doMC')
install.packages('caret')

##############################################
require('caret')
require('doMC')

#parallel processing
registerDoMC(cores = 5)  #these many parallel executions will hap#pen
ctrl <- trainControl(method = "repeatedcv",
number = 10, repeats = 10)  #method for training cross-validation and 10-folds will be created

grid_rf <- expand.grid(.C = c(2, 4, 8, 16), .gamma=c(1,10,100)) # model selection parameter in  this case mtry.
# Both trainControl and expand.grid is provided by caret
model_randomForest <- train(y~ ., data = data, method = "svm",
trControl = ctrl,
tuneGrid = grid_rf) #metric is used to measure the model accuracy.
• I am already using all the cores it's still very slow on full data set to make a good grid search. The data is very noisy and I need to use custom metric which is quite unstable, so I need to do many tests to be sure in quality parameter selection. – enedene Jun 4 '14 at 14:53
• Have you considered doing a logistic regression, this may yield nearly identical results but may run MUCH faster. Also, you can try stochastic gradient descent, this is very helpful is datasets with many rows and columns. You should really try to retain all the data you can when training, limiting it will only make a lesser model. – mike1886 Jun 4 '14 at 14:57
• Logistic regression will not yield nearly identical results as there are many unknown interactions between variables. The idea is to find the parameters on subset of the data and then use these parameters to train on a full set. And I want to know what is the dependence. I know there are many other models, but I'm currently interested in SVM. – enedene Jun 4 '14 at 15:05