# How to select hyperparameters for SVM regression after grid search?

I have a small data set of $150$ points each with four features. I plan to fit a SVM regression for the reason that the $\varepsilon$ value gives me the possibility of define a tolerance value, something that isn't possible in other regression techniques.

I have run cross-validated grid search on the $\gamma$ and $C$ values, at different values of $\varepsilon$. For different combinations for $\varepsilon$, $\gamma$, and $C$, I receive similar scores (as indicated in grid and results) .

Question: How do I define a criterion on improving the selection of hyper-parameters and making a rational model for my data set?

Results:

**Epsilon = 0.06**
The best parameters are {'C': 48.939009184774889, 'gamma': 0.03562247890262444} with a score of 0.64

**Epsilon = 0.09**
The best parameters are {'C': 48.939009184774889, 'gamma': 0.03562247890262444} with a score of 0.64

**Epsilon = 0.11**
The best parameters are {'C': 48.939009184774889, 'gamma': 0.03562247890262444} with a score of 0.66

**Epsilon = 0.14**
The best parameters are {'C': 48.939009184774889, 'gamma': 0.03562247890262444} with a score of 0.67

**Epsilon = 0.17**
The best parameters are {'C': 48.939009184774889, 'gamma': 0.03562247890262444} with a score of 0.66

**Epsilon = 0.19**
The best parameters are {'C': 48.939009184774889, 'gamma': 0.03562247890262444} with a score of 0.65

**Epsilon = 0.22**
The best parameters are {'C': 48.939009184774889, 'gamma': 0.03562247890262444} with a score of 0.64

**Epsilon = 0.25**
The best parameters are {'C': 14873.521072935118, 'gamma': 0.00072789538439831537} with a score of 0.65

**Epsilon = 0.27**
The best parameters are {'C': 621.0169418915616, 'gamma': 0.0038566204211634724} with a score of 0.65

**Epsilon = 0.30**
The best parameters are {'C': 4175.3189365604003, 'gamma': 0.0012689610031679235} with a score of 0.66


• This seems to be a very relevant question pertaining to the mission statement of this group. I would really like to see some positive interaction. – Ankit Bansal Mar 23 '17 at 3:26

Though I haven't fully understood the problem, I am answering as per my understanding of the question.

Have you tried including Epsilon in param_grid Dictionary of Grid_searchCV.

I see you have only used the C and gamma as the parameters in param_grid dict.

Then i think the system would itself pick the best Epsilon for you.

Example:

from sklearn.svm import SVR
import numpy as np
n_samples, n_features = 10, 5
np.random.seed(0)
y = np.random.randn(n_samples)
X = np.random.randn(n_samples, n_features)
parameters = {'kernel': ('linear', 'rbf','poly'), 'C':[1.5, 10],'gamma': [1e-7, 1e-4],'epsilon':[0.1,0.2,0.5,0.3]}
svr = svm.SVR()
clf = grid_search.GridSearchCV(svr, parameters)
clf.fit(X,y)
clf.best_params_


output: {'C': 1.5, 'epsilon': 0.1, 'gamma': 1e-07, 'kernel': 'poly'}

• GridSearchCV can be done only using two parameters. So I had to use Gamma and C for the grid search but I changed the value of epsilon for each run of GridSearchCV – Ankit Bansal Mar 27 '17 at 12:55
• No you can add any number of parameters.I have tried. once check the edit in the answer for the code. – phanny Mar 27 '17 at 13:18
• Have you tried this. Did my answer meet your requirements? – phanny Mar 28 '17 at 5:10
• I tried it, the GridSearchCV does work. I'm running the program on a large range of values to get finalize my parameters. Thank you for the input. – Ankit Bansal Mar 28 '17 at 5:12