0
$\begingroup$

I have a dataset of 1157 samples. The number of features are 6 and I have only one output. My training error is very poor although I have tried a wide range of cost values and followed the recommended procedure: I scaled my inputs and output to the interval 0 and 1. then, found the SVM error rate for cost c=2^{-15:1:15} and gamma g=2^{-15:1:15}

the best result I could get is with c=32768, g=4, rate=0.0291635.

Should I go for larger c values? or should I decide my features are bad predictors of the target?

$\endgroup$
  • 1
    $\begingroup$ Your question implies that you are adjusting the parameters to get good results on the training data. This is a very bad idea and will very likely lead to severe overfitting (and hence poor generalization performance). It would be better to optimise a cross-validation based selection criterion (use nested cross-validation for performance estimation). Have you tried other methods? It may be that it is just a difficult problem and no method will perform significantly better? $\endgroup$ – Dikran Marsupial Oct 10 '14 at 9:30
  • $\begingroup$ Oh I see. I am searching now more about "nested cross validation for performance estimation."Any good resource? $\endgroup$ – MAS Oct 12 '14 at 7:10
1
$\begingroup$

Your parameter search sweeps over a large area. So probably it is your features' fault. Open your data with a Tool like "Weka Explorer", when loading it make sure to use the apropriate "...ToNominal"-attribute-filter. Then examine your features in the "Select attribute"-tab. There you can get an impression how correlated your features are with the target variable. For some insights, take a look at Hall, "Correlation-based Feature Selection for Machine Learning", 1999, http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.37.4643

$\endgroup$
  • $\begingroup$ I would trust my features for now, and try to troubleshoot my SVR. $\endgroup$ – MAS Oct 12 '14 at 7:13
1
$\begingroup$

Are you using a non-linear kernel? try a linear kernel first. Actually, try logistic regression first, with 6 features it will be fast and easy and give results similar to an SVM with linear kernel.

It's hard to know whether an error rate of 0.03 is "poor" without knowing how many positive and negative samples are in your data. For example, it there's only 10% positives, then an a error rate of 10% is poor (not better than random). If there's 50%, then not as bad. Hence it's better to use other measures like area under ROC curve or precision/recall (precision still depends on % positives, ROC doesn't).

You should definitely use cross-validation like in the first answer, but if your model is still nominally "bad" in the training data regardless of how much you tune it, then there's little hope for it being good in cross-validation, and either your features are not useful or your model is not relevant to the problem.

$\endgroup$
  • $\begingroup$ you mean by ROC, Receiver operating characteristic curves? $\endgroup$ – MAS Oct 12 '14 at 7:25
  • $\begingroup$ yes, receiver operating characteristic curves $\endgroup$ – purple51 Oct 12 '14 at 9:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.