3
$\begingroup$

I am using the Kaggle Scikit data to learn R.

I am using the R e1071 SVM function to predict classes.

When I use:

svm(train, trainLabels, scale = TRUE, type = NULL, kernel = "polynomial")

I obtain this level of accuracy on a sample of the Train data:

> table(pred, trainLabels)
    trainLabels
pred   0   1
   0 478   8
1   12 502

which I interpret as being 98% accurate (8+12) / (478+8+12+502).

Though when I use the same prediction model on the Test data, Kaggle returns a 0.82 score, based on classification accuracy.

Can you explain why I can get such a different accuracy level?

Improve this question
$\endgroup$
6
  • 6
    $\begingroup$ You (almost) always will do a good deal better on the training data than the test data, since you optimize the fit on the training data. $\endgroup$
    – Glen_b
    Oct 10, 2013 at 8:52
  • $\begingroup$ Yes, that makes sense. I guess I did not separate the Train data into two sets. Thank you. $\endgroup$
    – Timothée HENRY
    Oct 10, 2013 at 9:01
  • $\begingroup$ I would start by looking at plots of learning curves, described here: youtube.com/watch?v=g4XluwGYPaA $\endgroup$
    – John Yetter
    Jan 27, 2018 at 15:27
  • 3
    $\begingroup$ Possible duplicate of What exactly is overfitting? or stats.stackexchange.com/questions/304613/… $\endgroup$
    – Sycorax ♦
    Aug 30, 2019 at 11:19
  • 1
    $\begingroup$ @PeterFlom There's any number of possible explanations, but when I hear hoofbeats I think "horse," not "zebra." In the absence of any differentiating detail about how this situation arose, overfitting seems like the most plausible explanation. On the other hand, perhaps the scope of possible explanations so wide as to make the question too broad or the lack of detail makes the question unclear. $\endgroup$
    – Sycorax ♦
    Aug 31, 2019 at 19:51

2 Answers 2

Reset to default
1
$\begingroup$

You may have an over-fitting problem. Where the performance on training is much better than testing. In other words, the model is too specific for your training data, even capturing some noise in the training data, and failed to generalized to new data.

To fix it, use simpler model or regularization (reduce polynomial order and chagne $C$ in svm)

Related discussions:

What is the influence of C in SVMs with linear kernel?

How to know if a learning curve from SVM model suffers from bias or variance?

Improve this answer
$\endgroup$
0
$\begingroup$

You should not expect train accuracy to match test accuracy unless you tune your model. You should cross validation your model before predicting on test data. Read about Bias/Variance and Cross validation in Machine Learning.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.