1
$\begingroup$

I created a classifier (a linear SVM in scikit-learn) to classify tweets about the fat acceptance movement (yeah that's a thing) as supporting the movement, opposing the movement, or having an unclear opinion. After converting the data into a TF-IDF, I found that I had around 30000 features compared to 1000 samples. Intuitively, I would expect a model to perform poorly and be overfit when there are more features than samples, and a quick search kind of confirmed this, so I used a TruncatedSVD to reduce the features to 100. However, doing k-fold cross validation has shown me that not reducing features does substantially better than reducing features (0.72 f1 compared to 0.63 f1). Why am I seeing this behavior?

$\endgroup$

1 Answer 1

0
$\begingroup$

It's likely that when you reduce the features to 100, you are under-fitting. At this point, it is hard to conclusively say whether overfitting and underfitting would give better results, hence your confusion. I would recommend treating the number of features as a hyper-parameter of your model and use cross-validation to see what is the optimal number of features for an SVM on your data.

$\endgroup$
4
  • $\begingroup$ Also, see scikit-learn.org/stable/modules/…, so we see that the SVM comes with some regularization to mitigate overfitting. $\endgroup$
    – Anon
    Commented Jan 29, 2020 at 5:52
  • $\begingroup$ I have already done cross val using values of (100, 500, 1000, 2000, 4000) as number of features, but non-reduced still produced the best results. Are there cases where a high variance model can perform the best? $\endgroup$ Commented Jan 29, 2020 at 22:40
  • $\begingroup$ If it's the case that cross-validation indicates better results when you have a larger number of features, it's likely you're still in the underfitting regime. What about 8000 features, 16000 features, 25000 features? Another plausible explanation is that all 30000 features are important predictors of the outcome, so dimensionality reduction results in a loss of information which hurts your model's performance $\endgroup$
    – Anon
    Commented Jan 30, 2020 at 5:00
  • $\begingroup$ I messed around some more and it turns out that 1000 features gives very similar performance to no reduction (0.711 f1 to 0.712 f1). In both cases, the training error is very close to 0, which I think indicates underfitting isn't occurring (feel free to correct me). I also tried larger number of features, but that was very slow, and performance was worse than both 1000 features and 30k features. It's kind of weird to me that 1000 and 30K are the best, while values in between perform worse. What's the explanation for that? $\endgroup$ Commented Jan 31, 2020 at 1:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.