I am doing a multiclass text classification problem. My data consists of tweets. I tried many variations of deep learning models (ex: LSTM, GRU and pre-trained word embeddings) and I also tried linear SVM. I applied 2-gram TF-IDF to the data before I feed it to the SVM.
The results are very similar but the linear SVM with 2-gram TF-IDF gives better results than the best performing deep learning model.
The accuracy of linear SVM with 2-gram TF-IDF is .538
The accuracy of the best performing deep learning model is .528
My questions are:
Isn't deep learning supposed to surpass machine learning models on such complex task like text classification?
Why does the machine learning model give better results?
Should I try harder to improve the deep learning model to surpass the machine learning model's accuracy?
-
$\begingroup$ Do you have many observations? $\endgroup$– DaveCommented Mar 25, 2022 at 17:41
-
$\begingroup$ @Dave yes. I have 458197 data points. but I took 90% of the data for training $\endgroup$– floydCommented Mar 25, 2022 at 17:44
-
2$\begingroup$ Accuracy of 0.538 versus 0.528 doesn't suggest that either model is doing terribly well. Perhaps the problem is very challenging, or the choice of features are not very informative. Or none of the models attempted have well-chosen hyperparameters. There's a lot of reasons. $\endgroup$– Sycorax ♦Commented Mar 25, 2022 at 18:12
-
$\begingroup$ "Isn't deep learning supposed to surpass machine learning models on such complex task like text classification?" no. Neural network research has gone through several hype-bust cycles, and it is this sort of hyperbolic claim that causes the cycles. If a problem has an optimal decision boundary that happens to be a hyperplane, a linear classifier is bound to do better than a complex non-linear model as there is simply less to go wrong (e.g. bad local minima to avoid) - estimation is easier. Always try simple classifiers as a baseline whenever you want to apply a complex non-linear model. $\endgroup$– Dikran MarsupialCommented Apr 10, 2023 at 12:37
-
$\begingroup$ ... it also gives a computationally inexpensive means of checking your pre-processing and evaluation framework/pipeline, so it is well worth doing. $\endgroup$– Dikran MarsupialCommented Apr 10, 2023 at 12:39
1 Answer
In general there is no single algorithm that is superior to all others at solving all problems, which is why so many classification algorithms exist. Some models/algorithms excel at certain problems, and we often don't know which will do best at a particular task until testing them.
- Isn't deep learning supposed to surpass machine learning models on such complex task like text classification?
To my knowledge, this has never been shown to be true for all classes of problems. What constitutes deep learning is not well defined, but even within the broad classification of deep learning, some models perform better at certain tasks than others.
- Why does the machine learning model give better results?
Impossible to say for sure, but SVMs are powerful models. It could be your data preparation and feature selection favor SVMs in some way.
- Should I try harder to improve the deep learning model to surpass the machine learning model's accuracy?
I don't see why you would want a certain type of model to succeed over other types. I wouldn't overlook the importance of feature engineering in model performance; it's often more important than model selection.
