4
$\begingroup$

I have a dataset of about 1000 features and 2000 training examples. I use Randomized Search with cross validation to compare Random Forest, Linear SVC, SVC with different non-linear kernels, and Logistic Regression.

I consistently get much worse scores with Linear SVC than with the other models, including Logistic regression (AUC ROC of 0.70 for Linear SVC vs 0.80 for other models).

From what I understand, the performance of the Linear SVMs should be comparable to Logistic Regression. What could be the reason for such poor performance?

Here are the hyperparamaters I'm checking:

rf_param_grid = {
    'randomforestclassifier__max_depth' : np.random.randint(5, 150, 30),
    'randomforestclassifier__min_samples_split': np.random.randint(2, 50, 30),
    'randomforestclassifier__n_estimators': np.random.randint(50, 400, 10),
    'randomforestclassifier__min_samples_leaf': np.random.randint(1, 20, 30),
    'randomforestclassifier__max_features': ['auto', 'sqrt', 'log2', 0.25, 0.5, 0.75, 1.0],
    'randomforestclassifier__criterion': ['gini', 'entropy'],
    'randomforestclassifier__class_weight':["balanced", "balanced_subsample", None],
    "randomforestclassifier__class_weight": ['balanced', None]
}
linear_svc_param_grid = {
    'svc__C': [0.1, 0.2, 0.3, 0.4, 0.5, 1, 5, 10],
    "svc__class_weight": ['balanced', None]
}
kernel_svc_param_grid = {
    'svc__C': loguniform(1e-1, 1e3),
    'svc__gamma': loguniform(1e-04, 1e+01),
    'svc__degree': uniform(2, 5),
    'svc__kernel': ['poly', 'rbf', 'sigmoid'],
    "svc__class_weight": ['balanced', None]
}
lr_param_grid = {
    'logisticregression__C': loguniform(1e-5, 1e4),
    'logisticregression__penalty': ['l1', 'l2', 'elasticnet'],
    'logisticregression__class_weight': ['balanced', None],
    'logisticregression__l1_ratio': uniform(0, 1)
}
$\endgroup$
1

2 Answers 2

5
$\begingroup$

Maybe the linear SVC grid doesn't cover enough of the parameter space, or the grid is too coarse. You only specify 8 grid values for linear SVC. You don't tell us anything about the random search configuration, but if you're using more than 8 random search iterations, you're just repeatedly testing one or more of those same 8 values for SVC. There might be a hyper-parameter value, not among those 8, that improves the model to be consistent with your expectations.

For other hyper-parameters, you use continuous probability distributions. That means that each random search iteration draws a random value that is different than every value you've already tested. Using the same approach for linear SVC lets you test new hyper-parameter values at each random search iteration.

$\endgroup$
8
$\begingroup$

In addition to @sycorax's answer (+1), another issue is that the SVM is not designed for AUROC maximisation - it is designed to estimate the optimal decision surface for a particular set of misclassification costs, determined by the values of $C$ for each class (typically only a single $C$ value is used, in which case the misclassification costs are equal). How it performs for other sets of misclassification costs (which is what ROC analysis is probing), is not of primary importance.

The logistic regression model on the other hand, aims to estimate the posterior probability of class membership, and rather than concentrating on a single decision threshold (usually $p=0.5$), it tries to estimate the posterior probabilities accurately everywhere. This means that different misclassification costs can be accommodated just by changing the threshold value. A consequence of this, there is more reason to expect that logistic regression will quite good at maximising ROC (at least more so than the SVM).

So if AUROC is the primary performance index, it probably isn't a suitable application for the SVM.

$\endgroup$
2
  • 2
    $\begingroup$ Good point (+1) On the other hand, there is reason to believe that a nonlinear SVM like a radial basis SVM would have a higher AUC than a logistic model that is linear in the features, if the decision boundary for the task is not linear in the feature space. In other words, a more complex model can overcome the limitation you cite. Naturally, a logistic regression with basis expansion could leverage the qualities you point out to out-perform an SVM in terms of AUC. $\endgroup$
    – Sycorax
    Dec 1, 2022 at 14:48
  • 3
    $\begingroup$ @Sycorax the kernel trick can be applied to logistic regression as well - Kernel Logistic Regression is one of my favourite machine learning tools. I tend not to use conventional SVMs that often. The Least-Squares Support Vector Machine (LS-SVM) is another good tool, with more reason to expect good ROC performance as it is optimising a proper scoring rule. $\endgroup$ Dec 1, 2022 at 15:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.