Why would a Linear SVM perform worse than Logistic Regression? I have a dataset of about 1000 features and 2000 training examples. I use Randomized Search with cross validation to compare Random Forest, Linear SVC, SVC with different non-linear kernels, and Logistic Regression.
I consistently get much worse scores with Linear SVC than with the other models, including Logistic regression (AUC ROC of 0.70 for Linear SVC vs 0.80 for other models).
From what I understand, the performance of the Linear SVMs should be comparable to Logistic Regression. What could be the reason for such poor performance?
Here are the hyperparamaters I'm checking:
rf_param_grid = {
    'randomforestclassifier__max_depth' : np.random.randint(5, 150, 30),
    'randomforestclassifier__min_samples_split': np.random.randint(2, 50, 30),
    'randomforestclassifier__n_estimators': np.random.randint(50, 400, 10),
    'randomforestclassifier__min_samples_leaf': np.random.randint(1, 20, 30),
    'randomforestclassifier__max_features': ['auto', 'sqrt', 'log2', 0.25, 0.5, 0.75, 1.0],
    'randomforestclassifier__criterion': ['gini', 'entropy'],
    'randomforestclassifier__class_weight':["balanced", "balanced_subsample", None],
    "randomforestclassifier__class_weight": ['balanced', None]
}
linear_svc_param_grid = {
    'svc__C': [0.1, 0.2, 0.3, 0.4, 0.5, 1, 5, 10],
    "svc__class_weight": ['balanced', None]
}
kernel_svc_param_grid = {
    'svc__C': loguniform(1e-1, 1e3),
    'svc__gamma': loguniform(1e-04, 1e+01),
    'svc__degree': uniform(2, 5),
    'svc__kernel': ['poly', 'rbf', 'sigmoid'],
    "svc__class_weight": ['balanced', None]
}
lr_param_grid = {
    'logisticregression__C': loguniform(1e-5, 1e4),
    'logisticregression__penalty': ['l1', 'l2', 'elasticnet'],
    'logisticregression__class_weight': ['balanced', None],
    'logisticregression__l1_ratio': uniform(0, 1)
}

 A: In addition to @sycorax's answer (+1), another issue is that the SVM is not designed for AUROC maximisation - it is designed to estimate the optimal decision surface for a particular set of misclassification costs, determined by the values of $C$ for each class (typically only a single $C$ value is used, in which case the misclassification costs are equal).  How it performs for other sets of misclassification costs (which is what ROC analysis is probing), is not of primary importance.
The logistic regression model on the other hand, aims to estimate the posterior probability of class membership, and rather than concentrating on a single decision threshold (usually $p=0.5$), it tries to estimate the posterior probabilities accurately everywhere.  This means that different misclassification costs can be accommodated just by changing the threshold value.  A consequence of this, there is more reason to expect that logistic regression will quite good at maximising ROC (at least more so than the SVM).
So if AUROC is the primary performance index, it probably isn't a suitable application for the SVM.
A: Maybe the linear SVC grid doesn't cover enough of the parameter space, or the grid is too coarse. You only specify 8 grid values for linear SVC. You don't tell us anything about the random search configuration, but if you're using more than 8 random search iterations, you're just repeatedly testing one or more of those same 8 values for SVC. There might be a hyper-parameter value, not among those 8, that improves the model to be consistent with your expectations.
For other hyper-parameters, you use continuous probability distributions. That means that each random search iteration draws a random value that is different than every value you've already tested. Using the same approach for linear SVC lets you test new hyper-parameter values at each random search iteration.
