SVM model never fires

Question

Problem Definition

The classification problem in hand is to detect an event that should occur (target=1) within the next forward window of fw days using the current environment state which contains 26 features.

The dataset is unbalanced, with 508 event occurrences (target=1) vs. 3024 non-occurrences (target=0).

The dataset was normalized (min-max-scaling) and the SVM was trained on 70% of the dataset. Classifying the unseen data returned class 0 for all, however about 12% should be 1.

I first suspected that this is due to the dataset being biased towards non-occurrences, so I balanced the dataset by up-sampling occurrences, however the same result was obtained.

A grid search on C and gamma for the RBF kernel was performed, each time the classification is 0.

Question

Is it possible that the result obtained is because the input features are not sufficient to detect the event occurrence? Or is there something else the could be experimented with prior to changing the dataset features?

Answer 1

Yes, samples not reflecting the relation between features and target variable sufficiently would be one explanation for always prediction the "regular case" - but there are other possible answers too. It could e.g. also that your model very strongly overfits, thereby only remembering exactly those occurrences to fire for that it saw in the training set - hence will never fire for any "just similar" occurrences (you can imagine this e.g. as "island" around a sample in your feature space, instead of the decision border you would expect).

1. If your training error is better than your test error (so the model is able to find a reasonable separation between 0 and 1 samples during fitting data), and this is the case for multiple hyperparameters: instead of training your model on all training data, perform (repeated) cross validation on your training data to evaluate different hyperparameters and select a suitable final model, which you then test on the held-back test set.

2. If you assumption about samples not sufficiently reflecting the feature-target relation is true you will see that the model is able to fit to the $n-1$ partitions during, hence the internal training error of each CV model will be small, while the internal evaluation error of each CV model on the $n$th partition will be higher. If this is the case you can try different hyperparameters now to see if you ran into a high variance before (use e.g. a $3^x$ parameter grid), or e.g. try to get more features and/or transform your data to better show the feature-target relation for your model (= represent your information differently by introducing new features based on the existing ones).

3. If you change the prevalence of your data, only do this for the training partition (you want to see how it performs on "real life", unchanged test data later).