How to use SVM for open-set classification?

Question

Training time:

I have sequential data of a single class. Let's class this class, class A. I trained a neural network on this sequential data from class A using the masked language modeling paradigm. The training data size was about 1.5 million sequences of length 200. This neural network now has a deep understanding of the construction of sequences in class A.

Inference time:

I have sequential test data of numerous unknown classes and class A. The test data imbalance is 99.8% of non-class-A sequences and 0.2% of class-A sequences. I feed the test data to the trained neural network and extract the embeddings. I now want to train a SVM to infer whether the embedding vectors belong to class A or not to class A. Basically, a binary classification task.

Training SVM:

To train the SVM, I created a training set as follows: I took 1000 sequences of class A from the training set and produced the embedding vectors. I then generated 1000 random sequences and again produced the embedding vectors. I trained the SVM on these 2000 sequences. I then used this trained SVM to perform the classification on the real test set.

This method actually works really well for my application. However, my problem is as follows: if I train the SVM on 1000 sequences of class A and 10000 random sequences, the results slightly change. If I do a ratio of 10K : 10K, the results change again.

Strangely enough, if I use class weights and weight class A higher when training the SVM, the results actually become worse regardless of the ratio of class-A to non-class-A.

Question: When I have such an imbalanced test set, how do I figure out exactly how to create the training data for the SVM. Should I keep the training ratio of class-A to non-class-A equal to the class imbalance in the test set (I know this is typically done in Gaussian Mixture Models)? Or does it not matter?

I am using scikit learn's SVM library. I've already identified the best kernel to use for my task. But if there's a different hyper parameter that I should toggle instead of the ratio of class-A to non-class-A, that would also be very helpful to know.

Disclaimer: I can't train the SVM with the entire 1.5 million sequences that I used to train the neural network. Because then the SVM's training data would become at least 3 million sequences and the computation becomes intractable.

Answer 1

Generally speaking you should not be using "random sequences" to try to model the non-class-A. You're likely better off using some kind of One-Class or Single-Class classification technique whereby you only present the class-A data.

For support vector models, two common approaches are the SVDD (which tries to capture the data in the smallest sphere) and One-Class SVM - these are equivalent to each other for the Guassian kernel. Note that libsvm (and thus scikit) use the latter.

In both cases, you effectively control the percentage of points in class-A that you are willing to misclassify to enforce the boundary for identifying A.