If one trains a model using a SVM from kernel data, the resultant trained model contains support vectors. Now consider the case of training a new model using the old data already present plus a small amount of new data as well. SO:

  1. Should the new data just be combined with the support vectors from the previously formed model to form the new training set? (If yes, then how to combine the support vectors with new graph data? I am working on libsvm.)


  2. Should the new data and the complete old data be combined together and form the new training set and not just the support vectors of the old one?

Which approach is better for retraining, and/or is more doable and efficient in terms of accuracy and memory?


1 and 2 both describe techniques that would be valid in various circumstances.

In (1), you're basically using a partially stacked model (holding out the "new data", by which I assume you mean additional variables not considered by the initial SVM model).

In (2), you would be applying the SVM model to the entire dataset including the new variables.

It's very difficult to say with no other contextual knowledge which of these approaches would produce "better" results for your model- it would, as always, be best to try both approaches (with proper cross-validation) to determine the "better" approach in terms of your goals.

Assuming you already have the support vectors for the initial training of the model, (1) will be much cheaper (proportionally so to the number of independent variables in the initial model) in terms of memory than (2), as the number of dimensions (variables) passed to the algorithm is comparatively lower in (1).

  • $\begingroup$ Consider any retraining scenario. How will it be possible to concatenate new data with old support vectors to form a new training set? I am not clear with how to merge the two types of data here. The other approach is simpler to do. $\endgroup$ – codeninja Feb 15 '16 at 18:23
  • $\begingroup$ Correct, you wouldn't be so much combining the support vectors themselves with the new data as combining the predicted class membership $$p$$ with the new data in a stacked approach. $\endgroup$ – Thomas Cleberg Feb 15 '16 at 18:30
  • $\begingroup$ So, class labels from the previous prediction model and the new data form the combined data? The support vectors will be indirectly involved then, right? $\endgroup$ – codeninja Feb 15 '16 at 18:38
  • $\begingroup$ Correct, though I would use the raw score rather than the labels themselves in that approach, as information is lost in the simplification. $\endgroup$ – Thomas Cleberg Feb 15 '16 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.