1
$\begingroup$

I have a binary classification problem. I trained my dataset using a Support Vector Machine (SVM). Now I want to report the model I trained to a 3rd party so that they can use.

For the primal probem of SVM, the classifier equation is of the form,

y = w.x + b

where w is the weight vector, b is the constant and x is the new instance. Depending on the value of y, we can predict the class.

I could report the weight vector w and constant b which was obtained my model.

But if I want to test my model using cross-validation, then how do I report my model? Everytime I cross-validate, the equation of the classifier built will change.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
6
$\begingroup$

You do cross-validation to estimate the generalization performance for a given parameter tuple. Once you have your estimate you train another model on the full training set using the same parameters and give that model to the third party.

Improve this answer
$\endgroup$
3
  • $\begingroup$ What exactly (in the context of this question) do you mean by parameter tuple? $\endgroup$
    – amoeba
    Commented Mar 3, 2014 at 22:00
  • 1
    $\begingroup$ @amoeba By parameter tuple I refer to the set of hyperparameters you use to construct the model (typically $C$ + kernel parameters in the context of SVM classifiers). You should use the same set in both cross-validation and training the overall model. $\endgroup$
    – Marc Claesen
    Commented Mar 3, 2014 at 22:02
  • 1
    $\begingroup$ @MarcClaessen: Right, I am just clarifying for the benefit of OP (whose question sounds like he is quite confused). Note that he seems to be using linear SVM, in which case there is probably only one parameter ($C$) if I am not mistaken. I think your answer is the best here, because others go into unnecessary digressions into what OP is not asking (+1). $\endgroup$
    – amoeba
    Commented Mar 3, 2014 at 22:10
3
$\begingroup$

Everytime I cross-validate, the equation of the classifier built will change.

Usually, there is a series of assumptions associated with the cross validation:

  1. First of all, you assume that each of the many surrogate models is equivalent (at least in its predicting power) to the model trained on the whole data set.

  2. Noticing that this assumption is violated (the pessimistic bias of cross validation is a symptom of this violation), you can make the weaker assumption that at least the surrogate models are equvaltent to each other. This allows you to pool the results of their testing.

  3. If even that assumption breaks down, you have to face the fact that your models (or their predictions) are unstable.

So first of all, as long as 1. or at least 2. is met at the level of stable support vectors, there is no practical problem with your reporting. Different support vector sets can lead to models that basically give the same predictions. As long as that is the case, there is still no trouble: you have different support vectors that essentially describe the same decision boundary.

The only situation where these changes become problematic is when the predictions become unstable. But e.g. with iterated $k$-fold cross validation you can measure whether there is any problem of unstable predictions. (See e.g. Confidence interval for cross-validated classification accuracy) If predictions are stable, you are fine with reporting the model trained on the whole data set. If there is, there are 3 possibilities.

  • You can report the validation results including the observed instability (validation means demonstrating that your model does its job, and stable predictions are probably part of that).
  • And you can add to your professional experience that you overfitted the model, and go for a more restricted one.
    (Strictly speaking, that will require its own independent test set. But you may get away here for practical reasons with cross validating again if you just step once towards a less complex model. If you do that, I'd recommend you make the reader aware that another validation should have been done, but that was impossible for practical reasons. And report the full "history" of developing the model.)
  • Slightly esoteric for SVM, but you can aggregate the surrogate models into an ensemble classifier.
Improve this answer
$\endgroup$
5
  • $\begingroup$ "But you may get away here for practical reasons with cross validating again if you just step once towards a less complex model. If you do that, I'd recommend you make the reader aware that another validation should have been done, but that was impossible for practical reasons.. While true, this is only useful if the other party disposes of statisticians. In my experience, often the actual users of predictive models (for example clinical physicians) do not want this information as they don't know what to do with it. Providing too much info is sometimes considered to be communication noise. $\endgroup$
    – Marc Claesen
    Commented Mar 3, 2014 at 21:24
  • $\begingroup$ @cbeleites: Wait, I am confused. Why don't you suggest to train the SVM on the full dataset after cross-validation? This seems to be the standard practice, and also an advice given in many other threads here (I can count three questions about that on the top-voted page of cross-validation tag :) $\endgroup$
    – amoeba
    Commented Mar 3, 2014 at 22:14
  • $\begingroup$ @amoeba this is not strictly necessary for stable algorithms (like SVM), since a classifier trained on the whole data set will only differ marginally from the classifiers trained during cross-validation. $\endgroup$
    – Marc Claesen
    Commented Mar 4, 2014 at 8:39
  • $\begingroup$ @amoeba: well, first of all I thought I had covered that in the explanation of assumption 1. Second: the only thing where you really don't do it is if you go for the ensemble model. There the final model would be the ensemble of all surrogate models. Thirdly, Marc already explained that aspect very clearly. $\endgroup$
    – cbeleites
    Commented Mar 4, 2014 at 17:02
  • $\begingroup$ @cbeleites: I see, thanks! Just wanted to clarify if there is any difference in views here (but it seems not). $\endgroup$
    – amoeba
    Commented Mar 4, 2014 at 17:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.