2
$\begingroup$

I want to cross-validate a model that plays the card game below (see image).

I trained the model on a dataset of 1000 games, with the goal to maximise the profit of each game. It works great on the training dataset, but I want to k-fold validate it, to ensure that it generalises well.

How can I apply k-fold validation in this case?. This model is not a regressor. It is a classifier, with a variable reward/penalty moreover. So I cannot calculate the MSE of validation folds, there is no MSE here. How can I apply k-fold validation in this case?

enter image description here

Improve this question
$\endgroup$
4
  • $\begingroup$ In what sense is this a classifier? In fact, in what sense is it even a supervised algorithm? It seems to be that what you're interested in is the profit of the "model" (the model here being a game-playing algorithm). Right..? $\endgroup$
    – Denziloe
    Commented Aug 22, 2018 at 23:06
  • $\begingroup$ CLASSIFIER: at every iteration, it has to guess ♥/♦/♣️/♠ based on the available data, i.e. the cards that have been already seen in previous iterations. SUPERVISED: when the current card is revealed, you get your learning feedback (i.e. the profit/loss of the table above). Don't get stuck on game specifics. This is just an instrumental example to discuss KFold validation, for a classification problem where classifying right or wrong yields different rewards/costs for each class. $\endgroup$
    – elemolotiv
    Commented Aug 23, 2018 at 5:21
  • $\begingroup$ But in your own words the model is not applied to iterations, it's applied to games. The iterations within games are not independent (you are sampling without replacement). The strategy for the first guess should obviously be very different for the strategy for the 52nd guess. It therefore makes little sense to me to treat that as a standard classification task. $\endgroup$
    – Denziloe
    Commented Aug 23, 2018 at 9:34
  • $\begingroup$ ok 🙂 i will repost with a different example so that the game doesnt get in the way of discussing kfold for a classifier with different rewards/costs for right/wrong guesses. thanks! $\endgroup$
    – elemolotiv
    Commented Aug 23, 2018 at 15:11

1 Answer 1

Reset to default
2
$\begingroup$

This model is not a regressor. It is a classifier, with a variable reward/penalty moreover. So I cannot calculate the MSE of validation folds, there is no MSE here. How can I apply k-fold validation in this case?

Nobody said that Cross Validation only applied to Mean Square Error (i.e. $L_2$ loss) ! In fact you can use CV with a wide range of loss functions..

Without going into the context and details which you can find here: Cross Validation and Confidence Interval of the True Error.. there are a number of loss functions which you can use with CV.

  • For regression $L_2$ or quadratic loss is commonly used: $L(f,x,y)) = (f(x) - y)^2$

  • For classification the ${0,1}$ loss is well known $L(f,x,y)) = 1_{f(x) \neq y}$

  • For comparing algorithms $A_1$ and $A_2$ you can use cross validation with matched pairs

  • The Jacknife cross validation

  • Many more....

Have a look at this wikipedia page: https://en.wikipedia.org/wiki/Loss_functions_for_classification and this SE post: What are the impacts of choosing different loss functions in classification to approximate 0-1 loss

Classification metrics in particular

The Sklearn documentation provides a nice summary http://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics

enter image description here

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.