How to do k-fold cross validation with classifiers? I want to cross-validate a model that plays the card game below (see image).
I trained the model on a dataset of 1000 games, with the goal to maximise the profit of each game. It works great on the training dataset, but I want to k-fold validate it, to ensure that it generalises well.
How can I apply k-fold validation in this case?. This model is not a regressor. It is a classifier, with a variable reward/penalty moreover. So I cannot calculate the MSE of validation folds, there is no MSE here. How can I apply k-fold validation in this case?

 A: 
This model is not a regressor. It is a classifier, with a variable reward/penalty moreover. So I cannot calculate the MSE of validation folds, there is no MSE here. How can I apply k-fold validation in this case?

Nobody said that Cross Validation only applied to Mean Square Error (i.e. $L_2$ loss) ! In fact you can use CV with a wide range of loss functions..
Without going into the context and details which you can find here: Cross Validation and Confidence Interval of the True Error.. there are a number of loss functions which you can use with CV. 


*

*For regression $L_2$ or quadratic loss is commonly used: $L(f,x,y)) = (f(x) - y)^2$ 

*For classification the ${0,1}$ loss is well known $L(f,x,y)) = 1_{f(x) \neq y}$

*For comparing algorithms $A_1$ and $A_2$ you can use cross validation with matched pairs 

*The Jacknife cross validation

*Many more.... 
Have a look at this wikipedia page: https://en.wikipedia.org/wiki/Loss_functions_for_classification and this SE post: What are the impacts of choosing different loss functions in classification to approximate 0-1 loss
Classification metrics in particular
The Sklearn documentation provides a nice summary
http://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics

