Matlab Classification Learner App vs. own code: Why is there a difference between Leave-One-Person-Out and 10-fold Crossvalidation? I am trying to find a good model to explain my dataset. 
The problem is that I want to do leave-one-person-out cross validation which is not available in the Matlab Classification Learner App. So I trained different models (e.g. Tree, SVM, KNN, LDA) using functions like fitctee, fitcsvm, fitcknn, and fitcdiscr. 
Following the leave-one-person-out procedure I have found average classification accuracy of about 70% for the best model. However, when I use the App to model the data using 10-Fold cross validation, it has much better accuracy and TPR and TNR about 98%. 
This is really confusing that why this is happening! I was wondering if there are some steps I am missing when I do the modeling programmatically. Or is there any way to do what the App does by writing scripts and probably customizing the cross validation scheme to leave-one-person-out?
 A: First of all, Leave-One-Out and 10-fold-Crossvalidation do not necessary return comparable results: See for example 10-fold Cross-validation vs leave-one-out cross-validation, I also suggest to learn about the Bias-Variance-tradeoff in model validation, e.g. by starting off with Variance and bias in cross-validation: why does leave-one-out CV have higher variance?.
Regarding the specifics of your question:


*

*When you use as many folds as you have samples in your data set, you force the k-fold-Crossvalidation to become Leave-One-Out (assuming that it is non-stratified crossvalidation).

*The documentation of the Matlab Classification Learner App states, that there is a way to export the steps done by the app to code for later reuse. So you could use this as starting point for your own experiments. Quote:



To use the model with new data, or to learn about programmatic
  classification, you can export the model to the workspace or generate
  MATLAB® code to recreate the trained model

