Will a hold-out set of different size than testing set be okey to evaluate performance? I am working in machine learning. I created a model with the following "parameters" (it is for biometrics):
classes: 21
training: 36 samples for each class
testing: 4 samples for each class
Cross validation: 10x10 StratifiedShufflesplit
*Training set and testing set are used during cross validation (it is a 90%-10%).
I also created a hold-out set of 4 samples (never used during training). I chose this size so that it is the same as the testing set.
I want to proof that my model is capable to generalise. So I was planning to do a barplot showing the difference in accuracy of model (Cross validation) and hold-out set.
But I would also like to test the model with the hold-out set, but only with 1 sample (it will be chosen randomly). Would that be ok or am I breaking any machine learning rule, etc? (training of 4 samples vs hold-out of 1 sample)
I want to do that because 30 seconds (1 sample) is better than 2 minutes (4 samples). Not because of accuracy but because of time.
You can see my confusion matrix and accuracy results. 
Thank you
1 sample. 95.24%

4 samples. 89.29%

 A: Look at it this way: once you have selected and fitted your model, you will use it to do "real" predictions. Perhaps you'll predict a single value. Perhaps you'll predict hundreds. Perhaps you don't even know now how many values you will predict with it.
Should the number $n$ of values you predict have an influence on how many folds you use in cross-validation?
I'd say no, it shouldn't. (Especially since you may not even know $n$ when you build the model!)
But the "real" prediction is essentially the same as your holdout. So if there is no reason to have the number of "real" predictions have an impact on your cross-validation, then analogously there is no reason why the number of cross-validation folds should have an impact on your holdout size.
Bottom line: there does not need to be any relationship between the number of folds and the holdout size.
(That said, your sample size of just 40 points for training and 4 for the holdout appears rather small to me. I hope you are not fitting a very complex model to this, cross-validation or not.)
