What is the best way to approach multiple-fold cross validation for a 1 nearest-neighbor model used for prediction?
A common approach to cross validation is to, for example, split the dataset into 10 folds, train on on 9 of them, and test on 1 (repeating for each of the 10 folds).
In a nearest-neighbors model the concepts of "training set" and "test set" do not apply in the same way that they do for a regression. An approximation of the above procedure would be to split the dataset into 10 folds, choose 1 fold as the "test set", and search for nearest neighbors in the remaining 9 (repeating for each fold). I could then calculate the MSE across the 10 folds by comparing predicted and actual responses. Is this a good approach to cross validation? Are there good alternative approaches in this case?
A specific alternative I am considering is doing splits that would leave more data in the test set than the training set. So, for example, I would choose 9 of the 10 folds as the "test set" and search for nearest neighbors in the remaining 1 (repeating for each fold). Is there an advantage/disadvantage to doing it this way?
Some specifics on my data: I have roughly 20K observations, roughly 30 features and am predicting multiple responses (roughly 100). I am interested in using cross validation for model selection / evaluation. Because I am using 1 nearest neighbors, I expect the variance of the model to be high. Multiple-fold cross validation is therefore desirable to get a better estimate of the prediction MSE.