I am having issues understanding some concepts of algorithm comparison/parameter optimization/cross-validation in R
Let's say I want to compare two classification algorithms, such as Random Forests and kNN. I have my data, and first I want to train my algorithms with R's train function from the caret package to find some optimal parameters (so, mtry for RF's and k for kNN, for example).
After that, I want to use those best parameters and estimate my algorithms' accuracies in order to compare them.
Am I supposed to:
A) Train using the full dataset, then perform k-fold cross validation by creating k models (using equal folds for both the algorithms) and averaging the k accuracies?
B) Split first into training and test data (let's say 75/25), train using training data, create one model for each algorithm using training data and then checking its accuracy with the test data
C) Something else which is probably correct
I know I am probably mixing things up and there probably is a correct way of doing this, what is it?