So, I am working on a binary classification problem (using R) and I am having some confusion on when/how to use data splitting and k-fold cv. I have about 50 labeled samples and I want to train various algorithms (SVM, KNN, NB) to make predictions on new data.
My question is: do I need to split my data into a train and test set and perform k-fold cv? To me it seems that if you just perform k-fold cv without a data split then you are training on the test data. However in my research on this topic I find people saying that you can use one or the other, or both. How could you use just k-fold cv without a data split?
Here is example code of my three approaches, could someone please explain to me which is the appropriate choice? I think I may have a fundamental misunderstanding on how cross validation works.
Approach 1:
# load the package
library(caret)
# load the iris dataset
data(iris)
# define training control
trainControl <- trainControl(method="repeatedcv", number=10, repeats=3)
# evaluate the model
fit <- train(Species~., data=iris, trControl=trainControl, method="nb")
# display the results
print(fit)
Approach 2
# load the packages
library(caret)
library(klaR)
# load the iris dataset
data(iris)
# define an 80%/20% train/test split of the dataset
trainIndex <- createDataPartition(iris$Species, p=0.80, list=FALSE)
dataTrain <- iris[ trainIndex,]
dataTest <- iris[-trainIndex,]
# train a naive Bayes model
fit <- NaiveBayes(Species~., data=dataTrain)
# make predictions
predictions <- predict(fit, dataTest[,1:4])
# summarize results
confusionMatrix(predictions$class, dataTest$Species)
Approach 3:
# load the packages
library(caret)
library(klaR)
# load the iris dataset
data(iris)
# define an 80%/20% train/test split of the dataset
trainIndex <- createDataPartition(iris$Species, p=0.80, list=FALSE)
dataTrain <- iris[ trainIndex,]
dataTest <- iris[-trainIndex,]
# define training control
trainControl <- trainControl(method="repeatedcv", number=10, repeats=3)
# evaluate the model
fit <- train(Species~., data=dataTrain, trControl=trainControl, method="nb")
# make predictions
predictions <- predict(fit, dataTest[,1:4])
# summarize results
confusionMatrix(predictions$class, dataTest$Species)