In order to do a data mining work I have to find the best classifiers for my data. What I wonder is if I have to divide my data set into a training test and a test set ? I have to choose between 5 models : Naive Bayes, Neural network, K-NN, decision tree and logistic regression. My point is that if I want to know the performance of each model (by comparing errors for each model), I intended to do a cross-validation and select the best one(using MSE). So my question is, why splitting my data at the beginning if I am assessing the performance of each of the 5 models with cross-validation. I hope I am clear enough. Thank for your help in advance.
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
So my question is, why splitting my data at the beginning if I am assessing the performance of each of the 5 models with cross-validation.
Every classifier has some parameter(s) to tune. You need to find the optimum parameters based on 10-fold cross validation in the training set. Later you have to evaluate the model with optimum parameters on the test set. This makes the division of the dataset important. For finding the optimum parameters you can use 50% of the dataset. Later on, you may evaluate the classifier with rest 50% which is the test set.
Additionally, after finding the optimum parameters with 50% training data, you can use the complete data and evaluate the classifiers based on 10-fold cross validation. Performances with 50% test data and the cross validation one would almost be close.
-
$\begingroup$ Okay, very clear! But for example, for a neural network, do I really need to base my parameters on a cross-validation, or can I directly run the model with = (for the neural network example; 10 neurones) as I have no idea how the code it? $\endgroup$– JulianMay 3, 2016 at 14:26
-
$\begingroup$ @Julian For neural networks if you want to keep the number of hidden layers as 10, then you wont require cross validation (CV) for parameter estimation. But you might get better performance, with say 5 hidden neurons. In that case a CV will be helpful. But finally for final evaluation the 10-fold CV is a must, as neural networks are prone to over-fitting. $\endgroup$ May 3, 2016 at 22:53
$\begingroup$
$\endgroup$
5
You have to divide your data set into a train and test set each time you create a model. For strategies and how to to do that, I would suggest making yourself familiar with cross validation techniques. A good starting point could this book.
-
1$\begingroup$ So, for example, if I split my data into a traning set (75% of my data) and a test set (25%) and I run a linear discriminant analysis. Do I have to run a cross validation on the test set to obtain the performance of my model or, do I only need to run my model with the test set and the results will be enough? (using miss-classification errors as results) $\endgroup$– JulianMay 3, 2016 at 13:42
-
$\begingroup$ What do you mean by "run a cross validation on the test set" and "run my model on test set"? Both, training and test set, are involved into the cross validation procedure. You build/train/create your model using your train set. You assess your models performance by letting it classify your test set. Afterwards the models predictions (predicted from the test set) are compared to the ground truth (the true classes/labels) of your test set. $\endgroup$ May 3, 2016 at 13:54
-
$\begingroup$ So, concretely if I want to assess the performance of my model by a k-fold cross-validation, it will be redudant because I already used the test set to assess the perfomance ? thanks for the attention given! $\endgroup$– JulianMay 3, 2016 at 14:01
-
-
$\begingroup$ Then I didn't understand "Afterwards the models predictions (predicted from the test set) are compared to the ground truth (the true classes/labels) of your test set. " ? $\endgroup$– JulianMay 3, 2016 at 14:20