How to use the datasets to practice machine learning algorithms?

Question

I'm a beginner in ML, and I want to practice with some algorithms I learned from the book, after searching stats.SE, I found some places where I can get some datasets. But, how to use them?

For example, the Iris dataset from UCI contains 150 examples, 50 instances for each classes. OK, I thought I should take bulk of this dataset as training examples, spare rest of the dataset as test examples, right? And the problem is how should I divide this dataset? Shuffle them randomly and then draw the training examples randomly?

I think different training set will affect the model learned from it, right? Please give me some advice on how to use the dataset, thanks.

Answer 1

There isn't exactly an established consensus on the "optimal" way to subdivide data into training and test subsets, but one common choice which works well for most purposes is stratified k-fold cross validation. The "k-fold" means that the data set is divided into k different equally sized parts; typically k is a relatively small number such as 5 or 10. "Stratified" means that each fold is constructed so that it will be distributed approximately equally across the class or response variable. For the iris data set, with 150 examples divided evenly among 3 classes, if 10-fold cross validation were used, then each fold would have 15 examples, with 5 examples drawn from each of the 3 classes.

Cross-validation then proceeds in round robin fashion: in 10-fold cross-validation, for example, you re-train the learning algorithm 10 times, each time selecting 9 of the folds as the training set, while reserving a different fold on each iteration for the test set. Repeating the training process 10 times in this way, each time using a different subset of the data for the test set, permits a more precise estimate of the accuracy of whichever ML algorithm is being used.

Answer 2

If your data set is small, you can leave one observation out, and train on the rest and test whether it gets that observation right, and do this for every observation. If you want to cross-validate using a larger dataset, it might be problematic because you have to train a model to test each observation. Therefore, you could leave out, say, five observations and train on the rest and see how many of these 5 the algorithm gets right, so if you have 100 observations you would train 20 models.

Yes, if you trained a model in different data it will make different predictions. Though if you trained the model on almost the same data it will make almost the same predictions.