Why only three partitions? (training, validation, test) When you are trying to fit models to a large dataset, the common advice is to partition the data into three parts: the training, validation, and test dataset.
This is because the models usually have three "levels" of parameters: the first "parameter" is the model class (e.g. SVM, neural network, random forest), the second set of parameters are the "regularization" parameters or "hyperparameters" (e.g. lasso penalty coefficient, choice of kernel, neural network structure) and the third set are what are usually considered the "parameters" (e.g. coefficients for the covariates.)
Given a model class and a choice of hyperparameters, one selects the parameters by choosing the parameters which minimize error on the training set.  Given a model class, one tunes the hyperparameters by minimizing error on the validation set.  One selects the model class by performance on the test set.
But why not more partitions?  Often one can split the hyperparameters into two groups, and use a "validation 1" to fit the first and "validation 2" to fit the second.  Or one could even treat the size of the training data/validation data split as a hyperparameter to be tuned.
Is this already a common practice in some applications?  Is there any theoretical work on the optimal partitioning of data?
 A: First, I think you're mistaken about what the three partitions do. You don't make any choices based on the test data. Your algorithms adjust their parameters based on the training data. You then run them on the validation data to compare your algorithms (and their trained parameters) and decide on a winner. You then run the winner on your test data to give you a forecast of how well it will do in the real world.
You don't validate on the training data because that would overfit your models. You don't stop at the validation step's winner's score because you've iteratively been adjusting things to get a winner in the validation step, and so you need an independent test (that you haven't specifically been adjusting towards) to give you an idea of how well you'll do outside of the current arena.
Second, I would think that one limiting factor here is how much data you have. Most of the time, we don't even want to split the data into fixed partitions at all, hence CV.
A: This is interesting question, and I found it is helpful with the answer from @Wayne. 
From my understanding, dividing the dataset into different partition depends on the purpose of the author, and the requirement of the model in real world application. 
Normally we have two datsets: training and testing. The training one is used to find the parameters of the models, or to fit the models. The testing one is used to evaluate the performance of the model in an unseen data (or real world data).
If we just do one step in training, it is obvious that there are a training and a testing (or validating) process. 
However, doing this way, it may raise the over-fitting problem when the model is trained with one dataset, onetime. This may lead to instability of the model in the real world problem. One way to solve this issue is to cross-validate (CV) the model in the training dataset. That means, we divide the training datset into different folds, keep one fold for testing the model which is trained with other folds. The winner is now the one which give minimum loss (based on our own objective function) in whole CV process. By doing this way, we can make sure that we minimize the chance of over fitting in training process, and select the right winner. The test set is again used to evaluate the winner in the unseen data.
