Different common meanings of training set, validation set, test set In this previous question of mine it was pointed out in an insightful comment by ReneBT that the usage of the 3 terms training data set, validation set and test set is not uniform across the cross validated community (which lead to different interpretation in the answers to my question, which in its first edit didn't fix the terminology regarding these terms). 
Can you please point out what the proper usage versus the common usage is? And what variations of common usage are often met in practice?
(This highly upvoted question has collected some definition, but even there there seems to be some disagreement about the terminology...)
It seems to me that there in particular regarding validation there are various different approaches: use crossvalidation on the training data set to do model selection (and find the optimal model parameters), or use a separate validation set for that.
 A: 
still to be completed. This is just a saved early version so far.

I think this a very good and important question: IMHO the common (frequently used) terminology causes a lot of confusion. 
I'll first outline how I think the common usage of the terms evolved, and why I find it particularly confusing. In the 2nd part I'll propose an alternative naming scheme that hopefully avoids this confusion.
Common/frequent usage
The most common terminology I see is train/validation/test (see e.g. Wikipedia: Training, validation, and test sets).
I think this splitting terminology developed (historically)


*

*In easy/comfortable circumstances, i.e. 


*

*small number of features $p$, 

*sample size $n \gg p$ - even if $n$ itself is not that large,

*no substructure (clustering, data hierarchy, correlation between [groups of] samples/cases) within the data and 

*low model complexity together with

*mathematically well understood model (e.g. linear model)


both model and generalization error can directly be derived (analytically) from within the data set (e.g. prediction intervals of univariate linear model). 
In this situation, 


*

*the risk of overfitting is negligible (due to $n >> p$ together with low complexity of model: we have many degrees of freedom left),

*we can therefor use training error as good approximation to generalization error

*$n \gg p$ may be reached with absolutely rather small $n$ (e.g. a univariate linear model with slope and intercept does fine with $n$ = 10.). Analytical expressions e.g. predicition intervals 


*As 


*

*the number of features $p$ increases

*it becomes more difficult to keep 



Difficulties with this terminology


*

*Verification and validation


Proposed terminology:


*

*training

*optimization

*verification

