Is validation set always necessary? Lets say I did the following steps:


*

*Used some separate development set to select some features.

*Decided a priori to use only one learning algorithm (SVM) with only default parameter values.

*Trained a single model on a training set.

*Tested this model on the test set.


Is it OK that I didn't use a validation set, given that I had only one model a priori?
Is this acceptable in a scientific work?
Have in mind that the purpose of my work was only to show that the feature selection was good by showing that even some standard learning algorithm with its default parameter values can learn these features and get good accuracy. I don't claim that I've found the best learning method for my problem (my work is on IR and only uses ML, it's not about ML).
 A: Thinking of training/test/validation as involving different subsets of the data is not necessarily a good idea.  First of it it takes enormous samples to be able to get precise accuracy estimates when data splitting.  More precise estimates of likely future performance of predictive models can be had by using rigorous Efron-Gong "optimism" bootstrapping using the whole sample to develop the model and the whole sample to get a nearly unbiased estimate of future performance for observations from the same stream.
Note that even if you pre-specify a single model, validation may be needed if there are many parameters in the model.
Regarding acceptance in scientific work, see the end of Chapter 9 of Biostatistics for Biomedical Research at http://biostat.mc.vanderbilt.edu/ClinStat .
A: No, I don't think you need a validation set in your case. ML is (broadly speaking) carried in these three phases:


*

*Training phase: Here, we try to learn the best approach for prediction.

*Validation/Test phase: We calculate the performance for our model(s) using validation/test set.

*Application phase: We apply our (final) chosen model to get the predictions.
Now, the Validation/Test phase comprises of two sub-parts:


*

*Validation: We look at our different models, and using the validation set choose the best performing one.

*Testing: We test the accuracy of our selected model on test set.
As you have already decided on the model beforehand, validation set is not needed.
A: If there are no parameters for tuning, or tuning doesn't require trading off something at the cost of something else (in other words either increasing or decreasing a parameter is known to enhance the accuracy of the model), I don't see any reason for having a validation set.
