Hyperparameters tuning on resampled validation set? Let's suppose we have trained a neural network on some data set, rather than estimating the hyper-parameters using a basic validation error would not it make more sense to:


*

*Generate N validation sets from the original validation set by sampling with replacement.

*Estimate N validation errors(using the same model) and average them to generate a final validation error.
I thought that this estimate would give a better estimate of the real error if the validation set is small?
 A: What you describe is known as out-of-bootstrap error, one valid resampling-based method to estimate generalization error. 
You can also do repetitions/iterations of cross validation.
For both out-of-bootstrap and repeated/iterated cross validation the repeated resampling helps to average variance that is due to model instability, but there's no way to cheat out of the fact that you have only a finite number of actual cases available. In other words, the more unstable your models are, the more do the iterations/repetitions help. But they do not work miracles. 
However, they are anyways better than a single random split hold-out estimate which is subject to both said model instability and to higher random uncertainty due to a smaller number of tested cases.

Update to answer comment:
If you train the model once and then bootstrap the validation set from data that is independent of the training set, you don't gain much. 


*

*You cannot estimate model instability as only one model is tested. 

*The number of actual cases available does not increase.

*The one thing you can estimate via bootstrapping is the random uncertainty due to the finite size of the validation set. However, in this respect, I'd compare bootstrapping vs. parametric calculation of a confidence interval for the error estimate. How much and whether you gain information at all by bootstrapping over parametric stats will (as usual) depend on sample size (if you reserve, say, only 4 independent cases* for validation, there are only 256 possibilities for the bootstrapped set) and whether the distribution assumptions for the parametric approach (e.g. binomial c.i. for classifier accuracy, normal distribution for regression errors) are met. If they are met, the parametric approach will be superior to bootstrapping.
So yes, bootstrapping would be better than ignoring the fact that finite validation set size implies random uncertainty on the error estimate. But IMHO it is often not superior to parametric estimation of that uncertainty.  

*Nevertheless, out-of-bootstrap would offer another substantial gain in information due to both larger sample base and including possible model instability. The latter is IMHO crucial if the result is to be used for optimization: while for a final model stability may be established in several ways and I'm OK with saying that only stable models will be considered, hyperparameter optimization routinely includes testing too complex and, in consequence, unstable models. So I consider detecting and penalizing instability crucial for optimization. 
* in case 4 sounds ridiculously low, I've been working with data where we maybe had 1000s of repeated measurement of the same subject, but only ≈ 20 subjects, so 4 for validation set is not unheard of (although it is nonsense). And such complex data structures would be one situation where I think bootstrapping the error estimate is better than parametric approaches.
