I have to compare the two strategies to choose optimal parameters of my NN. I have the following reasoning. Please tell if it is correct!
In train-validation-test split case, the validation dataset is being used to select model hyper-parameters and not to train the model hence reducing the training data might lead to higher bias. Also the validation error deoends a lot on which data points end up in the validation set and hence different validation sets might give different optimal parameters i.e. the evaluation results in higher variance.
In k fold cv , which is a more progressive procedure, each subset and hence every data point is used for validation exactly once. Since the RMSE is averaged over k subsets, the evaluation is less sensitive to the partitioning of data and variance of the resulting estimate is significantly reduced. Also since all the data points are used for training bias is also reduced.