Why does Cross-Validation in neural networks make sense?

Question

I have 2 similar questions about Cross-Validation in neural networks:

1. If I use the Cross-Validation technique to get a better evaluation of the network's performance. Let's say I use 10-fold Cross-Validation, and on each CV iteration I train the network and then test it on the left-out fold. in the end, the weights in each of those 10 neural networks will be different since it was trained on different data. That means that I compare the performance of 10 different neural networks. So how can I reveal something from all of this?

2. In the case when using CV to grid search for the ideal hyper-parameters (HP). what is the difference between testing each new HP on the same test set or on a new fold? We might get a biased test set either way. The first test set I use for all the HP might be unrepresentative, or the current fold I use to test a specific HP might give me great result only by chance.

Answer 1

1. Even though the 10 neural networks are different, they are built using the same principle on samples of data from the same population. Hence, you will reveal the performance of the prediction procedure (comprised of model selection, tuning, training and the like) on data from the given population. You will expect similar performance on a new sample from the same population predicted with the same procedure.
2. If your sample is homogenous (all points coming from the same population) and you split randomly between the training and test subsamples, then the test subsample will not be systematically biased. If you cross-validate model performance for every given hyperparameter, it is unlikely that the results will be far away from reality. You may be lucky (or unlucky) on one split between training and testing but what are the chances that you will be off on ten splits on average?