(1) "What is the probability that a resnet with 20 layers is on average better on these datasets than a resnet with 10 layers?" In order to define a probability, you need a probability model. I'm not familiar with resnet, however you list "variance due to test dataset sizes, variance due to different weight initializations". For weight initialisations you'd need to define a probability distribution in order to define the probability that you apparently want. If you can generate weight initialisations randomly using some probability mechanism (and some other input parameters you may want to explore) many times on all datasets (see the last paragraph in (1) for aggregating results over the 10 datasets), you can estimate the probability that resnet(20) is better than resnet (10) on the datasets just by looking at the relative frequencies; and standard Bernoulli/Binomial theory will give you confidence intervals. Of course the probability that you get refers to the specific distribution you used for choosing the input parameters, but without such a specification, no probability can be computed.
One thing that is important here is the question what is random and what is fixed. If you ask whether one method is significantly better than another on dataset X, there is no "variance due to test dataset size", because dataset X has just one size, which is given. The only thing that can be random here are random choices when running the methods. Also, as long as you are only asking what is better "on these datasets" you have just the fixed set of dataset sizes that you have, there is no variation.
This is what you asked. If you want to generalise to other datasets, you open a can of worms, because then you'd need to have a distribution of observed datasets drawn randomly from a well defined population, and I don't think you have that (this is why I asked earlier where the datasets are from).
There is also a certain issue with the problem definition. My interpretation here is "what is the probability that resnet(20) is better than resnet(10) if any of the 10 test datasets is randomly drawn", and this means that you should run things so that each time first you draw one of your test datasets at random before running both methods of it. However you may also be interested in something else, for example averaging accuracy differences over the 10 datasets, in which case you need to run each time each analysis on all 10 datasets, compute the accuracy average over all datasets, and record whether this is larger or smaller than zero. Other ways of operationalising this are conceivable.
(2) "How can you quantify whether such accuracy difference is by chance or because it indeed is the case that one of the methods is better on this particular dataset? (the concern here is the multiple hypothesis testing and how to account for it, while taking care of all the other sources of variance as well)."
Here's something important: As I tried to respond to your first question above literally as you asked it, the computation of the probability that resnet(20) is better than resnet (10) on that dataset is not a p-value, and what was done there was not a hypothesis test! A hypothesis test addresses the question: "How likely is it, under some null hypothesis (here probably "methods are equally good"), that a certain test statistic comparing results is as large or larger than what was actually observed, to make statements about to what extent the data are compatible with the null model. This means that a test will tell you how likely a value of a statistic is, assuming methods are equal, whereas what you asked was "how likely is it that one method is better than the other", which is a different question. This means that if you follow my response to (1), you don't actually run multiple tests.
One possibility to address the second question, assuming that there are in fact only random differences between methods, is using a permutation test.
(a) Run many replicates of analyses with both methods as explained in (1) on all 10 datasets.
(b) For every dataset, randomly permute results and assign a random sample of half of them to method 1 and the other half to method 2. Record the accuracy differences. Also record each time the maximum accuracy difference over all 10 datasets. If you run, say, 1000 replicates, you obtain a dataset of 1000 maximum accuracy differences.
(c) The relative frequency of those that are bigger than what you actually observed gives you a permutation p-value testing the null hypothesis that the two methods only differ randomly, i.e., if this is very small, it is evidence that your observed maximum accuracy difference is actually meaningful and the better method is properly better on at least that dataset.
(By the way, you can do the same thing on any single dataset to have test p-values for any specific dataset, if this is what you want more than what I had explained in (1).)
