Model selection and performance evaluation with different sample sizes Suppose there are K experimental units. Each unit is associated with its own dataset consisting of 400 observations. For each unit, we set up a two-sample test, 200 vs 200. Because of a large sample size, we can avoid making parametric assumptions and we estimate the “true” p-value for each unit via bootstrap.
We are interested in small sample (2-5 observations per sample) performance. We select 3 observations from each group at random. Because of small sample size, we can't apply bootstrap any longer, so we employ two parametric methods, Poisson and Normal regressions. For each parametric method we compute K p-values from the 3 vs 3 test. Then the following exchange takes place between two researchers, A and B: A: For a given model, I define its performance as the distance between the K-dimensional vector of p-values it generates and the K-dimensional vector of “true” p-values. The smaller the distance, the better the performance. In that sense, it's best to use Poisson for 3 vs 3 test.
B: You suggest we compare p-values from 3 vs 3 test with the p-values from 200 vs 200 test. As we know, p-value depends on the sample size, so I don't think these are comparable. A: That's beside the point. The purpose of any finite sample experiment is to obtain inference about the entire population. In practical terms, the conclusion I draw from 3 vs 3 test should be as close as possible to what I could get with a much larger study, such as 200 vs 200. For instance, if I were to rank  the units by statistical significance, I would love the 3 vs 3 test to generate the p-values as close as possible to those of 200 vs 200 test. I am more likely achieve that using Poisson than Normal.
B: But if we run Poisson and Normal on the entire sample, 200 vs 200, we can see that Normal p-values are much closer to the “true” p-values. That suggests that the population distribution is closer to Normal than to Poisson.A: Again, that's beside the point. We are lucky to have such a large dataset this time, but in the future we are bound to running two-sample tests with 2-5 observations per sample. What works or doesn't work for 200 vs 200 or even 30 vs 30 tests is irrelevant.  
B: Are you saying that it's ok to use a misspecified model on a small sample as long as it results in the same inference as a well-specified model on a much larger sample?A: Sure. The goal is to match the "true" p-values as close as possible in a small sample situation. The better the match, the better the model.
Is A right? If not, then in order to accommodate B, one has to generate the “true” p-values for the small sample test as well. Is there a way to do that? If you don't like A's approach in general, please suggest some other way to evaluate the small sample performance in this situation.
A: As Ryan said: A and B are equally wrong!
Assume we want to test for a difference in the mean of two populations with any distribution, and assume we know that the true p-value is 0.032 for a particular case (to choose a random value). 
1) The fact that a t-test, a Poisson or any other test are most close to the "true p-value" FOR THIS PARTICULAR CASE is absolutely no evidence in favor of this test IN GENERAL, because with wrong distributional assumptions, the tests may be arbitrary close to the true p-value for some differences in the mean and/or sample sizes, and totally wrong in some other situations.
2) To show that a test works better than another in general, you would need to show that it works better in ALL situations that are of practical relevance to you. 
3) To do this, the easy way would be to simply test if the assumptions of the test are met, i.e. if the distributions are normal etc. The you can rely on established theory about their properties. If the assumptions are not met for either test, and you still want to know which is better (likely both are bad choices though), you could a) resample or b) fit the distribution to generate new data, and use this to compare Type I error and power for different a) sample sizes b) differences of the mean 
