# Model selection and performance evaluation with different sample sizes

Suppose there are K experimental units. Each unit is associated with its own dataset consisting of 400 observations. For each unit, we set up a two-sample test, 200 vs 200. Because of a large sample size, we can avoid making parametric assumptions and we estimate the “true” p-value for each unit via bootstrap.

We are interested in small sample (2-5 observations per sample) performance. We select 3 observations from each group at random. Because of small sample size, we can't apply bootstrap any longer, so we employ two parametric methods, Poisson and Normal regressions. For each parametric method we compute K p-values from the 3 vs 3 test. Then the following exchange takes place between two researchers, A and B:

A: For a given model, I define its performance as the distance between the K-dimensional vector of p-values it generates and the K-dimensional vector of “true” p-values. The smaller the distance, the better the performance. In that sense, it's best to use Poisson for 3 vs 3 test.

B: You suggest we compare p-values from 3 vs 3 test with the p-values from 200 vs 200 test. As we know, p-value depends on the sample size, so I don't think these are comparable.

A: That's beside the point. The purpose of any finite sample experiment is to obtain inference about the entire population. In practical terms, the conclusion I draw from 3 vs 3 test should be as close as possible to what I could get with a much larger study, such as 200 vs 200. For instance, if I were to rank the units by statistical significance, I would love the 3 vs 3 test to generate the p-values as close as possible to those of 200 vs 200 test. I am more likely achieve that using Poisson than Normal.

B: But if we run Poisson and Normal on the entire sample, 200 vs 200, we can see that Normal p-values are much closer to the “true” p-values. That suggests that the population distribution is closer to Normal than to Poisson.

A: Again, that's beside the point. We are lucky to have such a large dataset this time, but in the future we are bound to running two-sample tests with 2-5 observations per sample. What works or doesn't work for 200 vs 200 or even 30 vs 30 tests is irrelevant.

B: Are you saying that it's ok to use a misspecified model on a small sample as long as it results in the same inference as a well-specified model on a much larger sample?

A: Sure. The goal is to match the "true" p-values as close as possible in a small sample situation. The better the match, the better the model.

Is A right? If not, then in order to accommodate B, one has to generate the “true” p-values for the small sample test as well. Is there a way to do that? If you don't like A's approach in general, please suggest some other way to evaluate the small sample performance in this situation.

• Even if we assume that this study design is reasonable (which it isn't), A is still wrong. Matching p-values doesn't give you any information about the underlying models or how well they fit. It's not sound, statistically. It's just p-hacking. Commented Jun 10, 2015 at 20:19
• A's point is that in the end, the decision is made based on p-values. E.g. one can use AIC to pick the model, but it doesn't guarantee that the small sample p-values will be aligned the best with the large sample p-values. Moreover, B thinks it's ok to use the p-value approach as long as the sample size is the same. You seem to suggest that even if sample size is the same, it's not ok to use p-values to determine the best model. It's not clear to me why you think so. Commented Jun 12, 2015 at 12:39
• What does it mean with respect to the models and the data if the p-value for a parametric model fit to 3 observations is similar to the p-value for a non-parametric model fit to 400 observations? Even leaving aside the question of what models you are using and how you are calculating these p-values, what exactly does that comparison mean conceptually? How meaningful is it that these two different models (with different tests applied to generate these p-values) share similar Type I error rates, especially without correcting for Type II/power? Commented Jun 12, 2015 at 15:42
• As "A" already mentioned, the meaning is as follows: when running a small sample test, we would like the decision to be as close as possible to what we could obtain with a much larger sample. Commented Jun 13, 2015 at 19:09
• The problem here is that we have a large sample, but still we don't know what experimental units should be called by rejecting the null. For instance, one could say that if the large sample p-value is below 0.001 / above 0.5, then H0 is considered false/true. Then it is possible to construct the ROC curve for a small sample test. Would that be ok? What "A" proposes is essentially the same, only on a continuous scale instead of true/false. Commented Jun 13, 2015 at 19:15

As Ryan said: A and B are equally wrong!

Assume we want to test for a difference in the mean of two populations with any distribution, and assume we know that the true p-value is 0.032 for a particular case (to choose a random value).

1) The fact that a t-test, a Poisson or any other test are most close to the "true p-value" FOR THIS PARTICULAR CASE is absolutely no evidence in favor of this test IN GENERAL, because with wrong distributional assumptions, the tests may be arbitrary close to the true p-value for some differences in the mean and/or sample sizes, and totally wrong in some other situations.

2) To show that a test works better than another in general, you would need to show that it works better in ALL situations that are of practical relevance to you.

3) To do this, the easy way would be to simply test if the assumptions of the test are met, i.e. if the distributions are normal etc. The you can rely on established theory about their properties. If the assumptions are not met for either test, and you still want to know which is better (likely both are bad choices though), you could a) resample or b) fit the distribution to generate new data, and use this to compare Type I error and power for different a) sample sizes b) differences of the mean

• The problem with 3) is that we can't generate the data with a known difference in means because we don't know what kind of relationship, if any, between the mean and the variance exists in the true distribution. What we can do is to resample, say, 3 observations per sample, apply a few models and get the corresponding p-values and regression coefficients. After that, how do we determine what model did the best? Commented Jun 15, 2015 at 21:06
• Also, here we talk about not just one test, but a large number, K, of tests, meaning that the difference in means covers the entire range that is of practical significance to us. Performance is also measured separately for each subsample size from 2 to 5. In that sense, we cover all of the situations that are of practical relevance. Finally, we obtain not just point estimates but also CI for our performance evaluation metric by resampling. Commented Jun 15, 2015 at 21:15
• If you don't know how the distribution behaves for other differences in the mean, it's pretty pointless to discuss what the best test would be. For the variance alone it shouldn't be a problem as long as you have a test that can adjust to different variance, so Gaussian assumptions would be more robust than Poisson. Commented Jun 16, 2015 at 9:34
• It seems nice that Normal has a separate variance parameter, but there is also the issue of bias-variance tradeoff. For a small sample, it may be better to apply a model with fewer parameters. Commented Jun 16, 2015 at 14:22