I have the feeling that this is a very basic question, but I have never had any formal schooling in statistics and after extensive googling I feel I just can't come up with the right search terms.
Suppose we have the question: are men faster runners than women? We have a limited group of, say, 30 subjects - 20 male, 10 female. But we have plenty of time, so we let those 30 subjects run a 2k distance every day for two weeks: in the end, we have 14 measurements (time in sec over 2k) for each subject.
What, then, is the best way to deal with this data?
You could collect all male measurements (280) and all female measurements (140) and do a two-sample t-test. But this is clearly wrong, as it would give you a false boost of statistical power (consider the more extreme example of 3 subjects, two male and one female: maybe one male is fast by chance (pretty likely under the null-hypothesis), but this could still cause a highly significant result).
You could do a two-sample t-test using the average time of each subject. (So far, this seems to me the best solution.) Say this test comes back insignificant. After all, your sample size is pretty small. Doing more tests with the same subjects would then not improve your statistical power for this particular question, which may be fair. But now consider the following:
If you do the test for the individual days, you do find that males are significantly faster about half of the time. Even when you correct the significance level for multiple comparison using bonferroni (ie. alpha = 0.05/14), males are significantly faster on two out of fourteen days.
So, can we then reject our null-hypothesis? Is there a more formal way to deal with this? How does one report the result? - "males were faster on two out of fourteen days, so we conclude males are faster" doesn't seem very convincing.
(Google gave many links to repeated-measures ANOVA, but from what I understand this test is designed to compare different points in time; I am not interested in that. All I want to know if men are faster runners than women)