It is widely known that one cannot validly test a hypothesis using the data in which the hypothesis-generating feature was observed. So one needs at least two samples, one for generating hypotheses and one to test them.
But the two samples need to be similar in some way, eg. the finding that shoe size correlates with intelligence in a database of 5-15 year olds will likely reproduce in samples of this age group from other countries and times, but not samples of 45-55 year olds.
I wonder: How different do these two datasets have to be? How similar? Are there general formal criteria for their difference/similarity?
Is it valid to examine the same population at different times? Is it valid to partition a dataset into one for hypothesis generation and one for hypothesis testing?