2
$\begingroup$

It is widely known that one cannot validly test a hypothesis using the data in which the hypothesis-generating feature was observed. So one needs at least two samples, one for generating hypotheses and one to test them.

But the two samples need to be similar in some way, eg. the finding that shoe size correlates with intelligence in a database of 5-15 year olds will likely reproduce in samples of this age group from other countries and times, but not samples of 45-55 year olds.

I wonder: How different do these two datasets have to be? How similar? Are there general formal criteria for their difference/similarity?

Is it valid to examine the same population at different times? Is it valid to partition a dataset into one for hypothesis generation and one for hypothesis testing?

$\endgroup$
  • 4
    $\begingroup$ Randomly splitting a dataset into two parts is not only valid, it is recommended. You want the two samples to be from the same population. $\endgroup$ – Peter Flom - Reinstate Monica Apr 27 '12 at 17:03
4
$\begingroup$

I would suggest that whether two data sets are comparable depends on whether the differences are likely to affect the thing we're interested in.

Shoe size is clearly related to age, so using a different age group would be silly. But employment status shouldn't be related (you would think), so it wouldn't matter if that didn't match.

As to whether it's valid to split a dataset: I would say yes, provided that you do the splitting randomly, and you do it before you've formed any ideas about what you're "expecting" to see. This is probably a lot harder than it sounds...

$\endgroup$
  • $\begingroup$ But "what affects what we're interested in" is THE research question, at least in my field (epidemiology). While it is clear that we don't either know or don't know, but rather have some quantity of knowledge we can build on, it is unrealistic to be certain two populations are comparable. Or is it? As to the data splitting (also @Peter Flom): This is a surprise. I thought that the answer was "no", for if there is an artifact in my current population, caused by some phenomenon unknown to me, it will likely exist in any two partitions regardless of how I split. $\endgroup$ – miura Apr 28 '12 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.