Is analysis of existing data always exploratory, or can it be used for hypothesis testing?

Question

This is a question about the rhetoric of describing analysis done using a public data set or any other pre-existing data set.

Here is the hypothetical situation: A researcher reports that they have a hypothesis. To test this, they take a sample, n, of individuals that meets study criteria from a database of N participants. They run a single, a priori, test of this hypothesis on the sample, and report that the hypothesized group difference, or correlation, or whatever, is greater than 0, p<.05.

Is this exploratory or is it a fine example of hypothesis testing?

Because any single reported analysis of an existing data set might be one of many interrogations, any such report should be framed as exploratory rather than hypothesis testing. In other words, presenting an analysis of an existing data set using the rhetoric of a single sample hypothesis test seems to be misleading the reader to over-interpret the results.

However, I can also see an argument being made for the researcher being given the benefit of the doubt as is similarly the case in all non-pre-registered studies. In other words, can we assume that the researcher did just test this one hypothesis on this single subset of the existing database?

Answer 1

I disagee that all analyses of pre-existing data are exploratory. The scenario you described seems like a textbook-perfect example of a hypothesis test, assuming the investigators generated their hypothesis without looking at the data first. If it was truly an a priori hypothesis, then what would have changed if they went out and made measurements instead of just downloading the data?

Issues with exploratory analysis (data dredging, multiple comparisons, etc) arise when the hypothesis is formed from the same data it is subsequently tested on. If your hypothetical researchers had thumbed through the data and noticed a potentially interesting relationship between two factors, a subsequent test of that relationship provides somewhat weaker evidence for it than if it were tested on an entirely new set of observations. In some cases, it might be possible to collect additional confirmatory data; you could also potentially use one subset of the data for developing your model and then test it on the rest of the data (there are also things like cross-validation if your 'exploration' is automated). I would be interested to hear how (for example) macro economists deal with this, as they often work with data that is collected over long timescales, can't be re-observed, and the researchers are often aware of many trends in the data.

As a practical matter, I think you more or less have to take the authors at their word. Ideally, the authors would explain how they arrived at their hypothesis; it is, of course, possible to come up with some tortured post hoc rationalization too, but those often stick out from the text. Pre-registration would definitely help--it's been going on for a while for clinical trials and some psychologists are advocating for it for basic science-type experiments--but that raises some big logistical hurdles too.

Finally, my inner Bayesian wants to point out that individual studies are rarely worth much in isolation; there's nothing wrong with updating your beliefs somewhat less if the study was either overtly exploratory or you think the authors may have peeked.

Answer 2

Astronomers and astro-physicists predominantly use data that other people have collected. And what one of these scientists collects as evidence will be used by a lot of other people who are doing good science by testing good hypotheses.

The example describes taking n out of a collected sample of N and performing a test. That surely is not a "fine example of hypothesis testing" -- Why not use the rest of the data?

On the other hand, it might be a particular example of testing a narrow hypothesis for the purpose of rejecting it (or modifying it) when something that is supposed to be universally true does not show up in this sample which is sufficiently large.

A "fine example" is going to have to belong to a sufficiently well-developed narrative that there is consequence to the acceptance or rejection. There is "known science" and the hypothesis has a chance of changing expectations. One data set is probably not going to be answer all the questions, but it can be enough to raise questions that are new.

What this has reminded me is various data analyzed for the Flynn Effect (see Wikip). The Flynn Effect is the observation that IQ scores have increased by a couple of points per decade. An early indicator of it was the observation that the manufacturers of IQ and achievement tests have found it necessary to re-standardize their tests every few years in order to keep the IQ mean at 100, etc. Early on, the presence of the Effect debunked the extent that IQ falls with ageing: older testees who answer exactly the same as they did 30 or 40 years earlier will be assigned lower scores on a test that is not cohort age-adjusted

Dozens or hundreds of data sets, mostly collected for other purposes, have been investigated in the subsequent tests of alternative explanations. What makes some of these "fine examples" is how well they address specific cogent arguments