Suppose I have a population with four (or more) disjoint sub-populations which differ from one another by traitishness, the union of which is the whole population. I have an outcome measure on the population, and I want to test the view that people have different outcomes based on trait. Having once studied econometrics and so thinking all statistical tests are regressions, I pick one of the groups to be the reference and set up dummy indicator variables for the other three.
Now, I don’t live in a vacuum, and I kind of know which group is highest and which is lowest on outcome. Moreover, questions about traitishness are often framed in terms of whether some groups do worse than the group that does best. To facilitate answering such questions, I pick the group that I expect to have the highest outcome as the reference group, and test whether the others differ significantly from the intercept.
Now here is my question: Is that cooking the books?
Because it seems to me that, for any four groups, there will be a highest and a lowest mean, and the difference between the group which happens to be the highest and the group which happens to be the lowest is expected to be higher than the difference between two groups picked at random. So by knowing which group is going to be highest and, indeed, the whole order, I am essentially selecting particular differences to test based on implicit pre-tests.
Is that right? I believe that one needs to increase the difference required for significance based on the number of pretests I have used. To how many pre-tests is my general knowledge equivalent? And here I’m just positing cultural knowledge. If I have also done a literature review, and thrown up a few graphs, and printed the VC matrix, I can not imagine how many pre-tests that is equivalent to. Dozens? Hundreds? Millions?
But for social science research you always do preliminary exploratory work. You always do a literature review. These are considered minimal requirements of competence. You couldn’t get your results published if you didn’t do these things.
Um, so I’ve let this question run away with me a little. Actually, I was wondering if the “cooking the books” problem could be avoided by comparing everyone to the overall mean, and whether there is a way to set up, e.g., some linear combination of the dummies (?) that will cause the coefficients to pop out that way, rather than as differences from the omitted category. And if this way has a name.