I'm in a little debate with a colleague about hypothesis testing and interpretation. Your insights, link, or edits to get us on track are really appreciated! I won't tell you which side of the debate I'm on, but here is the situation:
Subjects are divided into 4 groups, in each group N=100. Each participant is subjected to a different stressor determined by their group assignment, then their blood is tested for levels of some stress protein.
- Group 1 participants sit quietly in a room: protein level 100 +/- 4(SEM)
- Group 2 participants made to jog a half mile: protein is 140 +/- 4
- Group 3 participants verbally berated: protein is 200 +/- 4
- Group 4 participants mauled by bobcat: protein is 250 +/-4
For the sake of argument, let's say that in a prior experiment we operationally define stress in some way such that treatment 4 is exactly 4 times as stressful as treatment 1 (and likewise for 3 and 2 - they are 3 times and 2 times as stressful as treatment 1).
And we are assessing the validity of this claim: "Stress-protein levels are correlated with the degree of an applied stressor."
The two sides of the debate are:
(1) The claim in question is valid. Ignore the discrete nature of the group assignments and simply plot all of the stresses and protein responses in scatterplot form (x: individual's assignment 1, 2, 3, or 4; y: individual's protein response). By the group means and SEM's given, and with N=400 points in the scatterplot, the regression will clearly be significant. Treating the group identity as a continuous variable only makes the hypothesis test stronger - more stringent than the ANOVA; so we don't have to worry about this method adding extra structure to the data.
(2) The above method is ok, but N should be treated as 4, not as 400. Because the results within group may be correlated in ways we haven't measured (the different stressors might increase the stress protein in ways unrelated to stress itself, for example - the stress protein may also be an 'exercise protein'), we don't get to carry the large within-group N (or the SEMs) over into our correlational interpretation. With N=4, the correlation doesn't meet our significance criterion.
* Situation 2 *
This is a modification of situation 1 that tries to highlight (argument 2) a little more. Andy W, Peter, and whuber were responding to the first situation. Thanks to them for their comments! The answers really helped me formalize my thinking about this. So, if you are still interested in the question, please, imagine that the above study were run with only Group 1 and Group 2. Noone was verbally berated or mauled, only subjected to sitting in a room or compulsory track running.
And assume that, unknown to the experimenters, but known to us as people putting the hypothetical question forward, 'stress protein' only got its name by historical accident. The ground truth is that this protein is upregulated by track running, and track running only. In our experiment, the track running stressor is twice as stressful as sitting quietly in a room. Designing a study with many different stressors, some involving track running and others not, would break the correlation between stress incurred and stress protein produced, but those studies were not run.
If the rationale in Peter's and whuber's answers can be applied to situation 2, it seems like we would accept the claim "Stress-protein expression is significantly correlated with the degree of the applied stressor". Group 1 and Group 2 still each have N=100. In scatterplot mode, the beta parameter is still highly significant with N=200. And in discrete-group regression mode, B1 and B2 are significantly different.
The appealing thing about (argument 2) is that, for the purpose of assessing the claim in question, we can compose a new scatterplot with N=(number of groups). On the x axis is the degree of stress we attribute to each condition, and on y is the amount of protein produced, and the confidence intervals around Bt(i) in whuber's second model indicate the confidence in our placement of those points on the y axis. Because the SEM's are small, our confidence is high, but we are still only comparing a small number of levels of applied stress, and we are forced to make a weaker claim instead: "Stress protein levels are modulated differently by the different treatment groups." This weaker claim doesn't have the monotonic relationship that's kind of implicit in the real claim we're evaluating.
I think it's safe to say that a significant monotonic (much less, linear) relationship between stressor and protein response can't be made when stress is administered at only 2 levels. With 4 levels, a monotonic relationship might be apparent, but not with a low p value. With 100 levels (and some care taken to diversify also the types of stress, still assuming that we can perfectly quantify stress), there's real confidence that the stress IV is really the right level of description for the protein response. Other causes of elevated protein can't 'come along for the ride'.
I understand that in Situation 2 I'm clearly stacking the deck against the correlation claim (by telling you that more data would invalidate the claim). So, I hope the question isn't getting too.... semantic or arbitrary? But this gets more to the heart of my concern even in Situation 1. Doesn't ("Stress-protein levels are correlated with degree of stressor") imply monotonicity or linearity? What are the requirements for inferring that sort of monotonicity? Do we get to carry our large N from the original design all the way through? Or do we have to 'discount' the N to account for the fact that we are only testing the IV at a small number of levels, and when the number of levels (groups) is small, there are other ways that Group 1 results could significantly differ from Group 2 results - ways that are unrelated or only tangentially related to the IV we're making our claim about?
Another way of asking the question
Sorry for using so many words. Re-reading the question, I think I'm putting it in a way that's confusing. I don't want to gut the original question because that would probably put the answers I've gotten so far out of context. So I'm trying to add some clarity to the question here.
Forgetting about "samples sizes" (a term I'm probably misusing in the title and elsewhere in the question), I'd like to ask: In what conditions is it Ok to infer a linear relationship between independent and dependent variables? I mean that first on the scientific level (what aspects of the study design constrain the language that we are allowed to use when summarizing the results), and second on the mathematical level (what is the right statistical model to fit to the data to get our significance answer, given the language we chose to summarize the data in the first place).
And the specific case I'm most interested in is the case where the independent variable is sampled at only a small number of levels, with a lot of sampling within each level. The scientifically interesting claim in my case is that my manipulation causes an effect in a more-or-less linear way.
The problem is that I don't have direct control over the thing that I posit as the 'cause'. In the example above, I want to control 'stress', but I can only do so indirectly, by imposing different conditions on different subjects. I have a limited number of tricks to impose known amounts of stress on subjects, but I don't know what other effects those manipulations have.
In making the claim that my putative cause is linearly causing an effect, the first step is of course to fit a regression model of some sort. The second step, I think, is to fend off alternative explanations/rival hypotheses that assert that my significance test would give me a 'yes' answer even though the thing I claimed to be controlling does't linearly cause my effect.
One alternative explanation is simply: stress doesn't cause the increase in stress protein. Track running causes the increase, and my study design didn't control for a track running effect. A study design that administered track running and stress at many different combinations of levels would have made this clear. But, the study has already been run.
So the final form of the question may be more a matter of cultural norms, or may be field-specific. But I thought that maybe statisticians have formalized these kinds of issues about inference, or at least have had more experience with them than I've had and could give some insight.
Just statistically speaking, in 'Situation 1' above, there is probably a very significant linear relationship between applied stress, and the amount of produced stress hormone. But with the types of caveats in mind that I've laid out [with only 4 levels of administered stress and very little knowledge of how those 4 stress-inducing methods could have indirectly increased stress protein] can we make the claim that stress causes stress protein in a more-or-less linear way? (my feeling is that we can't).
If we can, then couldn't we draw the same conclusion from the data given in Situation 2? And by construction in situation 2, there is no such direct relationship - the apparent relationship only falls out by accident when certain choices of stressor are used.
If we can't draw our preferred conclusion in Situation 1, could we have drawn that conclusion if we had sampled the IV at many more levels, and we had also tried to dissociate the induced stress from as many other potential causes of increased stress protein as possible - by assigning treatments that combine high and low stress levels, in turn, with high and low levels of other potentially confounding causes)? (my feeling here is that, in this case, yes, we could draw the stress --> stress protein conclusion).
Proposed revision for the question title: 'How valid are correlational interpretations on study designs when the independent variable is sampled at only a few levels? Can we compensate for having few levels by heavy sampling within level?'