I'm in a little debate with a colleague about hypothesis testing and interpretation. Your insights, link, or edits to get us on track are really appreciated! I won't tell you which side of the debate I'm on, but here is the situation:

Situation 1

Subjects are divided into 4 groups, in each group N=100. Each participant is subjected to a different stressor determined by their group assignment, then their blood is tested for levels of some stress protein.

  • Group 1 participants sit quietly in a room: protein level 100 +/- 4(SEM)
  • Group 2 participants made to jog a half mile: protein is 140 +/- 4
  • Group 3 participants verbally berated: protein is 200 +/- 4
  • Group 4 participants mauled by bobcat: protein is 250 +/-4

For the sake of argument, let's say that in a prior experiment we operationally define stress in some way such that treatment 4 is exactly 4 times as stressful as treatment 1 (and likewise for 3 and 2 - they are 3 times and 2 times as stressful as treatment 1).

And we are assessing the validity of this claim: "Stress-protein levels are correlated with the degree of an applied stressor."

The two sides of the debate are:

(1) The claim in question is valid. Ignore the discrete nature of the group assignments and simply plot all of the stresses and protein responses in scatterplot form (x: individual's assignment 1, 2, 3, or 4; y: individual's protein response). By the group means and SEM's given, and with N=400 points in the scatterplot, the regression will clearly be significant. Treating the group identity as a continuous variable only makes the hypothesis test stronger - more stringent than the ANOVA; so we don't have to worry about this method adding extra structure to the data.

(2) The above method is ok, but N should be treated as 4, not as 400. Because the results within group may be correlated in ways we haven't measured (the different stressors might increase the stress protein in ways unrelated to stress itself, for example - the stress protein may also be an 'exercise protein'), we don't get to carry the large within-group N (or the SEMs) over into our correlational interpretation. With N=4, the correlation doesn't meet our significance criterion.

* Situation 2 *

This is a modification of situation 1 that tries to highlight (argument 2) a little more. Andy W, Peter, and whuber were responding to the first situation. Thanks to them for their comments! The answers really helped me formalize my thinking about this. So, if you are still interested in the question, please, imagine that the above study were run with only Group 1 and Group 2. Noone was verbally berated or mauled, only subjected to sitting in a room or compulsory track running.

And assume that, unknown to the experimenters, but known to us as people putting the hypothetical question forward, 'stress protein' only got its name by historical accident. The ground truth is that this protein is upregulated by track running, and track running only. In our experiment, the track running stressor is twice as stressful as sitting quietly in a room. Designing a study with many different stressors, some involving track running and others not, would break the correlation between stress incurred and stress protein produced, but those studies were not run.

If the rationale in Peter's and whuber's answers can be applied to situation 2, it seems like we would accept the claim "Stress-protein expression is significantly correlated with the degree of the applied stressor". Group 1 and Group 2 still each have N=100. In scatterplot mode, the beta parameter is still highly significant with N=200. And in discrete-group regression mode, B1 and B2 are significantly different.

The appealing thing about (argument 2) is that, for the purpose of assessing the claim in question, we can compose a new scatterplot with N=(number of groups). On the x axis is the degree of stress we attribute to each condition, and on y is the amount of protein produced, and the confidence intervals around Bt(i) in whuber's second model indicate the confidence in our placement of those points on the y axis. Because the SEM's are small, our confidence is high, but we are still only comparing a small number of levels of applied stress, and we are forced to make a weaker claim instead: "Stress protein levels are modulated differently by the different treatment groups." This weaker claim doesn't have the monotonic relationship that's kind of implicit in the real claim we're evaluating.

I think it's safe to say that a significant monotonic (much less, linear) relationship between stressor and protein response can't be made when stress is administered at only 2 levels. With 4 levels, a monotonic relationship might be apparent, but not with a low p value. With 100 levels (and some care taken to diversify also the types of stress, still assuming that we can perfectly quantify stress), there's real confidence that the stress IV is really the right level of description for the protein response. Other causes of elevated protein can't 'come along for the ride'.

I understand that in Situation 2 I'm clearly stacking the deck against the correlation claim (by telling you that more data would invalidate the claim). So, I hope the question isn't getting too.... semantic or arbitrary? But this gets more to the heart of my concern even in Situation 1. Doesn't ("Stress-protein levels are correlated with degree of stressor") imply monotonicity or linearity? What are the requirements for inferring that sort of monotonicity? Do we get to carry our large N from the original design all the way through? Or do we have to 'discount' the N to account for the fact that we are only testing the IV at a small number of levels, and when the number of levels (groups) is small, there are other ways that Group 1 results could significantly differ from Group 2 results - ways that are unrelated or only tangentially related to the IV we're making our claim about?

Another way of asking the question

Sorry for using so many words. Re-reading the question, I think I'm putting it in a way that's confusing. I don't want to gut the original question because that would probably put the answers I've gotten so far out of context. So I'm trying to add some clarity to the question here.

Forgetting about "samples sizes" (a term I'm probably misusing in the title and elsewhere in the question), I'd like to ask: In what conditions is it Ok to infer a linear relationship between independent and dependent variables? I mean that first on the scientific level (what aspects of the study design constrain the language that we are allowed to use when summarizing the results), and second on the mathematical level (what is the right statistical model to fit to the data to get our significance answer, given the language we chose to summarize the data in the first place).

And the specific case I'm most interested in is the case where the independent variable is sampled at only a small number of levels, with a lot of sampling within each level. The scientifically interesting claim in my case is that my manipulation causes an effect in a more-or-less linear way.

The problem is that I don't have direct control over the thing that I posit as the 'cause'. In the example above, I want to control 'stress', but I can only do so indirectly, by imposing different conditions on different subjects. I have a limited number of tricks to impose known amounts of stress on subjects, but I don't know what other effects those manipulations have.

In making the claim that my putative cause is linearly causing an effect, the first step is of course to fit a regression model of some sort. The second step, I think, is to fend off alternative explanations/rival hypotheses that assert that my significance test would give me a 'yes' answer even though the thing I claimed to be controlling does't linearly cause my effect.

One alternative explanation is simply: stress doesn't cause the increase in stress protein. Track running causes the increase, and my study design didn't control for a track running effect. A study design that administered track running and stress at many different combinations of levels would have made this clear. But, the study has already been run.

So the final form of the question may be more a matter of cultural norms, or may be field-specific. But I thought that maybe statisticians have formalized these kinds of issues about inference, or at least have had more experience with them than I've had and could give some insight.

Just statistically speaking, in 'Situation 1' above, there is probably a very significant linear relationship between applied stress, and the amount of produced stress hormone. But with the types of caveats in mind that I've laid out [with only 4 levels of administered stress and very little knowledge of how those 4 stress-inducing methods could have indirectly increased stress protein] can we make the claim that stress causes stress protein in a more-or-less linear way? (my feeling is that we can't).

If we can, then couldn't we draw the same conclusion from the data given in Situation 2? And by construction in situation 2, there is no such direct relationship - the apparent relationship only falls out by accident when certain choices of stressor are used.

If we can't draw our preferred conclusion in Situation 1, could we have drawn that conclusion if we had sampled the IV at many more levels, and we had also tried to dissociate the induced stress from as many other potential causes of increased stress protein as possible - by assigning treatments that combine high and low stress levels, in turn, with high and low levels of other potentially confounding causes)? (my feeling here is that, in this case, yes, we could draw the stress --> stress protein conclusion).

Proposed revision for the question title: 'How valid are correlational interpretations on study designs when the independent variable is sampled at only a few levels? Can we compensate for having few levels by heavy sampling within level?'


To add to whuber's and Peter Flom's responses, the continual reference to "N = 4" seems to suggest you are still confusing parameter estimates with sample size (as Peter said, your sample is 400).

Typically we represent a causal relationship as a dependent variable being some type of function of various independent variable(s). From observed samples we attempt to estimate the parameters (which is a characteristic of the population) from the sample. Ultimately, there are limits on our ability to make these estimates, and hence we always impose some type of functional form for how the independent variable(s) effect the dependent variable. Your option 1 imposes a linear relationship between X and Y (which is inherently monotonic). Option 2 sounds like (as whuber already stated) that you want to estimate 4 parameters (one for each group, which happen to be 4 discrete locations of the same independent variable). When restated like this, the question then becomes how to empirically evaluate the model produced by option 1 versus the model produced by option 2.

Then you go on to note, that even if you fit the 4 parameter model, what is to say that it is not truly 50, or 100, or 1,000 parameters that characterize the relationship between "stress" and "stress-proteins"? For empirical evidence, one may assess the absolute fit of the model (in a nutshell how well does the model appear to replicate our observations) as opposed to the relative fit between the two models (just because option 2 appears better than option 1 does not mean option 2 is inherently correct).

I hope reformulating your question in these terms helps to clarify your thoughts. I've avoided talking about mixed models (or referring to the actual parameters as random variables of some "super population" (i.e. multi-level modelling) because on a second reading of your question I don't think this is what you intended). Although you have added a substantial amount of text in the last edit, what your actual question is is still very confusing (IMO references to sample size and p-values do not belong anywhere in the question and only add confusion).

In regards to the updated question, as far as I can tell you now have two separate questions (in the final section, Another way of asking the question).

  1. I can only sample X at certain discrete levels, can I make inferences about the relationship between X and Y at locations along X that I haven't sampled?
  2. I can't manipulate X directly, and the best I can do is observe indirect proxies that theoretically influence X (as well as other things). Can I make statements about how X influences Y?

In regards to 1, of course we can't say anything about levels of X we can't observe (and over-sampling X within those specific discrete levels won't solve that problem either). But, that doesn't preclude one from logically applying the evidence for the relationship between X and Y within those discrete locations to areas outside of those sampled locations. If the evidence within the sampled locations is consistent with a linear and monotonic relationship between X and Y it would be silly to argue otherwise unless there is strong theoretical reasons to expect the relationship to be different outside of those sampled levels (between them IMO would be a tough sell in most situations).

For 2 (as I have phrased it), I would suggest asking a separate question on the forum. It's pretty distinct from the other issues broached so far in the question and answers (and is an interesting question deserving its own specific answers.)

  • $\begingroup$ Thanks Andy - yes paragraph 2 helps a bunch. I like your way of characterizing this: option 1 implies linearity, option 2 implies different parameters for each group. What I'm trying to get at is not whether option 1 or option 2 accounts for more of the variance, but instead, what are the requirements for accepting option 1. In the edit I tried to show why option 1 doesn't sit well with me. I'll try to take a third stab at phrasing the question - I agree it's very confusing; you guys have given great answers to what the question seems to ask, but I'm asking the wrong question. $\endgroup$ – ImAlsoGreg Jul 31 '11 at 13:22
  • $\begingroup$ @Im - I have added an updated response. While your question is well articulated, it is quite verbose and still difficult to parse. Instead of adding more text, I think it would be better to attempt to reduce your question to a few key points/questions. $\endgroup$ – Andy W Aug 1 '11 at 3:13

In (1), you are entertaining two possible models. In each, let $i$ be a subject index, $x_i$ be the level of the "applied stressor," and $y_i - \varepsilon_i$ be the measured protein level with iid, zero-mean random variates $\varepsilon_i$. The "continuous variable" model is

$$y_i = \beta x_i + \varepsilon_i.$$

It uses one parameter $\beta$.

Let $t(i)$ denote the treatment applied to subject $i$. The "discrete group" model is

$$y_i = \beta_{t(i)} + \varepsilon_i.$$

It uses four parameters corresponding to the four treatments.

Your data indicate the common standard deviation of the $\varepsilon_i$ is approximately $4 \cdot \sqrt{100}$ = $40$, estimated with 400 - 4 = 396 degrees of freedom (indicating the estimate is likely pretty accurate).

Both models are regression models. In the first, the estimate of $\beta$ is highly significant. In the second, the estimates of the $\beta_t$ are highly significantly different. In both, the variance of the $y_i$ is greatly reduced by the model, showing how effective it is.

Option (2) refers to results that are "correlated within group." I suspect this might be trying to say that the $\beta_t$ should be viewed as four independent realizations of a random variable. In other words, associated with a stressor $t$ is a random effect $\beta_t$ that applies uniformly to every subject in that stressor's group. The subjects' responses are the sum of this random effect and (as it turns out) a relatively small variation $\varepsilon_i$. This, however, does not make scientific sense. It (a) supposes that yes, there is a consistent effect associated with each treatment, yet (b) insists that these effects--which turn out to be much larger than the intrasubject variation $\varepsilon$--should be interpreted as accidental and unrelated to the treatment. The consistency mocks this interpretation.


The second choice is just silly. There are 400 people. That's the N. The question is what to do with those 400. As @AndyW pointed out, the second argument seems to be saying that these should be treated via a mixed model, but I agree with him. This makes no sense here.

The problem with the first argument is with the statement "ignore the discrete nature...."; by stating that you have previously determined that the level of stress in the groups is directly proportional to the group number, you are saying that group number is really a proxy for a continuous variable that just happens to be measured at only 4 levels. (How you would establish this is beyond me, but that's what you stated).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.