Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

A paper asks four research questions which deal with the measurement of latent trait x, a continuous variable which is acquired through learning or practice, and the speed with which the knowledge or skill can be accessed and used in a large group of participants ($n=500$). The first two research questions are:

  1. Is there a difference in response times between groups with different trait x?
  2. What is the correlation between response times and trait x?

Subjects are assigned to the groups referenced in RQ1 post hoc based on their performance on the trait x test. It seems to me that if there is a correlation between response times and trait x, then one could always find a grouping such that there will be a statistically significant difference between the groups.

So, my questions are:

  1. Can anyone explain to me why both questions need to be asked?
  2. Is there some fundamental statistical fallacy going on here or am I being overly simplistic in my thinking?

edit

The trait could be any psychometrically measured latent trait, but I have often seen these two questions appearing simultaneously in a variety of papers where something has been acquired through learning or practice such as in education or applied linguistics. In this example, it could be something trivial such as the ability to correctly pick the name of a famous person in a picture from a list of possible names.

From what I understand, one of the main problems with the first question is the arbitrary discretisation of a continuous variable (trait x) which results in information loss and possible bias through the arbitrary bins/thresholds which are created. Is this correct? Are there other problems happening here?

real life example

I've been hesitant to give an actual example because then replies usually focus on the specifics whereas I was trying to generalise, but here is one. There are two research questions:

  1. What is the size of the vocabulary of university juniors and how good is their reading comprehension?

  2. Do university students’ vocabulary knowledge and content knowledge influence their reading comprehension?

The researcher administers test of reading comprehension, vocabulary size, and content knowledge. Ignoring the double-barrelled RQ1, so far, so good. But here is the part I can't understand (which is similar to my more abstract question above): The analysis includes:

a correlation analysis
Variables               Vocabulary knowledge Content knowledge    
---------------------   -------------------- -----------------
Reading comprehension   .70**                .41**                
Vocabulary knowledge                         .22**                

**p < .01 (two-tailed)  
a comparison of groups based on vocabulary size test performance
                   Reading comprehension      Content knowledge
                   Mean   SD    t        df   Mean   SD    t     df
-----------------  -------------------------  ----------------------
Above (n = 83)     20.34  6.18  12.25**  244  30.10  4.31  2.06  192
Below (n = 163)    11.23  5.15                28.82  5.12

**p ≦ .01                            
and a multiple regression
Model        Sum of squares  df    Mean square  F       
----------   --------------  ----  -----------  --------
Regression   6719.39         2     154.91       154.91**
Residual     5270.35         243   21.67        
Total        11989.74        245 

Note. R2 = .56
**p < .01    

I simply can not figure out how the second test (a t test) is even justified because the groups are created from a variable which has already been shown to correlate with the DVs in the t tests. Isn't that a foregone conclusion?

note: I realise that the study is not particularly good and that given a multiple regression, both the t test and the correlation are not even relevant here, but this is simply an example of the phenomenon I'm asking about in the question. That is, if a correlation has already been established between two variables, does it make sense to then arbitrarily bin one of the variables into groups and test for a difference between groups. I see variations on this theme quite often and I can't figure out why it's justified.

share|improve this question

2 Answers 2

If the groups are really post-hoc and based on the level of x, then the first test seems unnecessary and also seems to violate the assumptions.

But the second test seems fine; I've seen this sort of thing done lots of times and it seems inherently reasonable

It would help if you provided context: What is x?

share|improve this answer
    
x could be any psychometrically measured trait. I've tried to keep it generic because I've seen the same sort of pattern in several papers and could never figure out why finding differences between post-hoc groups based on their performance was necessary when the analysis also includes a correlation. These tend to be in the field of education, applied linguistics, and other fields where statistics are commonly abused and/or misunderstood. Are there contexts when both a correlation and a test of differences is appropriate? –  post-hoc May 25 at 10:51
1  
A test of differences would be among groups; a correlation is between two continuous variables. So, on that level, it's never possible to do both on the exact same data. I can't think of a case where the exact situation in your RQ1 makes sense, but something similar might, if the groups are naturally occurring or decided a priori. –  Peter Flom May 25 at 11:37

Yours is a good question and valuable to think about. Up front I don't see that both questions really need to be asked, but in helping to convey the results there can be value in both. I think one reason you will see both is because of how hard it is for some (many/all) people to understand certain concepts. For example, in your example you show a correlation of 0.22 which is statistically significant, what does that mean to the average person? and how much would that be influenced by a single outlier? The table comparing means I think is much more meaningful. The difference in reading comprehension is about 9 points when individuals vary from the means by about 6 points on average (a very lay interpretation of standard deviation), so that is a meaningful difference. On the other hand there is only about a 1.3 point difference in content knowledge when individuals vary by more than 4 points due to random chance, that tells me that even if that difference were statistically significant, it is not a difference of practical importance. I think that is much easier for many (including me) to interpret than the correlation coefficient. Whether the significance should be reported is another matter since we already reported significance of the correlations. I would prefer to see a confidence interval on the differences along with some measure or graph showing the variation of the raw data. In the other direction you could also find examples of very strong correlations that represent a magnitude of change in the variable of interest so small that most people would not find interesting.

You are also correct that if you can choose the cut point then you can influence the size of the difference, but if an a priori cut-off (even a mean or median) can keep this reasonably objective for conveying information.

share|improve this answer
    
Surely the regression slope and intercept are much more useful in understanding and interpreting the results than comparing means of arbitrarily created groups, no? –  post-hoc May 31 at 15:40
    
@post-hoc, it depends on your audience and the scaling/units of your variables. I agree that you should not use an arbitrary cut-off, but something that makes some objective sense and is chosen before hand. For some audiences the difference between the averages of 2 groups may be more easily understood than the slope, especially if a 1 unit increase in x is not a representative measure. –  Greg Snow Jun 2 at 14:47

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.