1
$\begingroup$

Suppose I have some subjects and I give each of them a test. I would end up with a dataset like:

N_tests = 8
N_subjects = 20

### Generate fake subject IDs
subjectIDs = paste0("subject", 1:N_subjects)
print(subjectIDs)

### Generate fake tests
tests = paste0("test", LETTERS[1:N_tests])
print(tests)

### Generate fake test results
test_scores =
  # matrix(rpois(lambda=50, n=N_subjects*N_tests), nrow=N_tests, ncol=N_subjects) %>%
  matrix(rnbinom(size=1, p=0.01, n=N_subjects*N_tests), nrow=N_tests, ncol=N_subjects) %>%
  as.data.frame() %>%
  setNames(subjectIDs) %>%
  mutate(test=tests) %>%
  select(test, everything())
print(test_scores)
   test subject1 subject2 subject3 subject4 subject5 subject6 subject7 subject8 subject9 subject10
1 testA       65       17       68       44       69       26       37       16      243        24
2 testB      177       19       79      524       60       85        6        2        7        42
3 testC       49       16      185       88      377       19       20       86       49        10
4 testD       47       24       17       96      109       94       27      189       31        68
5 testE        7       66      167       32      341       92       13      105       48        59
6 testF       67      123      216       72       60       34      245        6       19       217
7 testG        2      109       86       24       11      128      139       83       25       225
8 testH       46      152       25       29      366       37      183      524      319       173
  subject11 subject12 subject13 subject14 subject15 subject16 subject17 subject18 subject19 subject20
1        99        44       118        63       246        49        63         2        56        30
2        34        96        37       115       177        54       182        65       120         4
3        22        89        69         4        40         5        36        25        81       260
4        37        26        43         0        44       123        41         9       121       101
5        22         4        32        46        72         5        51        59       258        64
6       173       217         6       215         4        70        89        64       127       119
7       250       389       138        59         7       404        11        70         1        82
8        14       123        46         1        35        37       214        56        49        50
> summary(test_scores)
     test              subject1        subject2         subject3         subject4         subject5    
 Length:8           Min.   : 24.0   Min.   : 13.00   Min.   :  4.00   Min.   : 21.00   Min.   :  6.0  
 Class :character   1st Qu.: 52.0   1st Qu.: 36.00   1st Qu.: 34.75   1st Qu.: 26.75   1st Qu.: 34.5  
 Mode  :character   Median :116.5   Median : 97.00   Median : 65.50   Median : 49.00   Median : 40.5  
                    Mean   :121.0   Mean   : 92.38   Mean   : 78.12   Mean   : 81.00   Mean   : 95.0  
                    3rd Qu.:159.2   3rd Qu.:102.25   3rd Qu.:123.75   3rd Qu.:112.75   3rd Qu.: 68.5  
                    Max.   :280.0   Max.   :269.00   Max.   :162.00   Max.   :243.00   Max.   :455.0  
    subject6         subject7         subject8        subject9        subject10        subject11     
 Min.   :  0.00   Min.   :  5.00   Min.   : 50.0   Min.   :  9.00   Min.   : 13.00   Min.   : 16.00  
 1st Qu.: 52.25   1st Qu.: 36.75   1st Qu.: 79.5   1st Qu.: 51.50   1st Qu.: 31.75   1st Qu.: 27.00  
 Median : 87.00   Median : 48.00   Median : 86.0   Median : 87.00   Median : 89.50   Median : 42.00  
 Mean   :156.38   Mean   : 60.00   Mean   :129.2   Mean   : 76.25   Mean   :117.25   Mean   : 84.38  
 3rd Qu.:165.50   3rd Qu.: 75.75   3rd Qu.:160.8   3rd Qu.:106.00   3rd Qu.:145.00   3rd Qu.:121.75  
 Max.   :638.00   Max.   :145.00   Max.   :311.0   Max.   :117.00   Max.   :357.00   Max.   :242.00  
   subject12        subject13       subject14       subject15        subject16        subject17     
 Min.   : 18.00   Min.   : 39.0   Min.   : 48.0   Min.   : 15.00   Min.   :  1.00   Min.   :  6.00  
 1st Qu.: 28.75   1st Qu.: 61.0   1st Qu.:173.2   1st Qu.: 42.75   1st Qu.: 22.75   1st Qu.: 11.25  
 Median : 67.50   Median :133.0   Median :205.5   Median : 47.50   Median : 40.50   Median : 26.50  
 Mean   : 75.62   Mean   :146.6   Mean   :223.0   Mean   : 66.62   Mean   : 66.25   Mean   : 67.12  
 3rd Qu.:127.50   3rd Qu.:209.2   3rd Qu.:221.5   3rd Qu.: 81.75   3rd Qu.: 83.25   3rd Qu.: 77.00  
 Max.   :139.00   Max.   :287.0   Max.   :578.0   Max.   :177.00   Max.   :198.00   Max.   :285.00  
   subject18        subject19        subject20     
 Min.   :  3.00   Min.   : 16.00   Min.   :  1.00  
 1st Qu.: 25.25   1st Qu.: 59.75   1st Qu.:  1.75  
 Median : 32.50   Median : 86.00   Median : 50.00  
 Mean   : 69.00   Mean   :103.50   Mean   : 55.00  
 3rd Qu.: 94.50   3rd Qu.: 95.50   3rd Qu.: 79.25  
 Max.   :193.00   Max.   :331.00   Max.   :159.00  

If I understand correctly, if I want to know if some students are doing significantly better than others on the exams in terms of their mean scores, I could perform ANOVA perhaps followed by adjusting the p-values. Although, probably Kruskal-Wallis because the scores are not normal.

Now consider a different study where I have a metadata table about the subjects like this:

### Generate fake study metadata table
metadata = data.frame(
  subjectID=subjectIDs,
  IQ=factor(sample(c('Smart','Dumb'), N_subjects, replace = TRUE), levels=c('Smart','Dumb')),
  Hair=factor(sample(c('Long', 'Short'), N_subjects, replace = TRUE), levels=c('Long','Short')),
  Social=factor(sample(c('Nice', 'Blah', 'Mean'), N_subjects, replace=T), levels=c('Nice', 'Blah', 'Mean')),
  Age=sample(1:100,N_subjects)
)
print(metadata)
   subjectID    IQ  Hair Social Age
1   subject1 Smart  Long   Blah  71
2   subject2  Dumb  Long   Blah  64
3   subject3 Smart Short   Mean  11
4   subject4 Smart Short   Blah  33
5   subject5  Dumb Short   Nice  10
6   subject6  Dumb  Long   Blah  83
7   subject7 Smart Short   Blah  75
8   subject8  Dumb Short   Nice  96
9   subject9  Dumb Short   Blah  65
10 subject10 Smart Short   Mean  17
11 subject11 Smart  Long   Nice  88
12 subject12 Smart Short   Blah  41
13 subject13 Smart  Long   Mean  61
14 subject14 Smart Short   Blah 100
15 subject15  Dumb  Long   Blah  45
16 subject16  Dumb Short   Blah  78
17 subject17 Smart Short   Mean  25
18 subject18  Dumb  Long   Nice   1
19 subject19 Smart Short   Nice  67
20 subject20  Dumb Short   Nice  50

We know people always get way dumber as they get older (especially if they have children). However, older people might do better on some of the tests such as ones about actually making good decisions. On the other hand, some of the tests might involve playing video games or crawling into small spaces. Likewise, mean people might be better at some tests than nice people.

In this new study, I am not interested in which subjects do good or bad on tests. Instead, I want to know which tests generally result in higher scores for smart people vs. dumb people while controlled for age and social skills.

How do I do this analysis?

$\endgroup$

1 Answer 1

1
$\begingroup$

First I would re-organise your data so that each row represents a test:

SubjectID  TestID  IQ     Social  Age  TestScore
subject1    testA  Smart  Blah    71   65
subject1    testB  Smart  Blah    71   177
subject1    testC  Smart  Blah    71   49
...
subject20   testH  Dumb   Nice    50   50

So, we have repeated measures for SubjectID and TestID (each subject takes all tests). Thus this is a fully crossed design.

One way to analyse these data would be with a mixed effects model, fitting random intercepts for Subject, but not for TestID since your research question is about the tests themselves. So TestID will be a fixed effect, and since you are interested in how the scores differ across different tests for the high/low IQ variable, you are also interested in the interaction between TestID and IQ. Age and social skills will enter the model as fixed effects too, to be treated a potential confounders (you might also want to consider treating the interaction between age and social skills as a potential confounder).

So, a good starting point would be a model such as

TestScore ~ TestID*IQ + Social + Age + (1|subjectID)

In R, you could fit this model using the package lme4, for example.

As a further consideration you might consider using the raw IQ scores instead of categorizing them into high/low.

$\endgroup$
1
  • $\begingroup$ @abalter does this answer your question ? If so, please consider marking it as the accepted answer, if not, perhaps I can help further ? $\endgroup$ Commented Apr 13, 2019 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.