1
$\begingroup$

I was taught in my statistics classes that whether a variable was varied between-subjects and within-subjects made a difference in the results of significance tests. For example, with t-tests, all else being equal (including the scores we get for the DV), for the case where the IV was manipulated between-subjects we may get a non-significant result, but for the case where the IV was manipulated within-subjects, we may get a significant result.

In other words, the same quantitative difference is much more significant when it is detected in the same group of people exposed to different treatments, as opposed to a case where different groups of people have been exposed to different treatments.

I can understand the reason behind this, and conceptually, it makes sense.

The question is, does this difference carry over to regression models as well?

To check this, I created the following fake data set, with made-up numbers:

enter image description here

Supposedly, 24 people were recruited for a wine-tasting experiment, and every person tasted 4 wines: 2 red and 2 white, and 1 red wine was produced in 1990, the other in 2000. The same structure is true for the white wines.

Judgment is the DV. Namely, people were asked to rate the quality of the wine on a scale of 0-100.

Now, for this made-up data set, I ran the following linear regression (in RStudio).

model1 <- lmer(Judgment ~ Wine + (1|ParticipantID), data=wine_within)
model1

Of course, this model may get more complicated if/when we add Year_of_harvest to that, but I am not dealing with it at the moment, because it is not necessary to illustrate my point.

As a next step, I changed the table a little bit and made Wine a between-subjects variable, as follows.

enter image description here

As you see, all the numbers are the same, but this time, we had to recruit 48 people for the experiment, and we assigned 24 of them to the red-wine group and 24 of them to the white-wine group (so nobody tasted the same kind of wine). But Year_of_harvest was still manipulated as a within-subjects variable, so one person tasted two different samples of white wine (one produced in 1990, another in 2000), and another person in the other condition tasted two samples of red wine (likewise, one produced in 1990, another in 2000, etc.).

Now, the model I built for this situation was very similar.

model2 <- lmer(Judgment ~ Wine + (1|ParticipantID), data=wine_between)
model2

I thought R would be smart enough to see that the number of participants was different and it wasn't the case that the same person tasted both kinds of wines - so it should treat Wine as a between-subjects variable here.

But when I ran the code, I realized that model2 gave the exact same output as model1.

What is the reason for this?

Did I do something wrong in the coding part? Should I have specified the between-subjects/within-subjects nature of the Wine variable at some point in the code?

I am really new to regression analyses, and I am mostly teaching myself how to use R based on online tutorials, etc., so I am sorry if my question sounds too naive.

I would really appreciate your help.

P.S.

Here is the first data set, in a reproducible format.

wine_within_cont <- tribble(
        ~ParticipantID, ~Wine, ~Year_of_harvest, ~Judgment,
        "1","Red","1990",79,
        "1","White","1990",80,
        "1","Red","2000",64,
        "1","White","2000",100,
        "2","Red","1990",46,
        "2","White","1990",34,
        "2","Red","2000",73,
        "2","White","2000",91,
        "3","Red","1990",45,
        "3","White","1990",12,
        "3","Red","2000",32,
        "3","White","2000",54,
        "4","Red","1990",90,
        "4","White","1990",96,
        "4","Red","2000",75,
        "4","White","2000",87,
        "5","Red","1990",83,
        "5","White","1990",19,
        "5","Red","2000",45,
        "5","White","2000",49,
        "6","Red","1990",95,
        "6","White","1990",91,
        "6","Red","2000",14,
        "6","White","2000",18,
        "7","Red","1990",73,
        "7","White","1990",26,
        "7","Red","2000",58,
        "7","White","2000",53,
        "8","Red","1990",80,
        "8","White","1990",70,
        "8","Red","2000",100,
        "8","White","2000",43,
        "9","Red","1990",10,
        "9","White","1990",48,
        "9","Red","2000",32,
        "9","White","2000",78,
        "10","Red","1990",43,
        "10","White","1990",96,
        "10","Red","2000",39,
        "10","White","2000",92,
        "11","Red","1990",74,
        "11","White","1990",56,
        "11","Red","2000",27,
        "11","White","2000",74,
        "12","Red","1990",79,
        "12","White","1990",70,
        "12","Red","2000",84,
        "12","White","2000",60,
        "13","Red","1990",61,
        "13","White","1990",53,
        "13","Red","2000",10,
        "13","White","2000",20,
        "14","Red","1990",29,
        "14","White","1990",75,
        "14","Red","2000",66,
        "14","White","2000",55,
        "15","Red","1990",98,
        "15","White","1990",37,
        "15","Red","2000",34,
        "15","White","2000",77,
        "16","Red","1990",65,
        "16","White","1990",90,
        "16","Red","2000",88,
        "16","White","2000",67,
        "17","Red","1990",65,
        "17","White","1990",54,
        "17","Red","2000",40,
        "17","White","2000",84,
        "18","Red","1990",85,
        "18","White","1990",68,
        "18","Red","2000",98,
        "18","White","2000",82,
        "19","Red","1990",82,
        "19","White","1990",13,
        "19","Red","2000",17,
        "19","White","2000",64,
        "20","Red","1990",69,
        "20","White","1990",100,
        "20","Red","2000",95,
        "20","White","2000",46,
        "21","Red","1990",60,
        "21","White","1990",50,
        "21","Red","2000",54,
        "21","White","2000",56,
        "22","Red","1990",58,
        "22","White","1990",56,
        "22","Red","2000",20,
        "22","White","2000",90,
        "23","Red","1990",78,
        "23","White","1990",67,
        "23","Red","2000",73,
        "23","White","2000",93,
        "24","Red","1990",37,
        "24","White","1990",60,
        "24","Red","2000",56,
        "24","White","2000",54)

And here is the second data set.

wine_between_cont <- tribble(
        ~ParticipantID, ~Wine, ~Year_of_harvest, ~Judgment,
        "1","Red","1990",79,
        "25","White","1990",80,
        "1","Red","2000",64,
        "25","White","2000",100,
        "2","Red","1990",46,
        "26","White","1990",34,
        "2","Red","2000",73,
        "26","White","2000",91,
        "3","Red","1990",45,
        "27","White","1990",12,
        "3","Red","2000",32,
        "27","White","2000",54,
        "4","Red","1990",90,
        "28","White","1990",96,
        "4","Red","2000",75,
        "28","White","2000",87,
        "5","Red","1990",83,
        "29","White","1990",19,
        "5","Red","2000",45,
        "29","White","2000",49,
        "6","Red","1990",95,
        "30","White","1990",91,
        "6","Red","2000",14,
        "30","White","2000",18,
        "7","Red","1990",73,
        "31","White","1990",26,
        "7","Red","2000",58,
        "31","White","2000",53,
        "8","Red","1990",80,
        "32","White","1990",70,
        "8","Red","2000",100,
        "32","White","2000",43,
        "9","Red","1990",10,
        "33","White","1990",48,
        "9","Red","2000",32,
        "33","White","2000",78,
        "10","Red","1990",43,
        "34","White","1990",96,
        "10","Red","2000",39,
        "34","White","2000",92,
        "11","Red","1990",74,
        "35","White","1990",56,
        "11","Red","2000",27,
        "35","White","2000",74,
        "12","Red","1990",79,
        "36","White","1990",70,
        "12","Red","2000",84,
        "36","White","2000",60,
        "13","Red","1990",61,
        "37","White","1990",53,
        "13","Red","2000",10,
        "37","White","2000",20,
        "14","Red","1990",29,
        "38","White","1990",75,
        "14","Red","2000",66,
        "38","White","2000",55,
        "15","Red","1990",98,
        "39","White","1990",37,
        "15","Red","2000",34,
        "39","White","2000",77,
        "16","Red","1990",65,
        "40","White","1990",90,
        "16","Red","2000",88,
        "40","White","2000",67,
        "17","Red","1990",65,
        "41","White","1990",54,
        "17","Red","2000",40,
        "41","White","2000",84,
        "18","Red","1990",85,
        "42","White","1990",68,
        "18","Red","2000",98,
        "42","White","2000",82,
        "19","Red","1990",82,
        "43","White","1990",13,
        "19","Red","2000",17,
        "43","White","2000",64,
        "20","Red","1990",69,
        "44","White","1990",100,
        "20","Red","2000",95,
        "44","White","2000",46,
        "21","Red","1990",60,
        "45","White","1990",50,
        "21","Red","2000",54,
        "45","White","2000",56,
        "22","Red","1990",58,
        "46","White","1990",56,
        "22","Red","2000",20,
        "46","White","2000",90,
        "23","Red","1990",78,
        "47","White","1990",67,
        "23","Red","2000",73,
        "47","White","2000",93,
        "24","Red","1990",37,
        "48","White","1990",60,
        "24","Red","2000",56,
        "48","White","2000",54)
$\endgroup$
  • 2
    $\begingroup$ Could you put you data in a format such that I can copy/paste? When I click on current data, it became a graph. $\endgroup$ – user158565 Aug 1 '19 at 1:28
  • $\begingroup$ Thanks so much! I will, in a moment. The current data is in a picture format, I think. $\endgroup$ – Freya Aug 1 '19 at 1:35
  • $\begingroup$ OK, I added some R code that you can copy and paste to generate the same tibbles. Thank you so much! $\endgroup$ – Freya Aug 1 '19 at 1:45
2
$\begingroup$

"But when I ran the code, I realized that model2 gave the exact same output as model1."

If you check the estimate of variance of error term and random intercept, two models give you different results. Also the variance of fixed effect are different, degree of freedom are not the same.

For the fixed effects (intercept and slope), fitting two models should should give the different estimates. If you increase the length of the display (for example, 20 digits after decimal), you may find the difference.

In your model, there are two variances: one for error term, and another for random intercept. The correlation coefficient between Y from the same participant is (Var(Intercept)/(Var(intercept)+Var(error)). If the correlation coefficient = 0, then the role of the random effect in the model = 0. Therefore your participant ID in data has no any information for model. It means the effect of participant ID depends on the correlation coefficient. In you data, estimate Var(error) = 538.64 and Var(Intercept) = 86.09. So when you change ID, the estimated fixed effect seems no change. It is reasonable.

If you want to verify the effect of changing ID on estimate of fixed effect, you can increase the correlation coefficient by increasing Var(intercept). You can realize it by generating 24 random number from N(0,400), adding one of them to four judgement from the same ID. And re-fit your two models; you will find the effect of manipulating the ID.

$\endgroup$
  • $\begingroup$ Thanks for your answer. So would you say that the analysis I ran for this particular problem is - at least conceptually - right? $\endgroup$ – Freya Aug 1 '19 at 2:46
  • 1
    $\begingroup$ I would say the first model is correct, the second one is wrong. Because ID is not informative, the model does not need a within-subjects variable. $\endgroup$ – user158565 Aug 1 '19 at 2:54
  • 1
    $\begingroup$ I do not think it is reasonable to add Wine as random effect. In fact when you specify random effect, you specify the covariance matrix indirectly. Obviously, 4 judgement from the same ID are correlated. Currently, you specified random ID- specific intercept, it is equal to say any pair of judgement from the same ID has the same correlation. It may not good. Maybe the correlation of pair judgement from the same year from the same ID should higher then pair judgement from different year from trhe same ID. Then you can add a random intercept for ID*year. $\endgroup$ – user158565 Aug 1 '19 at 3:05
  • 1
    $\begingroup$ In fact I am not clear about within- between- things. based on your data and model, I think you can add year and yearwine as fixed effect and yearid as random effect. Win can be random effect, if you think it is reasonable based on final variance-covariance matrix.For "How would I specify Wine as a between-subjects variable", you already has wine as fixed effect in model. What else you want to do? $\endgroup$ – user158565 Aug 1 '19 at 3:29
  • 1
    $\begingroup$ In your case, which one is the subject when you use between-subject, within-subject, wine or ID? $\endgroup$ – user158565 Aug 1 '19 at 3:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.