Overview of the question
Warning: This question requires a lot of set-up. Please bear with me.
A colleague of mine and I are working on an experiment design. The design must work around a large number of constraints, which I will list below. I have developed a design that satisfies the constraints and that gives us unbiased estimates of our effects of interest. However, my colleague believes that there is a confound in the design. We have argued this point ad nauseum without coming to a resolution, so at this point I would like some outside opinions.
I will describe the goal of the study, our constraints, the potential confound, and why I believe this "confound" is not a problem below. As you read each section, bear in mind my overall question:
Is there a confound in the design that I describe?
[The details of this experiment have been modified, but the essential elements required to ask my question remain the same]
Experiment goals
We wish to determine whether essays written by White males are evaluated more favorably than essays written by White females, Black males, or Black females (the essay authorship variable). We also wish to determine whether any bias we find shows up more in high or low quality grants (the quality variable). Finally, we wish to include essays written about 12 different topics (the topic variable). However, only the first two variables are of substantive interest; although topic must vary across essays, we are not substantively interested in how evaluations vary across topics.
Constraints
- There are limits to both the number of participants and the number of essays that we can collect. The result is that authorship cannot be manipulated entirely between participants, nor can it be manipulated entirely between essays (i.e., each individual essay must be assigned to multiple conditions).
- Although each essay can have White male, White female, Black male, and Black female versions, each essay can only be one of high and low quality and can only be about one topic. Or, to put this constraint in a different way, neither quality nor topic can be manipulated within essays, since they are inherent characteristics of a given essay.
- Due to fatigue, there is a limit to the number of essays a given participant can evaluate.
- All of the essays that a given person reads must be about a single topic. In other words, essays cannot be assigned entirely at random to participants, since we need to ensure that each participant only reads essays of a similar topic.
- Each participant can only view one essay supposedly authored by a non White male author, since we don't want participants to get suspicious about the purpose of the experiment because too many of their essays are written by Black or female authors.
The proposed design
My proposed design first manipulates each essay into the 4 different authorship versions (White male, White female, etc). Four essays from a similar topic are then used to define a "set", each of which consists of two high and two low quality essays. Each participant receives three essays from a given set as follows in the figure given below. Each participant then provides a single rating to each of the three essays that he or she is assigned.
The potential confound
My colleague believes that the above design contains a confound. The problem, he says, is that, when a high quality essay is assigned to be authored by a non White male writer, it is always paired with one high quality essay and one low quality essay (for Essay 1, see Participants 1-3 in the figure). On the other hand, when that same essay is assigned to be authored by White male writers, it is paired with one high quality essay and one low quality essay three times (for Essay 1, Participants 4-6) and two low quality essays three times (for Essay 1, Participants 7-9).
A similar problem exists for low quality essays. When a low quality essay has a non White male author, it is always seen with a low quality essay and a high quality essay (for Essay 3, see Participants 7-9). However, when that same essay has a White male author, it is seen with one high quality essay and one low quality essay three times (for Essay 3, Participants 10-12) and with two high quality essays three times (for Essay 3, Participants 1-3).
The reason the above patterns could be problematic is if we assume the existence of "contrast effects". Specifically, if high quality essays are evaluated more favorably on average when they are paired with two low quality essays than when they are paired with one low quality essay and one high quality essay (a reasonable assumption), White male essays may receive higher ratings than White female, Black male, and Black female essays for a reason other than authorship.
A contrast effect for high quality essays may or may not be balanced by a contrast effect for low quality essays; that is, it may or may not be the case that low quality essays paired with two high quality essays are evaluated especially unfavorably. Regardless, my colleague claims, the potential for contrast effects of any kind renders this design problematic for the purposes of determining whether the essays authored by White males are evaluated more favorably than the essays of other authors.
Why I believe the potential confound is not a problem
What matters for me is whether we are able to estimate the degree to which White male essays are evaluated differently than other essays (i.e., whether we can estimate our effects of interest), even in the presence of contrast effects. I therefore conducted a simulation where I simulated 50 datasets that contained contrast effects and fit a model that tests for our effects of interest.
The specific model is a mixed effects model with random intercepts for essay (each essay is evaluated by multiple participants) and participant (each participant evaluates multiple essays). The essay level also contains random slopes for race, gender, and their interaction (both variables are manipulated within essay) and the participant level contains a random slope for quality (quality is manipulated within participants). The effects of interest are the effects of race, gender, the interaction between race and gender, and the higher-order interactions between each of these variables and quality. The goal of this simulation was to determine whether introducing contrast effects into the data would create spurious effects of race, gender, the interaction between race and gender, and the higher-order interactions between these variables and quality. See the code chunk below for more details.
According to the simulation, the presence of contrast effects does not bias the estimates of any of our effects of interest. In addition, the size of the contrast effect can be estimated in the same statistical model as the other effects in the design; to me, this already suggests that the "contrast effects" identified by my colleague are not a confound. My colleague, however, remains skeptical.
require(lme4)
require(plyr)
participant <- rep(1:12, 3)
essay <- c(rep(1, 9), rep(2, 9), rep(3, 9), rep(4, 9))
quality <- ifelse(essay == 1 | essay == 2, "high", "low")
race <- c("white", "black", "black", "white", "white", "white", "white", "white", "white",
"white", "white", "white", "white", "white", "white", "white", "black", "black",
"white", "black", "black", "white", "white", "white", "white", "white", "white",
"white", "white", "white", "white", "white", "white", "white", "black", "black")
gender <- c("female", "male", "female", "male", "male", "male", "male", "male", "male",
"male", "male", "male", "male", "male", "male", "female", "male", "female",
"female", "male", "female", "male", "male", "male", "male", "male", "male",
"male", "male", "male", "male", "male", "male", "female", "male", "female")
d <- data.frame(participant, essay, quality, race, gender)
for(i in 1:35)
{
participant <- participant + 12
essay <- essay + 4
newdat <- data.frame(participant, essay, quality, race, gender)
d <- rbind(d, newdat)
}
check_var <- function(var)
{
tab <- table(var)
newvar <- character()
for(i in var)
{
if(i == names(tab[tab == 1]))
{
newvar <- c(newvar, "different")
} else
{
newvar <- c(newvar, "same")
}
}
return(newvar)
}
# Mark, for a given participant, which essay is "different"
d <- ddply(d, "participant", mutate, different = check_var(quality))
# Make each variable numeric for the purposes of the simulation
d$quality <- ifelse(d$quality == "low", -.5, .5)
d$race <- ifelse(d$race == "black", -.5, .5)
d$gender <- ifelse(d$gender == "female", -.5, .5)
d$different <- ifelse(d$different == "same", -.5, .5)
# Random seed
set.seed(2352)
# Number of simulations
reps <- 50
# Create a storage space for the effects
effs <- matrix(NA, ncol = 10, nrow = reps)
# For each simulation
for(i in 1:reps)
{
# Fixed effects. A quality effect and a contrast effect for quality
d$score <- .5 * d$quality + 1 * d$different * d$quality
# Random effects at the participant level
d <- ddply(d, "participant", mutate, r_int = rnorm(1, sd = .5),
r_q = rnorm(1, sd = .5),
score = score + r_int + r_q * quality)
# Random effects at the essay level
d <- ddply(d, "essay", mutate, g_int = rnorm(1, sd = .5),
g_r = rnorm(1, sd = .5),
g_g = rnorm(1, sd = .5),
g_r_g = rnorm(1, sd = .5),
score = score + g_int + g_r * race + g_g * gender + g_r_g * race * gender)
# Observation-level error
d$score <- d$score + rnorm(dim(d)[1], sd = 1)
# Fit the model
mod <- lmer(score ~ race * gender * quality + different * quality + (race * gender | essay) + (quality | participant), data = d)
# Store the coefficients
colnames(effs) <- names(fixef(mod))
effs[i, ] <- fixef(mod)
# Print the current simulation
print(i)
}
# Results
round(colMeans(effs), digits = 2)
(Intercept) race gender quality
0.00 -0.03 0.02 0.50
different race:gender race:quality gender:quality
0.01 -0.03 0.00 0.03
quality:different race:gender:quality
0.97 -0.02
Once again, my overall question is, is there a confound in the design I have described? If a confound is not present, I would be interested in a description of why the potential "contrast effects" are not confounds so that I can explain this to my colleague.
