In my work I've stumbled upon an interesting result when a classifier is applied to within-subjects data. My question is whether this is a known result, and if so, does it have a name? I can't find anything quite like it my searches.
Say we have some within-subjects data with multiple variables. Participants completed both phases of a manipulation (two conditions, A/B, test/control, etc), and we have measured at least two variables during both phases of the manipulation. So we have their scores for, say, both Test 1 and Test 2, during both the test and control conditions. Now, we have some dependent variable of interest. Our question is, is there relationship between how participant's test responses changed during the manipulation and the dependent variable? This is frequently operationalized as a difference score - e.g., cor(test1_visit1 - test1_visit2,outcome).
Now, the interesting result is as follows: instead of calculating these difference scores, let's instead train a classifier to distinguish the two conditions of the manipulation, using the responses to both Test 1 and Test 2. We will of course create test and train splits. We now have a probability for each the two conditions, for each participant. We then create a difference score using the probabilities for each of the two conditions. It turns out, that this difference between the condition probabilities will also be correlated with our dependent variable, almost always more strongly than either of the individual difference scores. Yes, there is some 'ML' happening, but it is completely without reference to our outcome variable of interest. In some ways it feels like a weighted factor analysis, but in a way I've never encountered before. Is this a known analytic approach that I've just never heard of before?
Some example code from a simulation I've been working on. All variables are multivariate normal, and the classifier is just a logistic regression. I've systematically varied all the parameters across several million simulations, and the result holds 99.8% of the time, as long as the classifier performs reasonably well (auc > 0.6).
library(dplyr)
library(faux)
library(tidyr)
##################
sim_dat = function(train_n,
test_n,
var.num,
main.effect.condition ,
within.obs.correl,
total.var ,
total_n ){
##################################
#### generate within-subjects correlation matrix
var.num2 = var.num*2
means = rep(main.effect.condition,var.num)
mean.vec = matrix(c(rep(0,var.num),means),nrow = 2,byrow = T) %>%
c()
cor.vec = rep(0, (var.num2^2-var.num2)/2)
index = 1
nextindex = var.num2-1
while(nextindex > 0){
index = c(index,max(index)+nextindex)
nextindex = nextindex - 1
}
index = index[(seq_len(length(index)) %% 2 ) == 1]
cor.vec[index] = rep(within.obs.correl,var.num)
#### simulate data
dat <- faux::rnorm_multi(
n = total_n,
vars = var.num2,
r = cor.vec,
mu = mean.vec,
sd = rep(1,var.num2),
varnames = letters[c(1:var.num2)]
)
### make difference scores
make_diff = function(dat,var.num2){
odd_col = seq_len(var.num2) %% 2
odd_col = c(1:var.num2)[odd_col == 1]
diff.mat = matrix(data = NA,nrow =total_n,var.num2/2 ) %>%
as.data.frame()
for(i in 1:length(odd_col)){
diff.mat[,i] = dat[,odd_col[i]+1] - dat[,odd_col[i]]
}
return(diff.mat)
}
diff.mat = make_diff(dat,var.num2)
#### simulate dependent variable
yeffect = sqrt(total.var/var.num)
betas = rep(yeffect,var.num)
M <- t(as.matrix(betas)) * diff.mat[1:length(betas)]
var.epsilon <- sum(cov(M))*((1 - total.var)/total.var)
epsilon <- rnorm(total_n, sd=sqrt(var.epsilon))
diff.mat$y <- apply(cbind(M,epsilon),1,sum)
summary(lm(y ~., data = diff.mat))
diff.mat$ID = c(1:total_n)
#### transform data with both conditions to long-format
make_long = function(tempdat, var.num){
var.num2 = var.num * 2
odd_col = seq_len(var.num2) %% 2
odd_col = c(1:var.num2)[odd_col == 1]
con0 = tempdat[,odd_col] %>% as.data.frame()
con1 = tempdat[,-odd_col] %>% as.data.frame()
colnames(con0) = paste("Var",c(1:var.num),sep="")
colnames(con1) = paste("Var",c(1:var.num),sep="")
con0$con = 0
con1$con = 1
con0$ID = c(1:total_n)
con1$ID = c(1:total_n)
tempdat_out = rbind(con0,con1)
return(tempdat_out)
}
dat2 = make_long(dat, var.num)
#############################
#### simple classifier model
IDs = unique(dat2$ID)
g1 = sample(IDs,size = train_n, replace = FALSE)
test_pool = IDs[!IDs %in% g1]
g2 = sample(test_pool,size = test_n, replace = FALSE)
train = dat2[dat2$ID %in% g1 ,]
test = dat2[dat2$ID %in% g2 ,]
standardize = function(tempdat){
tempdat[,grep("Var",colnames(tempdat))] =
tempdat[, grep("Var", colnames(tempdat))] %>%
apply(2,function(X){
X = scale(X, center = TRUE, scale=TRUE)
attributes(X) = NULL
return(X)
})
return(tempdat)
}
train = standardize(train)
test = standardize(test)
log_mod = glm(con ~., data = train[,-grep("ID", colnames(train))])
test$pred = predict(log_mod, test, type="response")
return_diff_sig = function(test_temp){
test0 = test_temp %>% dplyr::filter(con == 0) %>%
dplyr::select(c("ID","pred"))
test1 = test_temp %>% dplyr::filter(con == 1) %>%
dplyr::select(c("ID","pred"))
colnames(test0)[2] = "pred0"
colnames(test1)[2] = "pred1"
testdiff = merge(test1,test0)
testdiff$diff = testdiff$pred1 - testdiff$pred0
return(testdiff)
}
testdiff = return_diff_sig(test)
testdiff = merge(testdiff,diff.mat)
## return data
out = list()
out$orig = test
out$differencescores = testdiff
return(out)
}
Generate the simulated data:
dat = sim_dat(train_n = 1000, #training sample size
test_n = 1000, # test sample size
total_n = 10000, # total population size
var.num = 2, # number of independent tests measured
main.effect.condition = 1, # mean difference in test scores between
# conditions (Cohen's D), same for all tests
within.obs.correl = .8,
# correlation between test scores for a given test, across the two
# conditions, same for all tests
total.var = .2 # total variance of the outcome variable explained by
# the test difference scores
)
Both tests show an effect of the manipulation in the simulation
a1 = dat$orig %>% tidyr::pivot_longer(cols =
dplyr::starts_with("Var"), names_to = "Variable")
ggplot(a1,aes(x = Variable, y = value, fill = as.factor(con))) +
scale_x_discrete(labels = c("Test 1", "Test 2")) +
scale_fill_discrete(labels = c("Control","Manipulation")) +
geom_boxplot() +
theme_classic() +
labs(fill = "Condition") +
ylab(label = "Test score") + xlab(label = "")
Both difference scores are correlated with the outcome variable of interest (by design).
dat$differencescores[,-c(1:4)] %>% cor()
V1 V2 y
V1 1.00000000 0.02977112 0.2948523
V2 0.02977112 1.00000000 0.3871282
y 0.29485230 0.38712816 1.0000000
Interestingly, if we compute a difference score from the predicted condition probabilities, it is also correlated with our outcome variable, even more strongly than either of the individual difference scores.
cor(dat$differencescores$y,dat$differencescores$diff)
[1] 0.4749716
