Is self-selection for a treatment a problem after all? I have devised the following R code.
In this simulated dataset, we know that a certain treatment (e.g. a degree, an education) increases income by $ 1,000. Income is also caused by age.
In the first part, units are randomly assigned into treatment. Each unit has a 50% probability to receive the treatment, regardless of his/her age.
In the second part, subjects self-select into treatment: younger units have a higher probability to receive treatment than older ones. We can check that the units that receive treatment are younger: we do a boxplot, and a t-test rejects the null hypothesis that the average age is the same across the treated and non treated groups.
By running an OLS, I am able to estimate the effect of treatment on income in both cases, so also in the second case where people self-select themselves into treatment.
In the second part, can I infer from the OLS that the treatment causes an increase in income by $ 1,000?
Is self-selection a problem for that inference?
N = 100000
age_coeff = 100
treated_coeff = 1000

# PART 1: RANDOM SELECTION INTO TREATMENT
df = data.frame(id=seq(from=1, to=N, by=1))
df$age = runif(N, min=0, max=100)
# size -> number of trials (zero or more)
df$treated = rbinom(N, size=1, prob=0.5)
df$treated = df$treated==1
e = rnorm(N, mean=0, sd=50)

# Check age is the same across the two groups
r <- aggregate(df$age, list(df$treated), FUN=mean) 
colnames(r) <- c("Treated", "Age")
print(r)
t.test(df$age[df$treated], df$age[!df$treated])

df$income = 0 + age_coeff*df$age + treated_coeff*df$treated + e
income.lm = lm(income~age+treated, data=df)
summary(income.lm)

# PART 2: SELF-SELECTION INTO TREATMENT
library(sigmoid)
df$ptreated = logistic(50-df$age)
df$treated = rbinom(N, size=1, prob=df$ptreated)
df$treated = df$treated==1

# Check if selection into treatment is effective
boxplot(df$ptreated~df$treated)
r <- aggregate(df$ptreated, list(df$treated), FUN=mean) 
colnames(r) <- c("Treated", "p(Treated)")
print(r)

# Check if age differs in the two groups
r <- aggregate(df$age, list(df$treated), FUN=mean) 
colnames(r) <- c("Treated", "Age")
print(r)
boxplot(df$age~df$treated)
t.test(df$age[df$treated], df$age[!df$treated])

df$income = 0 + age_coeff*df$age + treated_coeff*df$treated + e
income.lm = lm(income~age+treated, data=df)
summary(income.lm)

# Check
df$age50 <- df$age >= 50
table(df$age50, df$treated, dnn=c("Age50", "treated"))
```

 A: *

*A t-test doesn't construct a certainty interval around the actual treatment difference, even the CI can't be interpreted that way. No NHST method will tell you the actual treated difference is 1000. However, both models should be powered to reject the null hypothesis that income does not differ by treatment assignment.


*In the self-selection scenario, age is a confounder. But in your inference you have adjusted for age to eliminate confounding. Therefore, the linear model is expected to adjust the counfounding effects. The model is not a t-test but an adjusted linear model. Had you used the t-test the result would be biased.


*In practice the issue doesn't boil down to known confounders with known functional forms, but unknown confounders with unknown functional forms. You are often presented with data that failed to collect certain values, or at the appropriate times, or with desired precision, subject to recall bias, reverse confounding, non-ignorable missingness. The methods for handling these cases when the precise mechanism is known is almost irrelevant, but understanding how to assess a data analysis and put together results from multiple studies and ascertain the impact of confounding is a subtler and more valuable art.
