How do Regression Models Handle Categorical Variables?
At the end of the day, can categorical variables only be used to create "cohorts" within your data - and regression models are then fit to each one of these cohorts?
I will try to demonstrate this using the R programming language. Suppose we have the following data:
var_1 <- c("A","B") var_1 <- sample(var_1, 1000, replace=TRUE, prob=c(0.3, 0.7)) var_1 <- as.factor(var_1) var_2 <- c("AA","BB", "CC") var_2 <- sample(var_2, 1000, replace=TRUE, prob=c(0.2, 0.1, 0.7)) var_2 <- as.factor(var_2) var_3 <- c("AA1","BB1") var_3 <- sample(var_3, 1000, replace=TRUE, prob=c(0.5, 0.5)) var_3 <- as.factor(var_3) my_data = data.frame(var_1, var_2, var_3) my_data$var4 = rnorm(1000,10,10) my_data$var5 = rnorm(1000,10,10) my_data$response = rnorm(1000,10,10) head(my_data) var_1 var_2 var_3 var4 var5 response 1 B BB BB1 10.533117 18.705875 16.097650 2 A CC AA1 10.423024 -3.491847 18.980985 3 B CC AA1 19.044563 20.717728 19.469486 4 B CC AA1 17.851860 8.085333 4.083343 5 A CC BB1 9.390858 -1.696962 2.007595 6 B CC BB1 5.562221 4.186413 8.062490
It seems to me that the Maximum Likelihood Equation can not handle factor variables (I am also not sure if Probability Distributions "exist" for non-continuous variables). In this case, we can split the above data into individual cohorts combinations (in this example, there are 12 of these):
lst1 <- split(my_data, my_data[c("var_1", "var_2", "var_3")], drop = TRUE) list2env(lst1, envir=.GlobalEnv) summary(lst1) Length Class Mode A.AA.AA1 6 data.frame list B.AA.AA1 6 data.frame list A.BB.AA1 6 data.frame list B.BB.AA1 6 data.frame list A.CC.AA1 6 data.frame list B.CC.AA1 6 data.frame list A.AA.BB1 6 data.frame list B.AA.BB1 6 data.frame list A.BB.BB1 6 data.frame list B.BB.BB1 6 data.frame list A.CC.BB1 6 data.frame list B.CC.BB1 6 data.frame list
Now, we can fit Regression Models to each one of these cohorts:
model_1 <- lm(response ~ var4 + var5 , data = A.AA.AA1) model_2 <- lm(response ~ var4 + var5 , data = B.AA.AA1) # ... etc ... model_12 <- lm(response ~ var4 + var5 , data = B.CC.BB1)
My Question: Can someone please tell me if what I have described above is correct? In the real world, when the data has factor variables - are you supposed to identify "meaningful cohort combinations" (e.g. clustering, data exploration, specifically requested cohorts, etc.) and then fit individual models to each of these combinations? Are there any other standards methods of dealing with this problem?
Note: I am aware of methods such as "one hot encoding" - but I have been told that they are achieve similar results as to the method I described above, and there are advantages to creating individual models on cohort combinations :
As a result, you will only deal with continuous variables - MLE is better suited for continuous variables compared to "binary one hot encoded variables"
"One Hot Encoded" variables tend to be sparse
Individual datasets that are isolated by cohort combinations might have lower variance compared to the entire dataset. Thus, it might be easier to create models on lower variance (i.e. more homogeneous) datasets compared to higher variance datasets.