How do Regression Models Handle Categorical Variables?

Question

How do Regression Models Handle Categorical Variables?

At the end of the day, can categorical variables only be used to create "cohorts" within your data - and regression models are then fit to each one of these cohorts?

I will try to demonstrate this using the R programming language. Suppose we have the following data:

    var_1 <- c("A","B")
    
    var_1 <- sample(var_1, 1000, replace=TRUE, prob=c(0.3, 0.7))
    
    var_1 <- as.factor(var_1)
    
    
    var_2 <- c("AA","BB", "CC")
    
    var_2 <- sample(var_2, 1000, replace=TRUE, prob=c(0.2, 0.1, 0.7))
    
    var_2 <- as.factor(var_2)
    
    
    var_3 <- c("AA1","BB1")
    
    var_3 <- sample(var_3, 1000, replace=TRUE, prob=c(0.5, 0.5))
    
    var_3 <- as.factor(var_3)
    
    my_data = data.frame(var_1, var_2, var_3)
    
    my_data$var4 = rnorm(1000,10,10)
    
    my_data$var5 = rnorm(1000,10,10)
    
    my_data$response = rnorm(1000,10,10)
    
    head(my_data)
      var_1 var_2 var_3      var4      var5  response
    1     B    BB   BB1 10.533117 18.705875 16.097650
    2     A    CC   AA1 10.423024 -3.491847 18.980985
    3     B    CC   AA1 19.044563 20.717728 19.469486
    4     B    CC   AA1 17.851860  8.085333  4.083343
    5     A    CC   BB1  9.390858 -1.696962  2.007595
    6     B    CC   BB1  5.562221  4.186413  8.062490

It seems to me that the Maximum Likelihood Equation can not handle factor variables (I am also not sure if Probability Distributions "exist" for non-continuous variables). In this case, we can split the above data into individual cohorts combinations (in this example, there are 12 of these):

    lst1 <- split(my_data, my_data[c("var_1", "var_2", "var_3")], 
                  drop = TRUE)
    
    list2env(lst1, envir=.GlobalEnv)

    summary(lst1)
             Length Class      Mode
    A.AA.AA1 6      data.frame list
    B.AA.AA1 6      data.frame list
    A.BB.AA1 6      data.frame list
    B.BB.AA1 6      data.frame list
    A.CC.AA1 6      data.frame list
    B.CC.AA1 6      data.frame list
    A.AA.BB1 6      data.frame list
    B.AA.BB1 6      data.frame list
    A.BB.BB1 6      data.frame list
    B.BB.BB1 6      data.frame list
    A.CC.BB1 6      data.frame list
    B.CC.BB1 6      data.frame list

Now, we can fit Regression Models to each one of these cohorts:

    model_1 <- lm(response ~ var4 + var5 , data = A.AA.AA1)
    model_2 <- lm(response ~ var4 + var5 , data = B.AA.AA1)
    
    # ... etc ...
    
    model_12 <- lm(response ~ var4 + var5 , data = B.CC.BB1)

My Question: Can someone please tell me if what I have described above is correct? In the real world, when the data has factor variables - are you supposed to identify "meaningful cohort combinations" (e.g. clustering, data exploration, specifically requested cohorts, etc.) and then fit individual models to each of these combinations? Are there any other standards methods of dealing with this problem?

Note: I am aware of methods such as "one hot encoding" - but I have been told that they are achieve similar results as to the method I described above, and there are advantages to creating individual models on cohort combinations :

• As a result, you will only deal with continuous variables - MLE is better suited for continuous variables compared to "binary one hot encoded variables"

• "One Hot Encoded" variables tend to be sparse

• Individual datasets that are isolated by cohort combinations might have lower variance compared to the entire dataset. Thus, it might be easier to create models on lower variance (i.e. more homogeneous) datasets compared to higher variance datasets.

Answer 1

Factor variables denote membership to a group; hence, they are equivalent to a set of indicator variables, which are numeric. For instance if a factor variable $x$ takes values $a,b,$ or $c$, then we can assess the membership of $x$ by checking if $x=a$ or $x=b$, if $x$ does not equal either, then clearly $x=c$. This is what you call "one hot encoding".

The linear models you have formed are known as stratification. This implies that a separate variance parameter is fit for each "cohort". Stratification with a continuous covariate also implies a separate slope parameter is fit for each "cohort". Thus we could fit a single weighted regression model where the variance-covariance matrix is a diagonal matrix with separate variance parameters for each "cohort" and the mean function is the sum of the factor variable "cohort" and interaction terms between "cohort" levels and the continuous covariates.

There are no advantages of stratification, perhaps it seems easier to the uninitiated. One only uses this stratified model if one has reason to believe that the variances differ by "cohort" and slope parameters for continuous covariates differ by "cohort."

Answer 2

Statistical models require a significant amount of study, and fortunately vast resources are available, including dozens of excellent texts. R code is not needed here, and if you do want to use R code to demonstrate a point you could have used 1/5th as many lines of code (see for example the expand.grid function).

Predictor variables in regression models are conditioned upon, so they need not have distributions. Better than R code, write out the models, e.g., $$E(Y | X_{1}, X_{2}) = \beta_{0} + \beta_{1}X_{1} + \beta_{2}[X_{2}=b] + \beta_{3}[X_{2}=c]$$ where $X_1$ is continuous or binary and $X_{2}$ is categorical with levels $a,b,c$. $[z]$ indicates 1 if $z$ holds and 0 otherwise.

Then the issue is whether or not you want to assume constant variance across categories of $X_2$ as has already been discussed.