Deal with categorical variables in regression with overlapping classes

Question

I want to perform a OLS regression analysis on some venue hire data (to get cost of venue hire). I have many categorical variables and some of them have correlated / duplicated information e.g. An example is one category called 'External_Hire' that defines whether you can hire external caterers for the venue. It has the following values:

• 'From Approved List Only'
• 'Any External Hire'
• 'Not allowed'

Then I have another variable called 'External Hire Fee' which defines whether a fee is required for the 'Any External Hire' option, it has values:

• 'Fee Required'
• 'No Fee Required'
• 'No External Hire Allowed'

Whenever the 'External Hire' category has value 'Any External Hire' then the variable 'External Hire Fee' must have values 'Fee Required' or 'No Fee Required'. Otherwise, it will have 'No External Hire Allowed'.

I could therefore combine these 2 categorical variables into a combined category with values:

• 'From Approved List Only'
• 'Not Allowed'
• 'Any External Hire (Hire Fee Required)'
• 'Any External Hire (No Hire Fee Required)'

My question is, should I keep these variables as separate categorical variables, and then dummy encode them separately or should I combine them into a single variable as above and then dummy encode that? Does that even make a difference?

Is it wasteful computationally to do it one way or another? Are there further considerations for different models e.g. Decision trees?

Should I review the relationship between these categories and the response variable?

Answer 1

Combining the External Hire and External Hire Fee columns recodes the data without loss (or gain) of information. You get the same regression model with both codings. The simplified coding is, as @dariober points out, more succinct and convenient.

# Have generated fake data for illustration...

model_with_original_coding <- lm(
  Y ~ `External Hire` + `External Hire Fee`,
  data = fake_venue_data
)
model_with_combined_coding <- lm(
  Y ~ `External Hire Combined`,
  data = fake_venue_data
)

The two specifications result in an equivalent fitted model, with the same summary statistics.

glance(
  model_with_original_coding
)
glance(
  model_with_combined_coding
) 
#>   coding   r.squared adj.r.squared sigma statistic p.value    df logLik   AIC   BIC deviance df.residual  nobs
#> 1 original    0.0236      -0.00693  1.00     0.773   0.512     3  -140.  290.  304.     96.7          96   100
#> 2 combined    0.0236      -0.00693  1.00     0.773   0.512     3  -140.  290.  304.     96.7          96   100

With the original coding, there is a redundancy (an exact linear dependence) in the predictors. What does this mean?

A regression model is linear in the input variables $X_1,\ldots,X_k$. So adding a "new" predictor $X_{k+1}$ which is a linear combination of the already included $X_i$s doesn't change the model; it overparametrizes the model. This manifests as a line of NA's in the estimated coefficients table.

tidy(
  model_with_original_coding
)
#>   term                                        estimate std.error statistic p.value
#> 1 (Intercept)                                   -0.187     0.205    -0.911   0.365
#> 2 `External Hire`From Approved List Only         0.142     0.277     0.511   0.610
#> 3 `External Hire`Not allowed                     0.332     0.273     1.21    0.227
#> 4 `External Hire Fee`No External Hire Allowed   NA        NA        NA      NA    
#> 5 `External Hire Fee`No Fee Required             0.417     0.324     1.29    0.201

tidy(
  model_with_combined_coding
)
#>   term                                            estimate std.error statistic p.value
#> 1 (Intercept)                                       -0.187     0.205    -0.911   0.365
#> 2 `External Hire Combined`From Approved List Only    0.142     0.277     0.511   0.610
#> 3 `External Hire Combined`No Fee Required            0.417     0.324     1.29    0.201
#> 4 `External Hire Combined`Not allowed                0.332     0.273     1.21    0.227

These CV discussions may be of interest:

How can adding a 2nd IV make the 1st IV significant?
What is the significance of a linear dependency in a polynomial regression?
Testing for linear dependence among the columns of a matrix

---

R code to generate fake venue hire data.

library("broom")
library("tidyverse")

external_hire <- c(
  "From Approved List Only",
  "Any External Hire",
  "Not allowed"
)
external_hire_fee <- c(
  "Fee Required",
  "No Fee Required"
)

set.seed(1234)
n <- 100

fake_venue_data <-
  tibble(
    `External Hire` = sample(external_hire, n, replace = TRUE),
    `External Hire Fee` = sample(external_hire_fee, n, replace = TRUE)
  ) %>%
  mutate(
    `External Hire Fee` = if_else(
      `External Hire` == "Any External Hire",
      `External Hire Fee`,
      "No External Hire Allowed"
    ),
    Y = rnorm(n)
  ) %>%
  unite(
    `External Hire Combined`,
    c(`External Hire`, `External Hire Fee`),
    remove = FALSE
  )
fake_venue_data %>%
  count(
    `External Hire`,
    `External Hire Fee`,
    `External Hire Combined`
  )

Answer 2

My gut feeling, which is often wrong, is that combining the two variables into a single, non-redundant one is preferable.

In the extreme case, you could have two variables that are completely overlapping. I.e. one is just a renaming of the other. If you keep them separate you are going to have issues with collinearity but if you combine them you just have more verbose names.

Also, if you keep variables separate you allow the model to consider parts of the parameter space that you know beforehand are not allowed. This means you spread the available information more thinly than necessary (apologies for sloppy phrasing).

On the other hand, by combining variables you lose information about their individual contribution. You may or may not be concerned about this.