# Multiple regression with “overlapping” categorical dummy variables

Using R, I want to run a linear regression to estimate the abnormal return on days with positive, negative and neutral news (CLASS). I'm a beginner in R, as well as in using regression models! First of all the data structure is as follows. CONTROLVAR just represents all the columns I use as control variables.

DATE <- c("1","2","3","4","5","6","7","1","2","3","4","5","6","7")
COMP <- c("A", "A", "A", "A", "A", "A", "A", "B", "B", "B", "B", "B", "B", "B")
RET <- c(-2.0,1.1,3,1.4,-0.2, 0.6, 0.1, -0.21, -1.2, 0.9, 0.3, -0.1,0.3,-0.12)
CLASS <- c("positive", "negative", "aneutral", "positive", "positive", "negative", "aneutral", "positive", "negative", "negative", "positive", "aneutral", "aneutral", "aneutral")
SUBJECT.1 <- c("LITIGATION","LAYOFF","POLLUTION","CHEMICAL DISASTER","PRESS RELEASE","PEOPLE","EMISSIONS","ENERGY","WASTE MANAGEMENT","EMPLOYEES","SUBJECT11","SUBJECT12","SUBJECT13","SUBJECT14")
SUBJECT.2 <- c("POLLUTION","EMPLOYEES","NUCLEAR","FUELS","STOCK OPTION PLAN","EXECUTIVES","CO2","SOLAR","POLLUTION","EXECUTIVES","SUBJECT21","SUBJECT22","SUBJECT23","SUBJECT24")
SUBJECT.3 <- c("ENVIRONMENT","JOB REDUCTIONS","POWER PLANTS","POLLUTION","EMPLOYEES","FRAUD","CLIMATE CHANGE","SUSTAINABILITY","HAZARDOUS WASTE","BONUS PAY","SUBJECT31","SUBJECT32","SUBJECT33","SUBJECT34")
CONTROLVAR <- c("11","13","13","14","13","14","12","11","13","13","14","13","14","12")

df <- data.frame(DATE, COMP, RET, CLASS, SUBJECT.1, SUBJECT.2, SUBJECT.3, CONTROLVAR, stringsAsFactors=F)

df

#    DATE COMP   RET    CLASS         SUBJECT.1         SUBJECT.2       SUBJECT.3 CONTROLVAR
# 1     1    A -2.00 positive        LITIGATION         POLLUTION     ENVIRONMENT         11
# 2     2    A  1.10 negative            LAYOFF         EMPLOYEES  JOB REDUCTIONS         13
# 3     3    A  3.00 aneutral         POLLUTION           NUCLEAR    POWER PLANTS         13
# 4     4    A  1.40 positive CHEMICAL DISASTER             FUELS       POLLUTION         14
# 5     5    A -0.20 positive     PRESS RELEASE STOCK OPTION PLAN       EMPLOYEES         13
# 6     6    A  0.60 negative            PEOPLE        EXECUTIVES           FRAUD         14
# 7     7    A  0.10 aneutral         EMISSIONS               CO2  CLIMATE CHANGE         12
# 8     1    B -0.21 positive            ENERGY             SOLAR  SUSTAINABILITY         11
# 9     2    B -1.20 negative  WASTE MANAGEMENT         POLLUTION HAZARDOUS WASTE         13
# 10    3    B  0.90 negative         EMPLOYEES        EXECUTIVES       BONUS PAY         13
# 11    4    B  0.30 positive         SUBJECT11         SUBJECT21       SUBJECT31         14
# 12    5    B -0.10 aneutral         SUBJECT12         SUBJECT22       SUBJECT32         13
# 13    6    B  0.30 aneutral         SUBJECT13         SUBJECT23       SUBJECT33         14
# 14    7    B -0.12 aneutral         SUBJECT14         SUBJECT24       SUBJECT34         12


This regression model would look like this:

mymodel1 <- lm(RET ~ CLASS + CONTROLVAR, data=df)


aneutral (neutral) will be the reference category. I would also like to see the effect of certain subjects of the article on the abnormal return. How can I do that? Let's say I want to include the subjects LITIGATION, POLLUTION and LAYOFF. I'd like to see how the effect of positive, negative and neutral news change, if the article is about POLLUTION for example. If I make "dummy columns" for the three subjects of interest, my data.frame looks like this:

DATE <- c("1","2","3","4","5","6","7","1","2","3","4","5","6","7")
COMP <- c("A", "A", "A", "A", "A", "A", "A", "B", "B", "B", "B", "B", "B", "B")
RET <- c(-2.0,1.1,3,1.4,-0.2, 0.6, 0.1, -0.21, -1.2, 0.9, 0.3, -0.1,0.3,-0.12)
CLASS <- c("positive", "negative", "aneutral", "positive", "positive", "negative", "aneutral", "positive", "negative", "negative", "positive", "aneutral", "aneutral", "aneutral")
SUBJECT.1 <- c("LITIGATION","LAYOFF","POLLUTION","CHEMICAL DISASTER","PRESS RELEASE","PEOPLE","EMISSIONS","ENERGY","WASTE MANAGEMENT","EMPLOYEES","SUBJECT11","SUBJECT12","SUBJECT13","SUBJECT14")
SUBJECT.2 <- c("POLLUTION","EMPLOYEES","NUCLEAR","FUELS","STOCK OPTION PLAN","EXECUTIVES","CO2","SOLAR","POLLUTION","EXECUTIVES","SUBJECT21","SUBJECT22","SUBJECT23","SUBJECT24")
SUBJECT.3 <- c("ENVIRONMENT","JOB REDUCTIONS","POWER PLANTS","POLLUTION","EMPLOYEES","FRAUD","CLIMATE CHANGE","SUSTAINABILITY","HAZARDOUS WASTE","BONUS PAY","SUBJECT31","SUBJECT32","SUBJECT33","SUBJECT34")
LITIGATION <- c(1,0,0,0,0,0,0,0,0,0,0,0,0,0)
POLLUTION  <- c(1,0,1,1,0,0,0,0,1,0,0,0,0,0)
LAYOFF     <- c(0,1,0,0,0,0,0,0,0,0,0,0,0,0)
CONTROLVAR <- c("11","13","13","14","13","14","12","11","13","13","14","13","14","12")

df2 <- data.frame(DATE, COMP, RET, CLASS, SUBJECT.1, SUBJECT.2, SUBJECT.3, LITIGATION, POLLUTION, LAYOFF, CONTROLVAR, stringsAsFactors=F)

df2

#    DATE COMP   RET    CLASS         SUBJECT.1         SUBJECT.2       SUBJECT.3 LITIGATION POLLUTION LAYOFF CONTROLVAR
# 1     1    A -2.00 positive        LITIGATION         POLLUTION     ENVIRONMENT          1         1      0         11
# 2     2    A  1.10 negative            LAYOFF         EMPLOYEES  JOB REDUCTIONS          0         0      1         13
# 3     3    A  3.00 aneutral         POLLUTION           NUCLEAR    POWER PLANTS          0         1      0         13
# 4     4    A  1.40 positive CHEMICAL DISASTER             FUELS       POLLUTION          0         1      0         14
# 5     5    A -0.20 positive     PRESS RELEASE STOCK OPTION PLAN       EMPLOYEES          0         0      0         13
# 6     6    A  0.60 negative            PEOPLE        EXECUTIVES           FRAUD          0         0      0         14
# 7     7    A  0.10 aneutral         EMISSIONS               CO2  CLIMATE CHANGE          0         0      0         12
# 8     1    B -0.21 positive            ENERGY             SOLAR  SUSTAINABILITY          0         0      0         11
# 9     2    B -1.20 negative  WASTE MANAGEMENT         POLLUTION HAZARDOUS WASTE          0         1      0         13
# 10    3    B  0.90 negative         EMPLOYEES        EXECUTIVES       BONUS PAY          0         0      0         13
# 11    4    B  0.30 positive         SUBJECT11         SUBJECT21       SUBJECT31          0         0      0         14
# 12    5    B -0.10 aneutral         SUBJECT12         SUBJECT22       SUBJECT32          0         0      0         13
# 13    6    B  0.30 aneutral         SUBJECT13         SUBJECT23       SUBJECT33          0         0      0         14
# 14    7    B -0.12 aneutral         SUBJECT14         SUBJECT24       SUBJECT34          0         0      0         12


The first problem is, that the dummy variables partially overlap. My model would look something like this.

mymodel2 <- lm(RET ~ CLASS + LITIGATION + POLLUTION + LAYOFF
+ LITIGATION * CLASS + POLLUTION * CLASS + LAYOFF * CLASS   # Interaction Variables
+ CONTROLVAR,     # Control Variables
data=df2)


I'm quite sure this model is wrong, but I don't know exactly what is wrong and how to implement a model that works for this task. Can anyone help me with this problem? Thank You!

Why are you "quite sure" that it's wrong? I'm "quite sure" that it's right.

The "overlapping" case you're describing is called an "interaction." An interaction between two variables has the interpretation of creating a different slope for one variable at each level of the other variable. The most common case is between a continuous and categorical variable, so that the categorical variable can be said to "modify" the effect of the continuous one by changing the slope (do some algebra to convince yourself of this). But this interpretation extends to your case as well.

An interaction with a categorical variable is conceptually, but not statistically, equivalent to fitting a separate regression for each level of that categorical variable. The difference lies in the way the variance of the error is estimated, and therefore in the test statistics for significance.

You can also interpret an interaction between two categorical variables with $k_1$ and $k_2$ categories, respectively, as a single variable with $k_1\cdot k_2$ categories. Again, convince yourself of this.

However, you don't need to specify the main effect terms in your formula in R. The * operator inside a formula includes these implicitly. Instead you can use RET ~ CLASS * (LITIGATION + POLLUTION + LAYOFF) which expands to RET ~ CLASS * LITIGATION + CLASS * POLLUTION + CLASS * LAYOFF)

summary(mymodel2)
#
# Call:
# lm(formula = RET ~ CLASS * (LITIGATION + POLLUTION + LAYOFF),
#     data = df2)
#
# Residuals:
#      Min       1Q   Median       3Q      Max
# -0.17333 -0.14875  0.00000  0.04125  0.33667
#
# Coefficients: (4 not defined because of singularities)
#                          Estimate Std. Error t value Pr(>|t|)
# (Intercept)               0.04500    0.11750   0.383  0.71494
# CLASSnegative             0.70500    0.20352   3.464  0.01340 *
# CLASSpositive            -0.08167    0.17949  -0.455  0.66511
# LITIGATION               -3.40000    0.33235 -10.230 5.09e-05 ***
# POLLUTION                 2.95500    0.26274  11.247 2.95e-05 ***
# LAYOFF                    0.35000    0.28782   1.216  0.26964
# CLASSnegative:LITIGATION       NA         NA      NA       NA
# CLASSpositive:LITIGATION       NA         NA      NA       NA
# CLASSnegative:POLLUTION  -4.90500    0.38971 -12.586 1.54e-05 ***
# CLASSpositive:POLLUTION  -1.51833    0.37772  -4.020  0.00696 **
# CLASSnegative:LAYOFF           NA         NA      NA       NA
# CLASSpositive:LAYOFF           NA         NA      NA       NA
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#
# Residual standard error: 0.235 on 6 degrees of freedom
# Multiple R-squared:  0.9816,    Adjusted R-squared:  0.9601
# F-statistic: 45.73 on 7 and 6 DF,  p-value: 8.687e-05


One thing that is wrong: don't use df as a variable name in R, or for that matter data. One is the function for the F density, and the other one loads built-in data sets.

• Thanks for your answer! I've still got some questions. Look at the first row of ´df2´. This article is about litigation AND pollution! isn't this a problem? I mean, doesn't a dummy variable need to be exclusive like male/female? And another thing: let's assume i have a model with the three dummy variables "class" (3 levels), "region" (5 levels) and "subject" (3 levels). Do I need to leave out one of the levels for EACH of the three dummy variables in the model? – cptn Jun 24 '14 at 7:09
• Second question first: Categorical variables are not the same as dummy variables. "Class," "region," and "subject" are categorical. Dummy variables are always binary. The way categorical variables are typically handled is by making a new dummy variable for each category and, yes, leaving one out as a baseline. You have three categorical variables. Each will need its own set of dummies. In lm, R does this for you automatically if it detects that a variable has class (or should be coerced to class) factor. – shadowtalker Jun 24 '14 at 13:54
• First question: Yes, dummy variables (and by extension categorical variables) have to be exclusive, but that's not a problem here. Each variable is exclusive within itself: e.g. the article is either about pollution or it isn't and that covers the entire space of possibility. If you're confused about why you don't have to leave out one of those dummy variables, it's precisely because they don't come from the same categorical variable. – shadowtalker Jun 24 '14 at 14:01
• @ ssdecontrol: Thanks a lot for your explanation! Have you got an idea on how to visualize/plot the effects of such a regression? Boxplots for the three categories positive, negative and neutral for each of the subjects? Is that possible? – cptn Jun 24 '14 at 19:48
• You could do that for every combination of 1's and 0's for subjects (so a plot for "POLLUTION", a plot for "LITIGATION & POLLUTION", etc). You could then ghost the actual (jittered) data right on top of the boxplot to see how well it matches. Or put an empirical box plot and a predicted box plot side by side. Haven't seen that one before but I like it. What I have recently seen is boxplots of the coefficients themselves, where the boxes and whiskers are derived from the standard errors. – shadowtalker Jun 24 '14 at 21:26