# Dealing with a categorical variable that can take multiple levels simultaneously

I recently posted a question with many parts and I'd like to focus in on just one issue that I didn't emphasize in the original post.

My data is a list of records, each one representing an educational seminar event. I have a continuous variable that represents the revenue brought in by each seminar, which is the response variable in my regression. I also have a number of categorical variables which are acting as factors/IVs.

One of those categorical factors is the speaker hosting the event. The trouble is that sometimes more than one speaker hosts a particular event. To date, all our speakers have been drawn from a pool of 154. Most of the time, just one speaker is used, but in about 10% of the data points, two, three or even four speakers were used. Currently, this is represented in my data with slashes ("Speaker One / Speaker Two / Speaker Three"). I've written a Python script that can find the average revenue on a given date interval for seminars which take a given level of a categorical variable (for example, it could return the average revenue for all seminars in 2008 that for which Speaker One was the host)...my script can read the multiple speaker format fine, reading names on opposite sides of a " / " as separate speakers.

Unfortunately, R doesn't seem to be able to do anything like that...I've run a multiple regression on my data and obviously it treats "Speaker One", "Speaker Two" and "Speaker One / Speaker Two" as three different speakers. My multiple r-squared value is less than 0.5 so I'm hoping that resolving this issue would help improve the model...how best to proceed?

• Can you write a Python script that will recode your speaker variable into a set of new variables speaker1, speaker2, (etc), with some instances where more than one variable has a 1 in it? Would that solve your problem? Jul 18 '12 at 16:28
• I could do so very easily, in fact. I just didn't know if that would mess with R's built-in dummy-coding. Assuming I do it this way, is it possible to group all the speaker variables together somehow in R? Somehow it seems clunky to me to have to tell R to include 154 variables by hand... (i.e. fit <- lm(REV ~ SPEAKER1 + SPEAKER2 + SPEAKER 3 + ... SPEAKER154...) Jul 18 '12 at 16:40
• It would be very tedious to enter 154 variables manually into an R formula. Note that if everything in a given data.frame is relevant & useable you can do y~.. Check out this for more info. Jul 18 '12 at 16:51
• Great tip, thanks. Another question: how should I deal with this problem for ANOVA, which doesn't seem to use dummy-coding at all? Jul 18 '12 at 17:01
• In R, if you enter a term as a factor, R dummy codes it for you behind the scenes w/ the 1st level as the reference category. In your case, I suspect you can't think of these as 154 levels of the same factor b/c of the multiple speaker issue, thus you can write a script & list them in the formula manually (only not...). As for ANOVA, it's actually just a multiple regression model with only categorical covariates; ie, there's no difference & dummy coding is being used (albeit out of sight). Jul 18 '12 at 17:11

Two models come to mind: the revenue may have a contribution from the presence of each speaker or it may have a contribution from the speaker's presence, weighted by their participation in the event. In either case the coding would be similar: to each speaker corresponds a variable that is zero when the speaker is not involved and is nonzero when they are. In the first model the variable's value would be one whenever a speaker is involved in an event and zero otherwise. In the second model, those ones might be reweighted a priori.

You can try several models with several weighting schemes to see what might work the best: after all, this problem has somewhat of an exploratory nature to it.

That leaves us the practical issue of coding the models. Creating one column for each speaker is straightforward, but the concerns expressed in comments have to do with the length and complexity of the resulting formula expressions. Fortunately, formulas can be created dynamically. Here is an illustration. First, let's create some simulated data. In each row at least one speaker and usually two or three speakers are involved:

set.seed(17)
n.records <- 1000
n.speakers <- 154
i <- c(rep(1,3), rep(0, n.speakers-2))
x.matrix <- sapply(1:n.records, function(j) sample(i, n.speakers))
x <- as.data.frame(t(x.matrix))


Let's name the columns "Speaker1", "Speaker2", etc (and retain this list of names for later):

colnames(x) <- colnames <- lapply(1:n.speakers, function(i) sprintf("Speaker%d",i))


Throw in a response variable:

x\$y <- rnorm(n.records)


Let's see how this response depends on the speaker data. To do this, we create a formula from the column names we retained earlier:

formula <- as.formula(paste("y ~", paste(colnames, collapse="+"))
fit <- lm(formula, data=x)
summary(fit)


There's no problem: R handles a formula of this length with aplomb. Extending this formula to include other variables is simple; e.g., hard-code the remainder and paste it on to the end of this computed formula.