# Categorical response variable prediction

I have the following kind of data (coded in R):

v.a = c('cat', 'dog', 'dog', 'goat', 'cat', 'goat', 'dog', 'dog')
v.b = c(1, 2, 1, 2, 1, 2, 1, 2)
v.c = c('blue', 'red', 'blue', 'red', 'red', 'blue', 'yellow', 'yellow')
set.seed(12)
v.d = rnorm(8)
aov(v.a ~ v.b + v.c + v.d) # Error


I would like to know if the value of v.b or the value of v.c has any ability to predict the value of v.a. I would run an ANOVA (as shown above) but I think it does not make any sense since my response variable is not ordinal (it is categorical). What should I do?

• Learn about multinomial logit. Here are two, free online books by Kenneth Train (elsa.berkeley.edu/books/choice.html and elsa.berkeley.edu/books/choice2.html). I think those are graduate level books. Or just google around for "multinomial logit."
– Bill
Nov 5 '13 at 13:59
• @Bill this looks like a good start on an answer. Please consider expanding a little bit on what multinomial logit models are, and posting it as an answer. Nov 5 '13 at 15:03

You could use ANY classifier. Including Linear Discriminants, multinomial logit as Bill pointed out, Support Vector Machines, Neural Nets, CART, random forest, C5 trees, there are a world of different models that can help you predict $$v.a$$ using $$v.b$$ and $$v.c$$. Here is an example using the R implementation of random forest:

# packages
library(randomForest)

#variables
v.a= c('cat','dog','dog','goat','cat','goat','dog','dog')
v.b= c(1,2,1,2,1,2,1,2)
v.c= c('blue', 'red', 'blue', 'red', 'red', 'blue', 'yellow', 'yellow')

# model fit
# note that you must turn the ordinal variables into factor or R wont use
# them properly
model <- randomForest(y=as.factor(v.a),x=cbind(v.b,as.factor(v.c)),ntree=10)

#plot of model accuracy by class
plot(model) # model confusion matrix
modelconfusion  Clearly these variables don't show a strong relation. • @JEquihua Could you please tell me a bit more about what is a "tree" and what is the meaning of the output(confusion matrix and the plot). Thanks a lot! Nov 8 '13 at 8:58 • I will. I'm very busy, give me a little time. @Remi.b Nov 19 '13 at 15:59 This is a more a partial practical answer, but it works for me to do some exercises before getting deeply into theory. This ats.ucla.edu link is a reference that might help beggining to understand about multinomial logistic regression (as pointed out by Bill), in a more practical way. It presents reproducible code to understand function multinom from nmet package in R and also gives a briefing about outputs interpretation. Consider this code: va = c('cat','dog','dog','goat','cat','goat','dog','dog') # cat will be the outcome baseline vb = c(1,2,1,2,1,2,1,2) vc = c('blue','red','blue','red','red','blue','yellow','yellow') # blue will be the vc predictor baseline set.seed(12) vd = round(rnorm(8),2) data = data.frame(cbind(va,vb,vc,vd)) library(nnet) fit <- multinom(va ~ as.numeric(vb) + vc + as.numeric(vd), data=data) # weights: 18 (10 variable) initial value 8.788898 iter 10 value 0.213098 iter 20 value 0.000278 final value 0.000070 converged fit Call: multinom(formula = va ~ as.numeric(vb) + vc + as.numeric(vd), data = data) Coefficients: (Intercept) as.numeric(vb) vcred vcyellow as.numeric(vd) dog -1.044866 120.3495 -6.705314 77.41661 -21.97069 goat 47.493155 126.4840 49.856414 -41.46955 -47.72585 Residual Deviance: 0.0001656705 AIC: 20.00017  This is how you can interpret the log-linear fitted multinomial logistic model: \begin{align} \ln\left(\frac{P(va={\rm cat})}{P(va={\rm dog})}\right) &= b_{10} + b_{11}vb + b_{12}(vc={\rm red}) + b_{13}(vc={\rm yellow}) + b_{14}vd \\ &\ \\ \ln\left(\frac{P(va={\rm cat})}{P(va={\rm goat})}\right) &= b_{20} + b_{21}vb + b_{22}(vc={\rm red}) + b_{23}(vc={\rm yellow}) + b_{24}vd \end{align} Here is an excerpt about how the model parameters can be interpreted: • A one-unit increase in the variable vd is associated with the decrease in the log odds of being "dog" vs. "cat" in the amount of 21.97069 (b_{14}$). the same logic for the second line but, considering "goat" vs. "cat" with ($b_{24}$=-47.72585). • The log odds of being "dog" vs. "cat" will increase by 6.705314 if moving from vc="blue" to vc="red"($b_{12}\$).

.....

There is much more in the article, but I thought this part to be the core.

Reference:

R Data Analysis Examples: Multinomial Logistic Regression. UCLA: Statistical Consulting Group.
from http://www.ats.ucla.edu/stat/r/dae/mlogit.htm (accessed November 05, 2013).