How to identify variable (from many variables) which is able to discriminate between groups?

Question

I currently have a data frame with 98 observations and 107 variables. All of the variables are numeric, but one variable is binary (yes or no). My goal is to determine which correlation and/or variable give the greatest segregation between the yes and no samples. I have been using the pairs () function to do this, but I can only do a few variables at a time. Is there a way to determine which correlation gives the greatest discernment between yes and no?

To Clarify - My table is 98 observations and 107 variables, but doing a correlation matrix with the pairs function is not able to fit all of the variables.

I have used this function:

pairs(x[70:80], ch=21, bg=c("red","green")[unclass(x$outcome)])

Answer 1

When you have multiple variable and you are looking for variable(s) which is the best for discriminating between groups ("yes" and "no" samples in this case) a tool for this is MANOVA.

# Suppose we have a data.frame with 7 variables and one group:
my.data<-data.frame(v1=rnorm(100),v2=rnorm(100),v3=rnorm(100),
v4=rnorm(100),v5=rnorm(100),v6=rnorm(100), v7=c(rnorm(50),
rnorm(50)+20),response=rep(c("yes","no"), each=50))

# run MANOVA
my.mnv<-manova(cbind(v1,v2,v3,v4,v5,v6,v7) ~ response, data=my.data)

# and look on p-values (if p-value < 0.05 then it is able to 
# significantly discriminate between "yes" and "no")
summary.aov(my.mnv)

# plot
pairs(my.data[c("v1","v2","v3","v4","v5","v6","v7")], pch=22,
bg=c("red", "yellow")[unclass(my.data$response)])

---

It's not good to make conclusions about statistical significance based on looking on the plot (although it is necessary to look on it). In you case of 107 variables the pairs() plot will be very chaotic.

Answer 2

An alternate method of finding variable importance is using random forests.

A package called varSelRF was built specifically for this purpose. This method isn't designed to be right all the time, but is a rather quick way of dealing with large dimensions to get a semblance of which variables could at the first level affect the response variable.

Combine this with an MANOVA and you stand a decent chance of finding your key variables.

Answer 3

I would advise against using statistical methods for determining the "best" variable. You have only 98 observations (how are "yes" and "no" answers distributed?) and more variables than cases. This is a recipe for disaster in the sense that any attempt to build a model with all variables is prone to overfit the data. You will find packages that try to do the trick and some careful cross-validation might help you to avoid some obvious pitfalls, but do not assume that you will learn much on a conceptual level. My suggestion would therefore be to eliminate variables that are weak on theoretical grounds before moving to the analysis step or to collect more cases if that is possible. And: test some simple models (equal weighting of variables) as competitors to get some benchmarks.

There is less literature on the soft and fuzzy process of conceptual variable selection than on algorithmic ways to "solve" this problem, but this is not necessarily due to the superiority of the latter. Some pointers in the literature could be:

Dawes, R. M. (1979). The robust beauty of improper linear models in decision making. American psychologist, 34(7), 571.

Freedman, D. A. (1991). Statistical models and shoe leather. Sociological methodology, 21, 291-313.

Freedman, D. (1999). From association to causation: some remarks on the history of statistics. Statistical Science, 14(3), 243-258.