0
$\begingroup$

Les say I have a data set with several measures and one factor (classification) like the one bellow (for the sake of simplicity, I'm simulating 10 rows and 5 variables only)

I'd like to know how much each variable contribute to the overall classification. I thought about running a linear regression, but I'm wondering if it makes sense to use it to "explain" a factor

When I run lm(classification ~ ., data =data) I get a warning saying

Warning messages:
1: In model.response(mf, "numeric") :
  using type = "numeric" with a factor response will be ignored
2: In Ops.factor(y, z$residuals) : ‘-’ not meaningful for factors

but I do get a result (intercept and coefficients for each variable).

My questions are: do they make any sense? And: is there a better way to get to the answer I'm looking for?

   classification  variable_1  variable_2 variable_3 variable_4 variable_5
1               5 -0.90174176 -0.64796703  1.2106427 -0.9229394 -0.6578518
2               5  1.75760214  0.18486432  0.2018499  0.1301168 -0.6510428
3               8 -0.29445029 -0.23108298 -2.6244614  0.3745607  0.3124868
4               4  0.78639724  1.04943276 -0.6047869 -0.4275781  0.6395614
5               3 -2.06554518  0.07336021  2.8142735  1.0558045 -0.1818247
6               4  0.04374419 -0.13775079  0.6132946 -0.5890983  1.9965892
7              10 -1.46731867  1.00367532 -0.8626940 -1.8378582  0.2702731
8               8  0.27206146 -0.13775707  2.6827356  1.5554446  0.1549394
9               5  0.58075881  2.03567118  0.2056770 -0.2935464 -1.3586576
10              9  0.57725709 -0.25396790  0.6640166 -1.9626897  0.3650243

Code to reproduce it:

data <- data.frame(classification=sample(3:10,replace=TRUE,size=10))

for(i in 1:5){
  data[,paste0("variable_",i)]<-rnorm(10)
}

thanks

$\endgroup$
  • $\begingroup$ Are you running a classification model at all, or is this linear regression the only modeling you've done? $\endgroup$ – Dougal Jul 13 '15 at 16:48
  • $\begingroup$ thats the only one I've done so far, wondering if it is a good option $\endgroup$ – Diego Jul 13 '15 at 22:16
1
$\begingroup$

When I need to answer similar question I usually: 1) Run a gradient boosting classifier against my data using the scikit-learn package (it has this algorithm built-in) 2) Get the feature_importances_.

Just in my experience, feature_importances_ show really good approximation of how important the features are. As far as I see R's package gbm also provides same classifier and similar importance approximation method so I suggest you to try it. The nature of GBM classifier is that it should approximate the importance of features relatively well. Same for RandomForest, by the way.

$\endgroup$
0
$\begingroup$

Variables 1,2, & 3 are indirectly related to classification; variables 4 & 5 are directly related to classification. One model I evolved shows that variable 5 and variable 3 have the strongest influence on classification but in opposite directions.

classification = 6.754 - 130*variable_1*variable_2*variable_4*variable_3^3*variable_5^3

R-squared ~0.79

John NAtlantis

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.