1
$\begingroup$

I have a dataset with 75 features that I plan to use to create a logistic regression model. However, I want to reduce this number of features as my computer can barely handle this large dataset. Accuracy wise, are there any negative effects of having too many features in a logistic regression model? Using dummy variables after one hot encoding also drastically increases the number of columns, does this too have a negative effect on accuracy?

In regards to Pearson, I am thinking of testing it on all of my continuous features and when two continuous features have a correlation of above .7, I will remove one of the features at random from the training set. I won't remove outliers, because outliers will reduce the Pearson correlation coefficient and therefore make me keep both features, which makes sense because both features impact the logistic regression model in different ways because of the presence of outliers. Is my understanding sound?

Finally, is there a good way to test correlation between features that are not continuous, specifically features that I one hot encode using dummy variables? Spearman's tests ordinal variables so I'm not sure it would work on one hot encoded features.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Regarding the first two questions

Accuracy wise, are there any negative effects of having too many features in a logistic regression model?

Yes, if you have too many features and too little examples you might end up over-fitting the model, you could use l1-regularization to ensure that even if you feed your model a lot of features some of the weights will be 0, however this requires adjusting a cost parameter.

Using dummy variables after one hot encoding also drastically increases the number of columns, does this too have a negative effect on accuracy?

Using dummy variables per say should not cause problems to the model, however increasing the number of features significantly might lead to over-fitting.

Improve this answer
$\endgroup$
3
  • $\begingroup$ OK, so while dummy variables leads to more columns, that technically does not sound as increasing the feature set? Or does it? Basically I'm just asking if dummy variables decrease accuracy because they add a ton of new columns in my dataset. $\endgroup$
    – tonychen
    Jul 14, 2017 at 20:48
  • $\begingroup$ Since in logistic regression each column is treated as different feature when you transform a column into dummy variables from the algorithms perspective these are separate features. For example if your dummy variable has 100 categories you create 100 features and you could end up overfitting to some of these regardless of their where they came from. $\endgroup$
    – Miguel
    Jul 15, 2017 at 21:17
  • $\begingroup$ Whether it does or not it decrease the accuracy depends on your particular case. $\endgroup$
    – Miguel
    Jul 15, 2017 at 21:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.