1
$\begingroup$

I want to run logistic regression to predict binary outcome , however I have 300+ independent variable.

I am new in analytics and statistics ,in my opinion first I need to dimension reduction.

I ran PCA in R and I am getting below error "Error in princomp.default(input, scores = TRUE, COR = TRUE) : covariance matrix is not non-negative definite"

I am not able to resolve above error also in terms of approach if anyone can provide guidance that would be good .

  1. what should I do to reduce number of variables , identifying powerful predictors ...
$\endgroup$
1
  • 3
    $\begingroup$ It is not appropriate to launch this analysis without intensive study of the statistical methods. $\endgroup$ Commented Dec 22, 2019 at 14:06

2 Answers 2

3
$\begingroup$

Cannot comment due to low reputation.

Are your variables continuous or discreete?

You can use regularized models such as Lasso or Elastic-net. See this R package. Both models are based on the L1-norm. Classifier built with this form of regularization are sparse, meaning that some of the weights are exactly zero.

From the weights you can derive some form of feature relevance.

$\endgroup$
1
  • 4
    $\begingroup$ +1 for suggestion of regularised logistic regression, this is usually a better approach to avoiding over-fitting than feature selection as feature selection is itself prone to over-fitting the feature selection criterion. L2 regularisation also works well (but doesn't identify the relevant features explicitly), and is easier to implement. In general, only perform feature selection is knowing which features are important is a goal of the analysis, if you are just interested in predictive performance, regularisation is usually a better approach. $\endgroup$ Commented Mar 7, 2016 at 11:00
1
$\begingroup$

Some observations:

  • 300+ variables is way too many for logistic regression. So dimension reduction techniques are indeed useful in this sense. Having said that, what are your variables and what is it that you are predicting? A bit of context would be very useful, as, for starters, it may help you select an appropriate dimension reduction technique (for example: PCA or Factor Analysis).

  • 300+ variables and your covariance/correlation matrix is not positive definite probably because it is singular (i.e. non-invertible). This essentially means that a number of your variables are redundant. To be a bit more technical, a singular matrix has a determinant of $0$ due to linear dependency, i.e. columns in your covariance/correlation matrix can be expressed as a linear combination of other columns in your covariance matrix. This would usually hint that in your data matrix, there is either very highly (or perfectly) correlated predictors or there is a substantial proportion of $0$ entries within a set of variables. Indeed with 300+ variables, I would be surprised if neither is the case.

  • Hence, before you plunge into logistic regression, I stress this: Do spend time understanding your variables and their relationships. For example, compute a correlation matrix. Check how many correlations between variables are very high (e.g. > |.85|). Examine descriptives and check histograms. Examine the proportion of $0$ entries per variable? Altogether, these basic steps will prepare you better for conducting dimension reduction techniques.

$\endgroup$
3
  • $\begingroup$ Why do you say that 300 is way too many variables for a logistic regression? What if there are a ton of observations? $\endgroup$
    – Dave
    Commented Dec 23, 2019 at 1:18
  • $\begingroup$ @Dave because it is very likely that some variables are redundant, and causing more troubles than adding benefit, if they are multicollinear for example. So the model can't sensibly distinguish their contribution so their beta and se might be very biased. Other examples would include a higher chance of singular or near singular covariance matrix. As I mentioned previously, I am glad to see otherwise, but I highly doubt that the OP has no troubles with 370 variables, of which some are probably very similar to one another $\endgroup$ Commented Dec 23, 2019 at 2:08
  • $\begingroup$ @Dave also, with that many variables the OP is very much capitalizing on the chance that some of the variables are statistically significant by chance. For example, with 370 variables about 18 variables are statistically significant by chance. Again, major interpretation problems. Just communicating a model with 370 predictors would give me a headache of how to even explain it in language simple enough to others $\endgroup$ Commented Dec 23, 2019 at 2:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.