150k views

### What is rank deficiency, and how to deal with it?

Fitting a logistic regression using lme4 ends with Error in mer_finalize(ans) : Downdated X'X is not positive definite. A likely cause of this error is ...
18k views

### Is regression with L1 regularization the same as Lasso, and with L2 regularization the same as ridge regression? And how to write “Lasso”?

I'm a software engineer learning machine learning, particularly through Andrew Ng's machine learning courses. While studying linear regression with regularization, I've found terms that are confusing: ...
5k views

### Why is the James-Stein estimator called a “shrinkage” estimator?

I have been reading about the James-Stein estimator. It is defined, in this notes, as $$\hat{\theta}=\left(1 - \frac{p-2}{\|X\|^2}\right)X$$ I have read the proof but I don't understand the ...
11k views

### Logistic Regression - Multicollinearity Concerns/Pitfalls

In Logistic Regression, is there a need to be as concerned about multicollinearity as you would be in straight up OLS regression? For example, with a logistic regression, where multicollinearity ...
1k views

### L1 and L2 penalty vs L1 and L2 norms

I understand the usages of L1 and L2 norms however I am unsure of usage of L1 and L2 penalty when building models. From what I understand: L1: Laplace Prior L2: Gaussian Prior are two of the ...
67 views

### Find correlations based on large multivariable time-based data with one output per dataset

I am not well versed in anything beyond basic statistics but have been tasked with coming up with a "grading" scale for wear on a part based on data we have collected. I am in need of help figuring ...
1k views

### Why we use Ridge regression instead of Least squares in Multicollinearity?

Why do we use Ridge regression instead of Least squares in Multicollinearity? Which one is correct: a. lower bias and higher variance b. lower bias with the same variance c. higher bias with a ...
135 views

### Logistic sample and case numbers

I have some questions about binary logistic regression. For my research, I am planning to use 12 predictors, and my sample consists of 129 cases. However, I know of a 1 to 10 rule. Additionally, my ...
33 views

### Is it useful to use sparse regression (e.g. Lasso) when the number of observations is significantly larger than the number of covariates?

I'm learning about penalized/sparse regression and I noticed that the examples used for penalized/sparse regression, e.g. Lasso, are usually cases where the number of observations is significantly ...
97 views

### Choosing between feature selection and regularization to overcome over-fitting in categorical regression

In order to overcome over-fitting during a regression process over categorical features, one can either 1) Apply L1/L2/Elastic regularization during the regression, for example as answered here ...
212 views

### How to identify important independent variables for a dependent variable?

I have a dependent variable (DV) and about 200 independent variables (IVs). I want to understand which of the 10-20 variables are important for this DV. I could do: PCA - However it'll only tell me ...