Linked Questions

2
votes
0answers
67 views

Find correlations based on large multivariable time-based data with one output per dataset

I am not well versed in anything beyond basic statistics but have been tasked with coming up with a "grading" scale for wear on a part based on data we have collected. I am in need of help figuring ...
1
vote
0answers
33 views

Is it useful to use sparse regression (e.g. Lasso) when the number of observations is significantly larger than the number of covariates?

I'm learning about penalized/sparse regression and I noticed that the examples used for penalized/sparse regression, e.g. Lasso, are usually cases where the number of observations is significantly ...
0
votes
0answers
13 views

Strategy to analyze large ( 20 mill rows and 200 columns) to predict a single variable

I am curious to understand how data scientists attack exceedingly large datasets in order to build a regression model for y? How does one decide where to start from? Reduce a large number of columns ...
2
votes
1answer
1k views

L1 and L2 penalty vs L1 and L2 norms

I understand the usages of L1 and L2 norms however I am unsure of usage of L1 and L2 penalty when building models. From what I understand: L1: Laplace Prior L2: Gaussian Prior are two of the ...
19
votes
1answer
5k views

Why is the James-Stein estimator called a “shrinkage” estimator?

I have been reading about the James-Stein estimator. It is defined, in this notes, as $$ \hat{\theta}=\left(1 - \frac{p-2}{\|X\|^2}\right)X$$ I have read the proof but I don't understand the ...
1
vote
2answers
1k views

Why we use Ridge regression instead of Least squares in Multicollinearity?

Why do we use Ridge regression instead of Least squares in Multicollinearity? Which one is correct: a. lower bias and higher variance b. lower bias with the same variance c. higher bias with a ...
33
votes
1answer
19k views

Is regression with L1 regularization the same as Lasso, and with L2 regularization the same as ridge regression? And how to write “Lasso”?

I'm a software engineer learning machine learning, particularly through Andrew Ng's machine learning courses. While studying linear regression with regularization, I've found terms that are confusing: ...
1
vote
0answers
99 views

Choosing between feature selection and regularization to overcome over-fitting in categorical regression

In order to overcome over-fitting during a regression process over categorical features, one can either 1) Apply L1/L2/Elastic regularization during the regression, for example as answered here ...
1
vote
0answers
213 views

How to identify important independent variables for a dependent variable?

I have a dependent variable (DV) and about 200 independent variables (IVs). I want to understand which of the 10-20 variables are important for this DV. I could do: PCA - However it'll only tell me ...
1
vote
1answer
135 views

Logistic sample and case numbers

I have some questions about binary logistic regression. For my research, I am planning to use 12 predictors, and my sample consists of 129 cases. However, I know of a 1 to 10 rule. Additionally, my ...
1
vote
0answers
464 views

Cross validation for variable selection and coefficient shrinkage?

Is cross validation an appropriate technique for variable selection and regression coefficient shrinkage? A former colleague of mine used 10-fold CV to compare the regression coefficients from the ...
87
votes
3answers
153k views

What is rank deficiency, and how to deal with it?

Fitting a logistic regression using lme4 ends with Error in mer_finalize(ans) : Downdated X'X is not positive definite. A likely cause of this error is ...
16
votes
1answer
11k views

Logistic Regression - Multicollinearity Concerns/Pitfalls

In Logistic Regression, is there a need to be as concerned about multicollinearity as you would be in straight up OLS regression? For example, with a logistic regression, where multicollinearity ...