Questions tagged [regression]
Techniques for analyzing the relationship between one (or more) "dependent" variables and "independent" variables.
30,269
questions
413
votes
17
answers
172k
views
What happens if the explanatory and response variables are sorted independently before regression?
Suppose we have data set $(X_i,Y_i)$ with $n$ points. We want to perform a linear regression, but first we sort the $X_i$ values and the $Y_i$ values independently of each other, forming data set $(...
- 3,811
287
votes
6
answers
49k
views
Is $R^2$ useful or dangerous?
I was skimming through some lecture notes by Cosma Shalizi (in particular, section 2.1.1 of the second lecture), and was reminded that you can get very low $R^2$ even when you have a completely linear ...
- 10.1k
281
votes
2
answers
236k
views
Interpretation of R's lm() output
The help pages in R assume I know what those numbers mean, but I don't.
I'm trying to really intuitively understand every number here. I will just post the output and comment on what I found out. ...
- 4,311
226
votes
8
answers
543k
views
In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?
Am I looking for a better behaved distribution for the independent variable in question, or to reduce the effect of outliers, or something else?
- 2,411
218
votes
5
answers
276k
views
How exactly does one “control for other variables”?
Here is the article that motivated this question: Does impatience make us fat?
I liked this article, and it nicely demonstrates the concept of “controlling for other variables” (IQ, career, income, ...
- 2,997
211
votes
9
answers
233k
views
How to deal with perfect separation in logistic regression?
If you have a variable which perfectly separates zeroes and ones in target variable, R will yield the following "perfect or quasi perfect separation" warning message:
...
- 7,281
210
votes
4
answers
176k
views
When should I use lasso vs ridge?
Say I want to estimate a large number of parameters, and I want to penalize some of them because I believe they should have little effect compared to the others. How do I decide what penalization ...
- 2,211
184
votes
11
answers
232k
views
When is it ok to remove the intercept in a linear regression model?
I am running linear regression models and wondering what the conditions are for removing the intercept term.
In comparing results from two different regressions where one has the intercept and the ...
- 2,041
174
votes
3
answers
209k
views
How are the standard errors of coefficients calculated in a regression?
For my own understanding, I am interested in manually replicating the calculation of the standard errors of estimated coefficients as, for example, come with the output of the ...
- 1,843
172
votes
4
answers
470k
views
When is R squared negative? [duplicate]
My understanding is that $R^2$ cannot be negative as it is the square of R. However I ran a simple linear regression in SPSS with a single independent variable and a dependent variable. My SPSS output ...
- 2,245
159
votes
8
answers
109k
views
Why L1 norm for sparse models
I am reading books about linear regression. There are some sentences about the L1 and L2 norm. I know the formulas, but I don't understand why the L1 norm enforces sparsity in models. Can someone give ...
- 1,833
150
votes
9
answers
326k
views
What is the difference between linear regression on y with x and x with y?
The Pearson correlation coefficient of x and y is the same, whether you compute pearson(x, y) or pearson(y, x). This suggests that doing a linear regression of y given x or x given y should be the ...
- 3,303
150
votes
9
answers
174k
views
Why does a time series have to be stationary?
Would like to understand primary reasons for making a data stationary?
I understand that a stationary time series is one whose mean and variance is constant over time. Can someone please explain why ...
- 1,501
140
votes
3
answers
51k
views
What if residuals are normally distributed, but y is not?
I've got a weird question. Assume that you have a small sample where the dependent variable that you're going to analyze with a simple linear model is highly left skewed. Thus you assume that $u$ is ...
- 6,003
140
votes
8
answers
180k
views
Is it necessary to scale the target value in addition to scaling features for regression analysis?
I'm building regression models. As a preprocessing step, I scale my feature values to have mean 0 and standard deviation 1. Is it necessary to normalize the target values also?
- 2,743
140
votes
3
answers
274k
views
What is the difference between linear regression and logistic regression?
What is the difference between linear regression and logistic regression?
When would you use each?
- 2,933
135
votes
4
answers
173k
views
What does a "closed-form solution" mean?
I have come across the term "closed-form solution" quite often. What does a closed-form solution mean? How does one determine if a close-form solution exists for a given problem? Searching online, I ...
- 2,673
133
votes
6
answers
215k
views
Difference between confidence intervals and prediction intervals
For a prediction interval in linear regression you still use $\hat{E}[Y|x] = \hat{\beta_0}+\hat{\beta}_{1}x$ to generate the interval. You also use this to generate a confidence interval of $E[Y|x_0]$....
- 1,525
133
votes
4
answers
270k
views
PCA and proportion of variance explained
In general, what is meant by saying that the fraction $x$ of the variance in an analysis like PCA is explained by the first principal component? Can someone explain this intuitively but also give a ...
- 3,303
132
votes
18
answers
119k
views
Including the interaction but not the main effects in a model
Is it ever valid to include a two-way interaction in a model without including the main effects? What if your hypothesis is only about the interaction, do you still need to include the main effects?
- 7,440
131
votes
9
answers
102k
views
Numerical example to understand Expectation-Maximization
I am trying to get a good grasp on the EM algorithm, to be able to implement and use it. I spent a full day reading the theory and a paper where EM is used to track an aircraft using the position ...
- 2,673
128
votes
1
answer
87k
views
What is an ablation study? And is there a systematic way to perform it?
What is an ablation study? And is there a systematic way to perform it? For example, I have $n$ predictors in a linear regression which I will call as my model.
How will I perform an ablation study ...
- 9,227
127
votes
7
answers
95k
views
Why use gradient descent for linear regression, when a closed-form math solution is available?
I am taking the Machine Learning courses online and learnt about Gradient Descent for calculating the optimal values in the hypothesis.
h(x) = B0 + B1X
why we ...
- 1,373
126
votes
3
answers
142k
views
Does an unbalanced sample matter when doing logistic regression?
Okay, so I think I have a decent enough sample, taking into account the 20:1 rule of thumb: a fairly large sample (N=374) for a total of 7 candidate predictor variables.
My problem is the following: ...
- 1,363
125
votes
3
answers
55k
views
Why does the Lasso provide Variable Selection?
I've been reading Elements of Statistical Learning, and I would like to know why the Lasso provides variable selection and ridge regression doesn't.
Both methods minimize the residual sum of squares ...
- 1,462
119
votes
4
answers
52k
views
Why isn't Logistic Regression called Logistic Classification?
Since Logistic Regression is a statistical classification model dealing with categorical dependent variables, why isn't it called Logistic Classification? Shouldn't the "Regression" name be reserved ...
- 2,176
119
votes
8
answers
55k
views
What is the benefit of breaking up a continuous predictor variable?
I'm wondering what the value is in taking a continuous predictor variable and breaking it up (e.g., into quintiles), before using it in a model.
It seems to me that by binning the variable we lose ...
- 1,801
117
votes
12
answers
69k
views
When should linear regression be called "machine learning"?
In a recent colloquium, the speaker's abstract claimed they were using machine learning. During the talk, the only thing related to machine learning was that they perform linear regression on their ...
- 1,527
117
votes
5
answers
21k
views
What skills are required to perform large scale statistical analyses?
Many statistical jobs ask for experience with large scale data. What are the sorts of statistical and computational skills that would be need for working with large data sets. For example, how about ...
- 2,827
114
votes
4
answers
279k
views
How does the correlation coefficient differ from regression slope?
I would have expected the correlation coefficient to be the same as a regression slope (beta), however having just compared the two, they are different. How do they differ - what different information ...
- 14.5k
108
votes
9
answers
57k
views
Is there an intuitive explanation why multicollinearity is a problem in linear regression?
The wiki discusses the problems that arise when multicollinearity is an issue in linear regression. The basic problem is multicollinearity results in unstable parameter estimates which makes it very ...
user28
108
votes
6
answers
42k
views
Principled way of collapsing categorical variables with many levels?
What techniques are available for collapsing (or pooling) many categories to a few, for the purpose of using them as an input (predictor) in a statistical model?
Consider a variable like college ...
- 12.8k
106
votes
10
answers
51k
views
What is a complete list of the usual assumptions for linear regression?
What are the usual assumptions for linear regression?
Do they include:
a linear relationship between the independent and dependent variable
independent errors
normal distribution of errors
...
- 1,069
106
votes
3
answers
90k
views
Solving for regression parameters in closed-form vs gradient descent
In Andrew Ng's machine learning course, he introduces linear regression and logistic regression, and shows how to fit the model parameters using gradient descent and Newton's method.
I know gradient ...
- 4,017
105
votes
9
answers
288k
views
What's the difference between correlation and simple linear regression?
In particular, I am referring to the Pearson product-moment correlation coefficient.
- 10.1k
104
votes
5
answers
161k
views
How to calculate Area Under the Curve (AUC), or the c-statistic, by hand
I am interested in calculating area under the curve (AUC), or the c-statistic, by hand for a binary logistic regression model.
For example, in the validation dataset, I have the true value for the ...
- 3,694
104
votes
1
answer
91k
views
Interpreting plot.lm()
I had a question about interpreting the graphs generated by plot(lm) in R. I was wondering if you guys could tell me how to interpret the scale-location and leverage-residual plots? Any comments ...
- 1,041
104
votes
5
answers
16k
views
Why is ANOVA taught / used as if it is a different research methodology compared to linear regression?
ANOVA is equivalent to linear regression with the use of suitable dummy variables. The conclusions remain the same irrespective of whether you use ANOVA or linear regression.
In light of their ...
user28
102
votes
1
answer
124k
views
What correlation makes a matrix singular and what are implications of singularity or near-singularity?
I am doing some calculations on different matrices (mainly in logistic regression) and I commonly get the error "Matrix is singular", where I have to go back and remove the correlated variables. My ...
- 1,441
100
votes
3
answers
51k
views
Shape of confidence interval for predicted values in linear regression
I have noticed that the confidence interval for predicted values in a linear regression tends to be narrow around the mean of the predictor and fat around the minimum and maximum values of the ...
- 14.5k
98
votes
9
answers
233k
views
How should outliers be dealt with in linear regression analysis?
Often times a statistical analyst is handed a set dataset and asked to fit a model using a technique such as linear regression. Very frequently the dataset is accompanied with a disclaimer similar to ...
- 4,424
97
votes
7
answers
191k
views
Explain the difference between multiple regression and multivariate regression, with minimal use of symbols/math
Are multiple and multivariate regression really different? What is a variate anyways?
- 10.1k
97
votes
2
answers
45k
views
When to use regularization methods for regression?
In what circumstances should one consider using regularization methods (ridge, lasso or least angles regression) instead of OLS?
In case this helps steer the discussion, my main interest is improving ...
- 5,601
92
votes
4
answers
81k
views
Why not approach classification through regression?
Some material I've seen on machine learning said that it's a bad idea to approach a classification problem through regression. But I think it's always possible to do a continuous regression to fit the ...
- 1,021
91
votes
10
answers
91k
views
Why is it possible to get significant F statistic (p<.001) but non-significant regressor t-tests?
In a multiple linear regression, why is it possible to have a highly significant F statistic (p<.001) but have very high p-values on all the regressor's t tests?
In my model, there are 10 ...
- 1,569
90
votes
6
answers
69k
views
Why is the L2 regularization equivalent to Gaussian prior?
I keep reading this and intuitively I can see this but how does one go from L2 regularization to saying that this is a Gaussian Prior analytically? Same goes for saying L1 is equivalent to a Laplacean ...
- 1,239
90
votes
5
answers
26k
views
What are modern, easily used alternatives to stepwise regression?
I have a dataset with around 30 independent variables and would like to construct a generalized linear model (GLM) to explore the relationship between them and the dependent variable.
I am aware that ...
- 4,997
89
votes
11
answers
80k
views
What are disadvantages of using the lasso for variable selection for regression?
From what I know, using lasso for variable selection handles the problem of correlated inputs. Also, since it is equivalent to Least Angle Regression, it is not slow computationally. However, many ...
- 2,208
88
votes
4
answers
34k
views
How to visualize what canonical correlation analysis does (in comparison to what principal component analysis does)?
Canonical correlation analysis (CCA) is a technique related to principal component analysis (PCA). While it is easy to teach PCA or linear regression using a scatter plot (see a few thousand examples ...
- 983
88
votes
3
answers
86k
views
What is the lasso in regression analysis?
I'm looking for a non-technical definition of the lasso and what it is used for.
- 881