Questions tagged [regression]
Techniques for analyzing the relationship between one (or more) "dependent" variables and "independent" variables.
28,283
questions
394
votes
19
answers
161k
views
What happens if the explanatory and response variables are sorted independently before regression?
Suppose we have data set $(X_i,Y_i)$ with $n$ points. We want to perform a linear regression, but first we sort the $X_i$ values and the $Y_i$ values independently of each other, forming data set $(...
- 3,621
279
votes
6
answers
44k
views
Is $R^2$ useful or dangerous?
I was skimming through some lecture notes by Cosma Shalizi (in particular, section 2.1.1 of the second lecture), and was reminded that you can get very low $R^2$ even when you have a completely linear ...
- 9,460
277
votes
2
answers
225k
views
Interpretation of R's lm() output
The help pages in R assume I know what those numbers mean, but I don't.
I'm trying to really intuitively understand every number here. I will just post the output and comment on what I found out. ...
- 4,241
213
votes
8
answers
491k
views
In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?
Am I looking for a better behaved distribution for the independent variable in question, or to reduce the effect of outliers, or something else?
- 2,281
206
votes
3
answers
165k
views
When should I use lasso vs ridge?
Say I want to estimate a large number of parameters, and I want to penalize some of them because I believe they should have little effect compared to the others. How do I decide what penalization ...
- 2,171
205
votes
5
answers
257k
views
How exactly does one “control for other variables”?
Here is the article that motivated this question: Does impatience make us fat?
I liked this article, and it nicely demonstrates the concept of “controlling for other variables” (IQ, career, income, ...
- 2,747
203
votes
10
answers
219k
views
How to deal with perfect separation in logistic regression?
If you have a variable which perfectly separates zeroes and ones in target variable, R will yield the following "perfect or quasi perfect separation" warning message:
...
- 6,971
176
votes
10
answers
207k
views
When is it ok to remove the intercept in a linear regression model?
I am running linear regression models and wondering what the conditions are for removing the intercept term.
In comparing results from two different regressions where one has the intercept and the ...
- 1,961
163
votes
4
answers
400k
views
When is R squared negative? [duplicate]
My understanding is that $R^2$ cannot be negative as it is the square of R. However I ran a simple linear regression in SPSS with a single independent variable and a dependent variable. My SPSS output ...
- 2,137
162
votes
3
answers
195k
views
How are the standard errors of coefficients calculated in a regression?
For my own understanding, I am interested in manually replicating the calculation of the standard errors of estimated coefficients as, for example, come with the output of the ...
- 1,723
150
votes
8
answers
96k
views
Why L1 norm for sparse models
I am reading books about linear regression. There are some sentences about the L1 and L2 norm. I know the formulas, but I don't understand why the L1 norm enforces sparsity in models. Can someone give ...
- 1,723
146
votes
9
answers
165k
views
Why does a time series have to be stationary?
I understand that a stationary time series is one whose mean and variance is constant over time. Can someone please explain why we have to make sure our data set is stationary before we can run ...
- 1,461
145
votes
9
answers
299k
views
What is the difference between linear regression on y with x and x with y?
The Pearson correlation coefficient of x and y is the same, whether you compute pearson(x, y) or pearson(y, x). This suggests that doing a linear regression of y given x or x given y should be the ...
- 3,183
139
votes
3
answers
273k
views
What is the difference between linear regression and logistic regression?
What is the difference between linear regression and logistic regression?
When would you use each?
- 2,913
137
votes
3
answers
47k
views
What if residuals are normally distributed, but y is not?
I've got a weird question. Assume that you have a small sample where the dependent variable that you're going to analyze with a simple linear model is highly left skewed. Thus you assume that $u$ is ...
- 5,755
128
votes
9
answers
100k
views
Numerical example to understand Expectation-Maximization
I am trying to get a good grasp on the EM algorithm, to be able to implement and use it. I spent a full day reading the theory and a paper where EM is used to track an aircraft using the position ...
- 2,533
126
votes
7
answers
144k
views
Is it necessary to scale the target value in addition to scaling features for regression analysis?
I'm building regression models. As a preprocessing step, I scale my feature values to have mean 0 and standard deviation 1. Is it necessary to normalize the target values also?
- 2,583
126
votes
4
answers
238k
views
PCA and proportion of variance explained
In general, what is meant by saying that the fraction $x$ of the variance in an analysis like PCA is explained by the first principal component? Can someone explain this intuitively but also give a ...
- 3,183
124
votes
4
answers
147k
views
What does a "closed-form solution" mean?
I have come across the term "closed-form solution" quite often. What does a closed-form solution mean? How does one determine if a close-form solution exists for a given problem? Searching online, I ...
- 2,533
123
votes
6
answers
195k
views
Difference between confidence intervals and prediction intervals
For a prediction interval in linear regression you still use $\hat{E}[Y|x] = \hat{\beta_0}+\hat{\beta}_{1}x$ to generate the interval. You also use this to generate a confidence interval of $E[Y|x_0]$....
- 1,407
120
votes
3
answers
125k
views
Does an unbalanced sample matter when doing logistic regression?
Okay, so I think I have a decent enough sample, taking into account the 20:1 rule of thumb: a fairly large sample (N=374) for a total of 7 candidate predictor variables.
My problem is the following: ...
- 1,303
119
votes
7
answers
85k
views
Why use gradient descent for linear regression, when a closed-form math solution is available?
I am taking the Machine Learning courses online and learnt about Gradient Descent for calculating the optimal values in the hypothesis.
h(x) = B0 + B1X
why we ...
- 1,293
119
votes
4
answers
51k
views
Why does the Lasso provide Variable Selection?
I've been reading Elements of Statistical Learning, and I would like to know why the Lasso provides variable selection and ridge regression doesn't.
Both methods minimize the residual sum of squares ...
- 1,402
116
votes
5
answers
21k
views
What skills are required to perform large scale statistical analyses?
Many statistical jobs ask for experience with large scale data. What are the sorts of statistical and computational skills that would be need for working with large data sets. For example, how about ...
- 2,677
116
votes
4
answers
45k
views
Why isn't Logistic Regression called Logistic Classification?
Since Logistic Regression is a statistical classification model dealing with categorical dependent variables, why isn't it called Logistic Classification? Shouldn't the "Regression" name be reserved ...
- 2,136
113
votes
18
answers
105k
views
Including the interaction but not the main effects in a model
Is it ever valid to include a two-way interaction in a model without including the main effects? What if your hypothesis is only about the interaction, do you still need to include the main effects?
- 6,570
113
votes
12
answers
59k
views
When should linear regression be called "machine learning"?
In a recent colloquium, the speaker's abstract claimed they were using machine learning. During the talk, the only thing related to machine learning was that they perform linear regression on their ...
- 1,479
113
votes
1
answer
77k
views
What is an ablation study? And is there a systematic way to perform it?
What is an ablation study? And is there a systematic way to perform it? For example, I have $n$ predictors in a linear regression which I will call as my model.
How will I perform an ablation study ...
- 8,457
106
votes
4
answers
261k
views
How does the correlation coefficient differ from regression slope?
I would have expected the correlation coefficient to be the same as a regression slope (beta), however having just compared the two, they are different. How do they differ - what different information ...
- 13.2k
106
votes
8
answers
49k
views
What is the benefit of breaking up a continuous predictor variable?
I'm wondering what the value is in taking a continuous predictor variable and breaking it up (e.g., into quintiles), before using it in a model.
It seems to me that by binning the variable we lose ...
- 1,641
105
votes
9
answers
51k
views
Is there an intuitive explanation why multicollinearity is a problem in linear regression?
The wiki discusses the problems that arise when multicollinearity is an issue in linear regression. The basic problem is multicollinearity results in unstable parameter estimates which makes it very ...
user28
104
votes
6
answers
37k
views
Principled way of collapsing categorical variables with many levels?
What techniques are available for collapsing (or pooling) many categories to a few, for the purpose of using them as an input (predictor) in a statistical model?
Consider a variable like college ...
- 12k
103
votes
9
answers
281k
views
What's the difference between correlation and simple linear regression?
In particular, I am referring to the Pearson product-moment correlation coefficient.
- 9,662
103
votes
5
answers
15k
views
Why is ANOVA taught / used as if it is a different research methodology compared to linear regression?
ANOVA is equivalent to linear regression with the use of suitable dummy variables. The conclusions remain the same irrespective of whether you use ANOVA or linear regression.
In light of their ...
user28
102
votes
2
answers
81k
views
Solving for regression parameters in closed-form vs gradient descent
In Andrew Ng's machine learning course, he introduces linear regression and logistic regression, and shows how to fit the model parameters using gradient descent and Newton's method.
I know gradient ...
- 3,605
101
votes
1
answer
87k
views
Interpreting plot.lm()
I had a question about interpreting the graphs generated by plot(lm) in R. I was wondering if you guys could tell me how to interpret the scale-location and leverage-residual plots? Any comments ...
- 1,011
99
votes
5
answers
151k
views
How to calculate Area Under the Curve (AUC), or the c-statistic, by hand
I am interested in calculating area under the curve (AUC), or the c-statistic, by hand for a binary logistic regression model.
For example, in the validation dataset, I have the true value for the ...
- 3,554
97
votes
1
answer
111k
views
What correlation makes a matrix singular and what are implications of singularity or near-singularity?
I am doing some calculations on different matrices (mainly in logistic regression) and I commonly get the error "Matrix is singular", where I have to go back and remove the correlated variables. My ...
- 1,391
97
votes
2
answers
43k
views
When to use regularization methods for regression?
In what circumstances should one consider using regularization methods (ridge, lasso or least angles regression) instead of OLS?
In case this helps steer the discussion, my main interest is improving ...
- 5,461
96
votes
10
answers
44k
views
What is a complete list of the usual assumptions for linear regression?
What are the usual assumptions for linear regression?
Do they include:
a linear relationship between the independent and dependent variable
independent errors
normal distribution of errors
...
- 969
93
votes
10
answers
214k
views
How should outliers be dealt with in linear regression analysis?
Often times a statistical analyst is handed a set dataset and asked to fit a model using a technique such as linear regression. Very frequently the dataset is accompanied with a disclaimer similar to ...
- 4,236
91
votes
3
answers
46k
views
Shape of confidence interval for predicted values in linear regression
I have noticed that the confidence interval for predicted values in an linear regression tends to be narrow around the mean of the predictor and fat around the minimum and maximum values of the ...
- 13.2k
89
votes
6
answers
63k
views
Why is the L2 regularization equivalent to Gaussian prior?
I keep reading this and intuitively I can see this but how does one go from L2 regularization to saying that this is a Gaussian Prior analytically? Same goes for saying L1 is equivalent to a Laplacean ...
- 1,209
88
votes
6
answers
176k
views
Explain the difference between multiple regression and multivariate regression, with minimal use of symbols/math
Are multiple and multivariate regression really different? What is a variate anyways?
- 9,662
88
votes
3
answers
86k
views
What is the lasso in regression analysis?
I'm looking for a non-technical definition of the lasso and what it is used for.
- 881
88
votes
5
answers
24k
views
What are modern, easily used alternatives to stepwise regression?
I have a dataset with around 30 independent variables and would like to construct a generalized linear model (GLM) to explore the relationship between them and the dependent variable.
I am aware that ...
- 4,847
88
votes
4
answers
75k
views
Why not approach classification through regression?
Some material I've seen on machine learning said that it's a bad idea to approach a classification problem through regression. But I think it's always possible to do a continuous regression to fit the ...
- 981
87
votes
9
answers
81k
views
Why is it possible to get significant F statistic (p<.001) but non-significant regressor t-tests?
In a multiple linear regression, why is it possible to have a highly significant F statistic (p<.001) but have very high p-values on all the regressor's t tests?
In my model, there are 10 ...
- 1,519
87
votes
11
answers
75k
views
What are disadvantages of using the lasso for variable selection for regression?
From what I know, using lasso for variable selection handles the problem of correlated inputs. Also, since it is equivalent to Least Angle Regression, it is not slow computationally. However, many ...
- 2,138
86
votes
7
answers
18k
views
Line of best fit does not look like a good fit. Why?
Have a look at this Excel graph:
The 'common sense' line-of-best-fit would appear be an almost vertical line straight through the center of the points (edited by hand in red). However the linear ...
- 931