Questions tagged [regression]

Techniques for analyzing the relationship between one (or more) "dependent" variables and "independent" variables.

Filter by
Sorted by
Tagged with
407 votes
17 answers

What happens if the explanatory and response variables are sorted independently before regression?

Suppose we have data set $(X_i,Y_i)$ with $n$ points. We want to perform a linear regression, but first we sort the $X_i$ values and the $Y_i$ values independently of each other, forming data set $(...
arbitrary user's user avatar
288 votes
6 answers

Is $R^2$ useful or dangerous?

I was skimming through some lecture notes by Cosma Shalizi (in particular, section 2.1.1 of the second lecture), and was reminded that you can get very low $R^2$ even when you have a completely linear ...
raegtin's user avatar
  • 9,940
281 votes
2 answers

Interpretation of R's lm() output

The help pages in R assume I know what those numbers mean, but I don't. I'm trying to really intuitively understand every number here. I will just post the output and comment on what I found out. ...
Alexander Engelhardt's user avatar
222 votes
8 answers

In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?

Am I looking for a better behaved distribution for the independent variable in question, or to reduce the effect of outliers, or something else?
d_2's user avatar
  • 2,381
216 votes
5 answers

How exactly does one “control for other variables”?

Here is the article that motivated this question: Does impatience make us fat? I liked this article, and it nicely demonstrates the concept of “controlling for other variables” (IQ, career, income, ...
JackOfAll's user avatar
  • 2,977
211 votes
9 answers

How to deal with perfect separation in logistic regression?

If you have a variable which perfectly separates zeroes and ones in target variable, R will yield the following "perfect or quasi perfect separation" warning message: ...
user333's user avatar
  • 7,221
209 votes
4 answers

When should I use lasso vs ridge?

Say I want to estimate a large number of parameters, and I want to penalize some of them because I believe they should have little effect compared to the others. How do I decide what penalization ...
Larry Wang's user avatar
  • 2,201
182 votes
10 answers

When is it ok to remove the intercept in a linear regression model?

I am running linear regression models and wondering what the conditions are for removing the intercept term. In comparing results from two different regressions where one has the intercept and the ...
analyticsPierce's user avatar
172 votes
3 answers

How are the standard errors of coefficients calculated in a regression?

For my own understanding, I am interested in manually replicating the calculation of the standard errors of estimated coefficients as, for example, come with the output of the ...
ako's user avatar
  • 1,823
170 votes
4 answers

When is R squared negative? [duplicate]

My understanding is that $R^2$ cannot be negative as it is the square of R. However I ran a simple linear regression in SPSS with a single independent variable and a dependent variable. My SPSS output ...
Anne's user avatar
  • 2,225
154 votes
8 answers

Why L1 norm for sparse models

I am reading books about linear regression. There are some sentences about the L1 and L2 norm. I know the formulas, but I don't understand why the L1 norm enforces sparsity in models. Can someone give ...
Yongwei Xing's user avatar
  • 1,773
150 votes
9 answers

Why does a time series have to be stationary?

Would like to understand primary reasons for making a data stationary? I understand that a stationary time series is one whose mean and variance is constant over time. Can someone please explain why ...
alex's user avatar
  • 1,501
148 votes
9 answers

What is the difference between linear regression on y with x and x with y?

The Pearson correlation coefficient of x and y is the same, whether you compute pearson(x, y) or pearson(y, x). This suggests that doing a linear regression of y given x or x given y should be the ...
user9097's user avatar
  • 3,263
140 votes
3 answers

What if residuals are normally distributed, but y is not?

I've got a weird question. Assume that you have a small sample where the dependent variable that you're going to analyze with a simple linear model is highly left skewed. Thus you assume that $u$ is ...
MarkDollar's user avatar
  • 5,955
139 votes
3 answers

What is the difference between linear regression and logistic regression?

What is the difference between linear regression and logistic regression? When would you use each?
B Seven's user avatar
  • 2,913
138 votes
8 answers

Is it necessary to scale the target value in addition to scaling features for regression analysis?

I'm building regression models. As a preprocessing step, I scale my feature values to have mean 0 and standard deviation 1. Is it necessary to normalize the target values also?
user2806363's user avatar
  • 2,723
132 votes
4 answers

What does a "closed-form solution" mean?

I have come across the term "closed-form solution" quite often. What does a closed-form solution mean? How does one determine if a close-form solution exists for a given problem? Searching online, I ...
arjsgh21's user avatar
  • 2,633
131 votes
6 answers

Difference between confidence intervals and prediction intervals

For a prediction interval in linear regression you still use $\hat{E}[Y|x] = \hat{\beta_0}+\hat{\beta}_{1}x$ to generate the interval. You also use this to generate a confidence interval of $E[Y|x_0]$....
question's user avatar
  • 1,495
131 votes
4 answers

PCA and proportion of variance explained

In general, what is meant by saying that the fraction $x$ of the variance in an analysis like PCA is explained by the first principal component? Can someone explain this intuitively but also give a ...
user9097's user avatar
  • 3,263
130 votes
9 answers

Numerical example to understand Expectation-Maximization

I am trying to get a good grasp on the EM algorithm, to be able to implement and use it. I spent a full day reading the theory and a paper where EM is used to track an aircraft using the position ...
arjsgh21's user avatar
  • 2,633
127 votes
18 answers

Including the interaction but not the main effects in a model

Is it ever valid to include a two-way interaction in a model without including the main effects? What if your hypothesis is only about the interaction, do you still need to include the main effects?
Glen's user avatar
  • 7,290
126 votes
1 answer

What is an ablation study? And is there a systematic way to perform it?

What is an ablation study? And is there a systematic way to perform it? For example, I have $n$ predictors in a linear regression which I will call as my model. How will I perform an ablation study ...
cgo's user avatar
  • 9,147
124 votes
7 answers

Why use gradient descent for linear regression, when a closed-form math solution is available?

I am taking the Machine Learning courses online and learnt about Gradient Descent for calculating the optimal values in the hypothesis. h(x) = B0 + B1X why we ...
Purus's user avatar
  • 1,343
124 votes
3 answers

Does an unbalanced sample matter when doing logistic regression?

Okay, so I think I have a decent enough sample, taking into account the 20:1 rule of thumb: a fairly large sample (N=374) for a total of 7 candidate predictor variables. My problem is the following: ...
Michiel's user avatar
  • 1,343
122 votes
4 answers

Why does the Lasso provide Variable Selection?

I've been reading Elements of Statistical Learning, and I would like to know why the Lasso provides variable selection and ridge regression doesn't. Both methods minimize the residual sum of squares ...
Shiwen's user avatar
  • 1,432
118 votes
4 answers

Why isn't Logistic Regression called Logistic Classification?

Since Logistic Regression is a statistical classification model dealing with categorical dependent variables, why isn't it called Logistic Classification? Shouldn't the "Regression" name be reserved ...
Ismael Ghalimi's user avatar
117 votes
5 answers

What skills are required to perform large scale statistical analyses?

Many statistical jobs ask for experience with large scale data. What are the sorts of statistical and computational skills that would be need for working with large data sets. For example, how about ...
bit-question's user avatar
  • 2,817
116 votes
8 answers

What is the benefit of breaking up a continuous predictor variable?

I'm wondering what the value is in taking a continuous predictor variable and breaking it up (e.g., into quintiles), before using it in a model. It seems to me that by binning the variable we lose ...
Tom's user avatar
  • 1,771
115 votes
12 answers

When should linear regression be called "machine learning"?

In a recent colloquium, the speaker's abstract claimed they were using machine learning. During the talk, the only thing related to machine learning was that they perform linear regression on their ...
jvriesem's user avatar
  • 1,507
114 votes
4 answers

How does the correlation coefficient differ from regression slope?

I would have expected the correlation coefficient to be the same as a regression slope (beta), however having just compared the two, they are different. How do they differ - what different information ...
luciano's user avatar
  • 14.3k
108 votes
9 answers

Is there an intuitive explanation why multicollinearity is a problem in linear regression?

The wiki discusses the problems that arise when multicollinearity is an issue in linear regression. The basic problem is multicollinearity results in unstable parameter estimates which makes it very ...
user avatar
108 votes
6 answers

Principled way of collapsing categorical variables with many levels?

What techniques are available for collapsing (or pooling) many categories to a few, for the purpose of using them as an input (predictor) in a statistical model? Consider a variable like college ...
shadowtalker's user avatar
  • 12.6k
106 votes
3 answers

Solving for regression parameters in closed-form vs gradient descent

In Andrew Ng's machine learning course, he introduces linear regression and logistic regression, and shows how to fit the model parameters using gradient descent and Newton's method. I know gradient ...
Jeff's user avatar
  • 3,957
104 votes
10 answers

What is a complete list of the usual assumptions for linear regression?

What are the usual assumptions for linear regression? Do they include: a linear relationship between the independent and dependent variable independent errors normal distribution of errors ...
tony's user avatar
  • 1,049
104 votes
1 answer

Interpreting plot.lm()

I had a question about interpreting the graphs generated by plot(lm) in R. I was wondering if you guys could tell me how to interpret the scale-location and leverage-residual plots? Any comments ...
Guest's user avatar
  • 1,041
103 votes
9 answers

What's the difference between correlation and simple linear regression?

In particular, I am referring to the Pearson product-moment correlation coefficient.
Neil McGuigan's user avatar
103 votes
5 answers

How to calculate Area Under the Curve (AUC), or the c-statistic, by hand

I am interested in calculating area under the curve (AUC), or the c-statistic, by hand for a binary logistic regression model. For example, in the validation dataset, I have the true value for the ...
Matt Reichenbach's user avatar
103 votes
5 answers

Why is ANOVA taught / used as if it is a different research methodology compared to linear regression?

ANOVA is equivalent to linear regression with the use of suitable dummy variables. The conclusions remain the same irrespective of whether you use ANOVA or linear regression. In light of their ...
user avatar
101 votes
1 answer

What correlation makes a matrix singular and what are implications of singularity or near-singularity?

I am doing some calculations on different matrices (mainly in logistic regression) and I commonly get the error "Matrix is singular", where I have to go back and remove the correlated variables. My ...
Error404's user avatar
  • 1,431
99 votes
3 answers

Shape of confidence interval for predicted values in linear regression

I have noticed that the confidence interval for predicted values in a linear regression tends to be narrow around the mean of the predictor and fat around the minimum and maximum values of the ...
luciano's user avatar
  • 14.3k
98 votes
9 answers

How should outliers be dealt with in linear regression analysis?

Often times a statistical analyst is handed a set dataset and asked to fit a model using a technique such as linear regression. Very frequently the dataset is accompanied with a disclaimer similar to ...
Sharpie's user avatar
  • 4,374
97 votes
2 answers

When to use regularization methods for regression?

In what circumstances should one consider using regularization methods (ridge, lasso or least angles regression) instead of OLS? In case this helps steer the discussion, my main interest is improving ...
NPE's user avatar
  • 5,591
95 votes
7 answers

Explain the difference between multiple regression and multivariate regression, with minimal use of symbols/math

Are multiple and multivariate regression really different? What is a variate anyways?
Neil McGuigan's user avatar
92 votes
4 answers

Why not approach classification through regression?

Some material I've seen on machine learning said that it's a bad idea to approach a classification problem through regression. But I think it's always possible to do a continuous regression to fit the ...
Strin's user avatar
  • 1,021
91 votes
10 answers

Why is it possible to get significant F statistic (p<.001) but non-significant regressor t-tests?

In a multiple linear regression, why is it possible to have a highly significant F statistic (p<.001) but have very high p-values on all the regressor's t tests? In my model, there are 10 ...
Ηλίας's user avatar
  • 1,569
91 votes
6 answers

Why is the L2 regularization equivalent to Gaussian prior?

I keep reading this and intuitively I can see this but how does one go from L2 regularization to saying that this is a Gaussian Prior analytically? Same goes for saying L1 is equivalent to a Laplacean ...
Anonymous's user avatar
  • 1,249
90 votes
5 answers

What are modern, easily used alternatives to stepwise regression?

I have a dataset with around 30 independent variables and would like to construct a generalized linear model (GLM) to explore the relationship between them and the dependent variable. I am aware that ...
fmark's user avatar
  • 4,977
89 votes
11 answers

What are disadvantages of using the lasso for variable selection for regression?

From what I know, using lasso for variable selection handles the problem of correlated inputs. Also, since it is equivalent to Least Angle Regression, it is not slow computationally. However, many ...
xuexue's user avatar
  • 2,188
88 votes
3 answers

What is the lasso in regression analysis?

I'm looking for a non-technical definition of the lasso and what it is used for.
Paul Vogt's user avatar
  • 881
87 votes
4 answers

How to visualize what canonical correlation analysis does (in comparison to what principal component analysis does)?

Canonical correlation analysis (CCA) is a technique related to principal component analysis (PCA). While it is easy to teach PCA or linear regression using a scatter plot (see a few thousand examples ...
figure's user avatar
  • 973

2 3 4 5