Questions tagged [regression]
Techniques for analyzing the relationship between one (or more) "dependent" variables and "independent" variables.
4,067
questions
207
votes
10
answers
225k
views
How to deal with perfect separation in logistic regression?
If you have a variable which perfectly separates zeroes and ones in target variable, R will yield the following "perfect or quasi perfect separation" warning message:
...
- 7,101
109
votes
8
answers
52k
views
What is the benefit of breaking up a continuous predictor variable?
I'm wondering what the value is in taking a continuous predictor variable and breaking it up (e.g., into quintiles), before using it in a model.
It seems to me that by binning the variable we lose ...
- 1,691
179
votes
10
answers
217k
views
When is it ok to remove the intercept in a linear regression model?
I am running linear regression models and wondering what the conditions are for removing the intercept term.
In comparing results from two different regressions where one has the intercept and the ...
- 1,991
105
votes
6
answers
39k
views
Principled way of collapsing categorical variables with many levels?
What techniques are available for collapsing (or pooling) many categories to a few, for the purpose of using them as an input (predictor) in a statistical model?
Consider a variable like college ...
- 11.9k
282
votes
6
answers
46k
views
Is $R^2$ useful or dangerous?
I was skimming through some lecture notes by Cosma Shalizi (in particular, section 2.1.1 of the second lecture), and was reminded that you can get very low $R^2$ even when you have a completely linear ...
- 9,560
118
votes
18
answers
111k
views
Including the interaction but not the main effects in a model
Is it ever valid to include a two-way interaction in a model without including the main effects? What if your hypothesis is only about the interaction, do you still need to include the main effects?
- 6,680
122
votes
3
answers
131k
views
Does an unbalanced sample matter when doing logistic regression?
Okay, so I think I have a decent enough sample, taking into account the 20:1 rule of thumb: a fairly large sample (N=374) for a total of 7 candidate predictor variables.
My problem is the following: ...
- 1,323
210
votes
5
answers
264k
views
How exactly does one “control for other variables”?
Here is the article that motivated this question: Does impatience make us fat?
I liked this article, and it nicely demonstrates the concept of “controlling for other variables” (IQ, career, income, ...
- 2,917
138
votes
3
answers
49k
views
What if residuals are normally distributed, but y is not?
I've got a weird question. Assume that you have a small sample where the dependent variable that you're going to analyze with a simple linear model is highly left skewed. Thus you assume that $u$ is ...
- 5,805
216
votes
8
answers
513k
views
In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?
Am I looking for a better behaved distribution for the independent variable in question, or to reduce the effect of outliers, or something else?
- 2,311
98
votes
10
answers
46k
views
What is a complete list of the usual assumptions for linear regression?
What are the usual assumptions for linear regression?
Do they include:
a linear relationship between the independent and dependent variable
independent errors
normal distribution of errors
...
- 989
25
votes
1
answer
12k
views
Goodness of fit and which model to choose linear regression or Poisson
I need some advice regarding two main dilemmas in my research, which is a case study of 3 big pharmaceuticals and innovation. Number of patents per year is the dependent variable.
My questions are
...
- 331
30
votes
1
answer
13k
views
Omitted variable bias in logistic regression vs. omitted variable bias in ordinary least squares regression
I have a question about omitted variable bias in logistic and linear regression.
Say I omit some variables from a linear regression model. Pretend that those omitted variables are uncorrelated with ...
68
votes
2
answers
25k
views
Is there a difference between 'controlling for' and 'ignoring' other variables in multiple regression?
The coefficient of an explanatory variable in a multiple regression tells us the relationship of that explanatory variable with the dependent variable. All this, while 'controlling' for the other ...
- 1,515
118
votes
4
answers
48k
views
Why isn't Logistic Regression called Logistic Classification?
Since Logistic Regression is a statistical classification model dealing with categorical dependent variables, why isn't it called Logistic Classification? Shouldn't the "Regression" name be reserved ...
- 2,166
147
votes
9
answers
310k
views
What is the difference between linear regression on y with x and x with y?
The Pearson correlation coefficient of x and y is the same, whether you compute pearson(x, y) or pearson(y, x). This suggests that doing a linear regression of y given x or x given y should be the ...
- 3,233
64
votes
3
answers
76k
views
Interpretation of log transformed predictor and/or response
I'm wondering if it makes a difference in interpretation whether only the dependent, both the dependent and independent, or only the independent variables are log transformed.
Consider the case of
<...
- 3,097
79
votes
5
answers
39k
views
How can adding a 2nd IV make the 1st IV significant?
I have what is probably a simple question, but it is baffling me right now, so I am hoping you can help me out.
I have a least squares regression model, with one independent variable and one ...
- 1,170
33
votes
1
answer
14k
views
How do you deal with "nested" variables in a regression model?
Consider a statistical problem where you have a response variable that you want to describe conditional on an explanatory ...
- 116k
60
votes
4
answers
38k
views
Box-Cox like transformation for independent variables?
Is there a Box-Cox like transformation for independent variables? That is, a transformation that optimizes the $x$ variable so that the y~f(x) will make a more ...
- 21k
36
votes
3
answers
27k
views
How to tell the difference between linear and non-linear regression models?
I was reading the following link on non linear regression SAS Non Linear. My understanding from reading the first section "Nonlinear Regression vs. Linear Regression" was that the equation below is ...
- 657
19
votes
1
answer
14k
views
Question on how to normalize regression coefficient
Not sure if normalize is the correct word to use here, but I will try my best to illustrate what I am trying to ask. The estimator used here is least squares.
Suppose you have $y=\beta_0+\beta_1x_1$, ...
- 809
81
votes
0
answers
64k
views
How can a regression be significant yet all predictors be non-significant? [duplicate]
My multiple regression analysis model has a statistically significant F value however all beta values are statistically non-significant.
All the regression assumptions are met. No multicollinearity ...
- 811
36
votes
2
answers
35k
views
Interpretation of simple predictions to odds ratios in logistic regression
I'm somewhat new to using logistic regression, and a bit confused by a discrepancy between my interpretations of the following values which I thought would be the same:
exponentiated beta values
...
- 847
279
votes
2
answers
230k
views
Interpretation of R's lm() output
The help pages in R assume I know what those numbers mean, but I don't.
I'm trying to really intuitively understand every number here. I will just post the output and comment on what I found out. ...
- 4,271
88
votes
10
answers
85k
views
Why is it possible to get significant F statistic (p<.001) but non-significant regressor t-tests?
In a multiple linear regression, why is it possible to have a highly significant F statistic (p<.001) but have very high p-values on all the regressor's t tests?
In my model, there are 10 ...
- 1,529
65
votes
5
answers
140k
views
How to derive the ridge regression solution?
I am having some issues with the derivation of the solution for ridge regression.
I know the regression solution without the regularization term is given by:
$$\beta = (X^\top X)^{-1}X^\top y.$$
But ...
- 6,447
9
votes
1
answer
2k
views
Interpreting nonlinear regression $R^2$
In ordinary least squares linear regression, $R^2=1-\frac{SSRes}{SSTotal}$ is described as the “proportion of variance explained”. Does this apply to nonlinear regression, too?
- 55.2k
100
votes
5
answers
155k
views
How to calculate Area Under the Curve (AUC), or the c-statistic, by hand
I am interested in calculating area under the curve (AUC), or the c-statistic, by hand for a binary logistic regression model.
For example, in the validation dataset, I have the true value for the ...
- 3,584
66
votes
4
answers
54k
views
Regression for an outcome (ratio or fraction) between 0 and 1
I am thinking of building a model predicting a ratio $a/b$, where $a \le b$ and $a > 0$ and $b > 0$. So, the ratio would be between $0$ and $1$.
I could use linear regression, although it doesn'...
- 3,296
9
votes
2
answers
1k
views
How to motivate the definition of $R^2$ in `sklearn.metrics.r2_score`?
TLDR: What motivates the definition of $R^2$ in the Python function sklearn.metrics.r2_score?
DETAILS
The Python machine learning package ...
- 55.2k
63
votes
1
answer
21k
views
Logistic regression in R resulted in perfect separation (Hauck-Donner phenomenon). Now what? [duplicate]
I'm trying to predict a binary outcome using 50 continuous explanatory variables (the range of most of the variables is $-\infty$ to $\infty$). My data set has almost 24,000 rows. When I run ...
- 763
60
votes
4
answers
27k
views
Can a random forest be used for feature selection in multiple linear regression?
Since RF can handle non-linearity but can't provide coefficients, would it be wise to use random forest to gather the most important features and then plug those features into a multiple linear ...
54
votes
3
answers
36k
views
Why is polynomial regression considered a special case of multiple linear regression?
If polynomial regression models nonlinear relationships, how can it be considered a special case of multiple linear regression?
Wikipedia notes that "Although polynomial regression fits a nonlinear ...
- 1,085
128
votes
6
answers
201k
views
Difference between confidence intervals and prediction intervals
For a prediction interval in linear regression you still use $\hat{E}[Y|x] = \hat{\beta_0}+\hat{\beta}_{1}x$ to generate the interval. You also use this to generate a confidence interval of $E[Y|x_0]$....
- 1,457
85
votes
4
answers
32k
views
How to visualize what canonical correlation analysis does (in comparison to what principal component analysis does)?
Canonical correlation analysis (CCA) is a technique related to principal component analysis (PCA). While it is easy to teach PCA or linear regression using a scatter plot (see a few thousand examples ...
- 953
39
votes
1
answer
30k
views
Is there a way to use the covariance matrix to find coefficients for multiple regression?
For simple linear regression, the regression coefficient is calculable directly from the variance-covariance matrix $C$, by
$$
C_{d, e}\over C_{e,e}
$$
where $d$ is the dependent variable's index, ...
- 493
102
votes
1
answer
88k
views
Interpreting plot.lm()
I had a question about interpreting the graphs generated by plot(lm) in R. I was wondering if you guys could tell me how to interpret the scale-location and leverage-residual plots? Any comments ...
- 1,021
62
votes
2
answers
41k
views
How does centering the data get rid of the intercept in regression and PCA?
I keep reading about instances where we center the data (e.g., with regularization or PCA) in order to remove the intercept (as mentioned in this question). I know it's simple, but I'm having a hard ...
- 2,355
59
votes
5
answers
104k
views
Regression when the OLS residuals are not normally distributed
There are several threads on this site discussing how to determine if the OLS residuals are asymptotically normally distributed. Another way to evaluate the normality of the residuals with R code is ...
- 4,368
120
votes
4
answers
52k
views
Why does the Lasso provide Variable Selection?
I've been reading Elements of Statistical Learning, and I would like to know why the Lasso provides variable selection and ridge regression doesn't.
Both methods minimize the residual sum of squares ...
- 1,412
20
votes
5
answers
3k
views
Definition and delimitation of regression model
An embarrassingly simple question -- but it seems it has not been asked on Cross Validated before:
What is the definition of a regression model?
Also a support question,
What is not a regression ...
- 63.7k
14
votes
2
answers
24k
views
Bayesian logit model - intuitive explanation?
I must confess that I previously haven't heard of that term in any of my classes, undergrad or grad.
What does it mean for a logistic regression to be Bayesian? I'm looking for an explanation with a ...
- 2,394
139
votes
3
answers
273k
views
What is the difference between linear regression and logistic regression?
What is the difference between linear regression and logistic regression?
When would you use each?
- 2,913
167
votes
3
answers
200k
views
How are the standard errors of coefficients calculated in a regression?
For my own understanding, I am interested in manually replicating the calculation of the standard errors of estimated coefficients as, for example, come with the output of the ...
- 1,773
69
votes
4
answers
73k
views
Does it make sense to add a quadratic term but not the linear term to a model?
I have a (mixed) model in which one of my predictors should a priori only be quadratically related to the predictor (due to the experimental manipulation). Hence, I would like to add only the ...
- 14k
52
votes
1
answer
43k
views
Regression: Transforming Variables
When transforming variables, do you have to use all of the same transformation? For example, can I pick and choose differently transformed variables, as in:
Let, $x_1,x_2,x_3$ be age, length of ...
- 6,942
41
votes
2
answers
42k
views
Dropping one of the columns when using one-hot encoding
My understanding is that in machine learning it can be a problem if your dataset has highly correlated features, as they effectively encode the same information.
Recently someone pointed out that ...
- 718
129
votes
4
answers
251k
views
PCA and proportion of variance explained
In general, what is meant by saying that the fraction $x$ of the variance in an analysis like PCA is explained by the first principal component? Can someone explain this intuitively but also give a ...
- 3,233
97
votes
1
answer
116k
views
What correlation makes a matrix singular and what are implications of singularity or near-singularity?
I am doing some calculations on different matrices (mainly in logistic regression) and I commonly get the error "Matrix is singular", where I have to go back and remove the correlated variables. My ...
- 1,391