Questions tagged [regression]

Techniques for analyzing the relationship between one (or more) "dependent" variables and "independent" variables.

Filter by
Sorted by
Tagged with
196 votes
10 answers
209k views

How to deal with perfect separation in logistic regression?

If you have a variable which perfectly separates zeroes and ones in target variable, R will yield the following "perfect or quasi perfect separation" warning message: ...
user avatar
  • 6,701
100 votes
8 answers
45k views

What is the benefit of breaking up a continuous predictor variable?

I'm wondering what the value is in taking a continuous predictor variable and breaking it up (e.g., into quintiles), before using it in a model. It seems to me that by binning the variable we lose ...
user avatar
  • 1,531
95 votes
6 answers
32k views

Principled way of collapsing categorical variables with many levels?

What techniques are available for collapsing (or pooling) many categories to a few, for the purpose of using them as an input (predictor) in a statistical model? Consider a variable like college ...
user avatar
  • 11.5k
162 votes
9 answers
185k views

When is it ok to remove the intercept in a linear regression model?

I am running linear regression models and wondering what the conditions are for removing the intercept term. In comparing results from two different regressions where one has the intercept and the ...
user avatar
270 votes
6 answers
41k views

Is $R^2$ useful or dangerous?

I was skimming through some lecture notes by Cosma Shalizi (in particular, section 2.1.1 of the second lecture), and was reminded that you can get very low $R^2$ even when you have a completely linear ...
user avatar
  • 9,200
134 votes
3 answers
44k views

What if residuals are normally distributed, but y is not?

I've got a weird question. Assume that you have a small sample where the dependent variable that you're going to analyze with a simple linear model is highly left skewed. Thus you assume that $u$ is ...
user avatar
  • 5,625
111 votes
3 answers
114k views

Does an unbalanced sample matter when doing logistic regression?

Okay, so I think I have a decent enough sample, taking into account the 20:1 rule of thumb: a fairly large sample (N=374) for a total of 7 candidate predictor variables. My problem is the following: ...
user avatar
  • 1,213
205 votes
8 answers
455k views

In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?

Am I looking for a better behaved distribution for the independent variable in question, or to reduce the effect of outliers, or something else?
user avatar
  • 2,201
102 votes
18 answers
96k views

Including the interaction but not the main effects in a model

Is it ever valid to include a two-way interaction in a model without including the main effects? What if your hypothesis is only about the interaction, do you still need to include the main effects?
user avatar
  • 6,360
192 votes
5 answers
245k views

How exactly does one “control for other variables”?

Here is the article that motivated this question: Does impatience make us fat? I liked this article, and it nicely demonstrates the concept of “controlling for other variables” (IQ, career, income, ...
user avatar
  • 2,617
91 votes
10 answers
42k views

What is a complete list of the usual assumptions for linear regression?

What are the usual assumptions for linear regression? Do they include: a linear relationship between the independent and dependent variable independent errors normal distribution of errors ...
user avatar
  • 919
64 votes
2 answers
22k views

Is there a difference between 'controlling for' and 'ignoring' other variables in multiple regression?

The coefficient of an explanatory variable in a multiple regression tells us the relationship of that explanatory variable with the dependent variable. All this, while 'controlling' for the other ...
user avatar
24 votes
1 answer
10k views

Goodness of fit and which model to choose linear regression or Poisson

I need some advice regarding two main dilemmas in my research, which is a case study of 3 big pharmaceuticals and innovation. Number of patents per year is the dependent variable. My questions are ...
user avatar
  • 311
136 votes
9 answers
277k views

What is the difference between linear regression on y with x and x with y?

The Pearson correlation coefficient of x and y is the same, whether you compute pearson(x, y) or pearson(y, x). This suggests that doing a linear regression of y given x or x given y should be the ...
user avatar
  • 3,023
55 votes
3 answers
68k views

Interpretation of log transformed predictor and/or response

I'm wondering if it makes a difference in interpretation whether only the dependent, both the dependent and independent, or only the independent variables are log transformed. Consider the case of <...
user avatar
  • 2,677
106 votes
4 answers
39k views

Why isn't Logistic Regression called Logistic Classification?

Since Logistic Regression is a statistical classification model dealing with categorical dependent variables, why isn't it called Logistic Classification? Shouldn't the "Regression" name be reserved ...
user avatar
25 votes
1 answer
10k views

Omitted variable bias in logistic regression vs. omitted variable bias in ordinary least squares regression

I have a question about omitted variable bias in logistic and linear regression. Say I omit some variables from a linear regression model. Pretend that those omitted variables are uncorrelated with ...
user avatar
72 votes
5 answers
36k views

How can adding a 2nd IV make the 1st IV significant?

I have what is probably a simple question, but it is baffling me right now, so I am hoping you can help me out. I have a least squares regression model, with one independent variable and one ...
user avatar
  • 1,100
81 votes
0 answers
64k views

How can a regression be significant yet all predictors be non-significant? [duplicate]

My multiple regression analysis model has a statistically significant F value however all beta values are statistically non-significant. All the regression assumptions are met. No multicollinearity ...
user avatar
  • 811
18 votes
1 answer
12k views

Question on how to normalize regression coefficient

Not sure if normalize is the correct word to use here, but I will try my best to illustrate what I am trying to ask. The estimator used here is least squares. Suppose you have $y=\beta_0+\beta_1x_1$, ...
user avatar
  • 779
34 votes
2 answers
31k views

Interpretation of simple predictions to odds ratios in logistic regression

I'm somewhat new to using logistic regression, and a bit confused by a discrepancy between my interpretations of the following values which I thought would be the same: exponentiated beta values ...
user avatar
  • 827
274 votes
2 answers
214k views

Interpretation of R's lm() output

The help pages in R assume I know what those numbers mean, but I don't. I'm trying to really intuitively understand every number here. I will just post the output and comment on what I found out. ...
user avatar
34 votes
3 answers
24k views

How to tell the difference between linear and non-linear regression models?

I was reading the following link on non linear regression SAS Non Linear. My understanding from reading the first section "Nonlinear Regression vs. Linear Regression" was that the equation below is ...
user avatar
  • 637
57 votes
3 answers
36k views

Box-Cox like transformation for independent variables?

Is there a Box-Cox like transformation for independent variables? That is, a transformation that optimizes the $x$ variable so that the y~f(x) will make a more ...
user avatar
  • 20.1k
24 votes
1 answer
8k views

How do you deal with "nested" variables in a regression model?

Consider a statistical problem where you have a response variable that you want to describe conditional on an explanatory ...
user avatar
  • 94.7k
84 votes
9 answers
73k views

Why is it possible to get significant F statistic (p<.001) but non-significant regressor t-tests?

In a multiple linear regression, why is it possible to have a highly significant F statistic (p<.001) but have very high p-values on all the regressor's t tests? In my model, there are 10 ...
user avatar
  • 1,439
95 votes
5 answers
143k views

How to calculate Area Under the Curve (AUC), or the c-statistic, by hand

I am interested in calculating area under the curve (AUC), or the c-statistic, by hand for a binary logistic regression model. For example, in the validation dataset, I have the true value for the ...
user avatar
61 votes
1 answer
19k views

Logistic regression in R resulted in perfect separation (Hauck-Donner phenomenon). Now what? [duplicate]

I'm trying to predict a binary outcome using 50 continuous explanatory variables (the range of most of the variables is $-\infty$ to $\infty$). My data set has almost 24,000 rows. When I run ...
user avatar
  • 743
58 votes
5 answers
109k views

How to derive the ridge regression solution?

I am having some issues with the derivation of the solution for ridge regression. I know the regression solution without the regularization term: $$\beta = (X^TX)^{-1}X^Ty.$$ But after adding the ...
user avatar
  • 6,127
57 votes
4 answers
42k views

Regression for an outcome (ratio or fraction) between 0 and 1

I am thinking of building a model predicting a ratio $a/b$, where $a \le b$ and $a > 0$ and $b > 0$. So, the ratio would be between $0$ and $1$. I could use linear regression, although it doesn'...
user avatar
  • 2,846
56 votes
4 answers
24k views

Can a random forest be used for feature selection in multiple linear regression?

Since RF can handle non-linearity but can't provide coefficients, would it be wise to use random forest to gather the most important features and then plug those features into a multiple linear ...
user avatar
116 votes
4 answers
47k views

Why does the Lasso provide Variable Selection?

I've been reading Elements of Statistical Learning, and I would like to know why the Lasso provides variable selection and ridge regression doesn't. Both methods minimize the residual sum of squares ...
user avatar
  • 1,372
99 votes
1 answer
84k views

Interpreting plot.lm()

I had a question about interpreting the graphs generated by plot(lm) in R. I was wondering if you guys could tell me how to interpret the scale-location and leverage-residual plots? Any comments ...
user avatar
  • 991
83 votes
4 answers
28k views

How to visualize what canonical correlation analysis does (in comparison to what principal component analysis does)?

Canonical correlation analysis (CCA) is a technique related to principal component analysis (PCA). While it is easy to teach PCA or linear regression using a scatter plot (see a few thousand examples ...
user avatar
  • 933
52 votes
1 answer
35k views

How does centering the data get rid of the intercept in regression and PCA?

I keep reading about instances where we center the data (e.g., with regularization or PCA) in order to remove the intercept (as mentioned in this question). I know it's simple, but I'm having a hard ...
user avatar
  • 2,215
33 votes
1 answer
27k views

Is there a way to use the covariance matrix to find coefficients for multiple regression?

For simple linear regression, the regression coefficient is calculable directly from the variance-covariance matrix $C$, by $$ C_{d, e}\over C_{e,e} $$ where $d$ is the dependent variable's index, ...
user avatar
  • 433
13 votes
2 answers
22k views

Bayesian logit model - intuitive explanation?

I must confess that I previously haven't heard of that term in any of my classes, undergrad or grad. What does it mean for a logistic regression to be Bayesian? I'm looking for an explanation with a ...
user avatar
  • 2,176
18 votes
5 answers
2k views

Definition and delimitation of regression model

An embarrassingly simple question -- but it seems it has not been asked on Cross Validated before: What is the definition of a regression model? Also a support question, What is not a regression ...
user avatar
138 votes
3 answers
272k views

What is the difference between linear regression and logistic regression?

What is the difference between linear regression and logistic regression? When would you use each?
user avatar
  • 2,903
56 votes
5 answers
95k views

Regression when the OLS residuals are not normally distributed

There are several threads on this site discussing how to determine if the OLS residuals are asymptotically normally distributed. Another way to evaluate the normality of the residuals with R code is ...
user avatar
47 votes
3 answers
30k views

Why is polynomial regression considered a special case of multiple linear regression?

If polynomial regression models nonlinear relationships, how can it be considered a special case of multiple linear regression? Wikipedia notes that "Although polynomial regression fits a nonlinear ...
user avatar
  • 1,015
92 votes
1 answer
101k views

What correlation makes a matrix singular and what are implications of singularity or near-singularity?

I am doing some calculations on different matrices (mainly in logistic regression) and I commonly get the error "Matrix is singular", where I have to go back and remove the correlated variables. My ...
user avatar
  • 1,291
50 votes
2 answers
40k views

Regression: Transforming Variables

When transforming variables, do you have to use all of the same transformation? For example, can I pick and choose differently transformed variables, as in: Let, $x_1,x_2,x_3$ be age, length of ...
user avatar
44 votes
4 answers
53k views

Should covariates that are not statistically significant be 'kept in' when creating a model?

I have several covariates in my calculation for a model, and not all of them are statistically significant. Should I remove those that are not? This question discusses the phenomenon, but does not ...
user avatar
  • 669
120 votes
6 answers
185k views

Difference between confidence intervals and prediction intervals

For a prediction interval in linear regression you still use $\hat{E}[Y|x] = \hat{\beta_0}+\hat{\beta}_{1}x$ to generate the interval. You also use this to generate a confidence interval of $E[Y|x_0]$....
user avatar
  • 1,367
120 votes
4 answers
210k views

PCA and proportion of variance explained

In general, what is meant by saying that the fraction $x$ of the variance in an analysis like PCA is explained by the first principal component? Can someone explain this intuitively but also give a ...
user avatar
  • 3,023
65 votes
4 answers
68k views

Does it make sense to add a quadratic term but not the linear term to a model?

I have a (mixed) model in which one of my predictors should a priori only be quadratically related to the predictor (due to the experimental manipulation). Hence, I would like to add only the ...
user avatar
  • 13.4k
37 votes
2 answers
30k views

Dropping one of the columns when using one-hot encoding

My understanding is that in machine learning it can be a problem if your dataset has highly correlated features, as they effectively encode the same information. Recently someone pointed out that ...
user avatar
  • 668
157 votes
3 answers
183k views

How are the standard errors of coefficients calculated in a regression?

For my own understanding, I am interested in manually replicating the calculation of the standard errors of estimated coefficients as, for example, come with the output of the ...
user avatar
  • 1,673
60 votes
2 answers
140k views

What does having "constant variance" in a linear regression model mean?

What does having "constant variance" in the error term mean? As I see it, we have a data with one dependent variable and one independent variable. Constant variance is one of the assumptions of linear ...
user avatar
  • 777

1
2 3 4 5
75