Questions tagged [multiple-regression]

Regression that includes two or more non-constant independent variables.

Filter by
Sorted by
Tagged with
404 votes
7 answers

When conducting multiple regression, when should you center your predictor variables & when should you standardize them?

In some literature, I have read that a regression with multiple explanatory variables, if in different units, needed to be standardized. (Standardizing consists in subtracting the mean and dividing ...
mathieu_r's user avatar
  • 4,451
116 votes
12 answers

When should linear regression be called "machine learning"?

In a recent colloquium, the speaker's abstract claimed they were using machine learning. During the talk, the only thing related to machine learning was that they perform linear regression on their ...
jvriesem's user avatar
  • 1,509
93 votes
7 answers

Explain the difference between multiple regression and multivariate regression, with minimal use of symbols/math

Are multiple and multivariate regression really different? What is a variate anyways?
Neil McGuigan's user avatar
81 votes
2 answers

Multivariate multiple regression in R

I have 2 dependent variables (DVs) each of whose score may be influenced by the set of 7 independent variables (IVs). DVs are continuous, while the set of IVs consists of a mix of continuous and ...
Andrej's user avatar
  • 2,213
81 votes
0 answers

How can a regression be significant yet all predictors be non-significant? [duplicate]

My multiple regression analysis model has a statistically significant F value however all beta values are statistically non-significant. All the regression assumptions are met. No multicollinearity ...
Serene's user avatar
  • 811
80 votes
5 answers

How can adding a 2nd IV make the 1st IV significant?

I have what is probably a simple question, but it is baffling me right now, so I am hoping you can help me out. I have a least squares regression model, with one independent variable and one ...
EvKohl's user avatar
  • 1,190
75 votes
12 answers

What are some of the most common misconceptions about linear regression?

I'm curious, for those of you who have extensive experience collaborating with other researchers, what are some of the most common misconceptions about linear regression that you encounter? I think ...
70 votes
2 answers

Is there a difference between 'controlling for' and 'ignoring' other variables in multiple regression?

The coefficient of an explanatory variable in a multiple regression tells us the relationship of that explanatory variable with the dependent variable. All this, while 'controlling' for the other ...
Siddharth Gopi's user avatar
67 votes
3 answers

What is the effect of having correlated predictors in a multiple regression model?

I learned in my linear models class that if two predictors are correlated and both are included in a model, one will be insignificant. For example, assume the size of a house and the number of ...
Vivek Subramanian's user avatar
62 votes
5 answers

Is adjusting p-values in a multiple regression for multiple comparisons a good idea?

Lets assume you are a social science researcher/econometrician trying to find relevant predictors of demand for a service. You have 2 outcome/dependent variables describing the demand (using the ...
Mikael M's user avatar
  • 723
58 votes
9 answers

Are we exaggerating importance of model assumption and evaluation in an era when analyses are often carried out by laymen

Bottom line, the more I learn about statistics, the less I trust published papers in my field; I simply believe that researchers are not doing their statistics well enough. I'm a layman, so to speak. ...
Adam Robinsson's user avatar
58 votes
4 answers

How to visualize a fitted multiple regression model?

I am currently writing a paper with several multiple regression analyses. While visualizing univariate linear regression is easy via scatter plots, I was wondering whether there is any good way to ...
Shawn Wang's user avatar
  • 1,385
56 votes
3 answers

Multivariate linear regression vs neural network?

It seems that it is possible to get similar results to a neural network with a multivariate linear regression in some cases, and multivariate linear regression is super fast and easy. Under what ...
Hugh Perkins's user avatar
  • 4,547
54 votes
3 answers

Why is polynomial regression considered a special case of multiple linear regression?

If polynomial regression models nonlinear relationships, how can it be considered a special case of multiple linear regression? Wikipedia notes that "Although polynomial regression fits a nonlinear ...
gavinmh's user avatar
  • 1,085
54 votes
3 answers

Suppression effect in regression: definition and visual explanation/depiction

What is a suppressor variable in multiple regression and what might be the ways to display suppression effect visually (its mechanics or its evidence in results)? I'd like to invite everybody who has ...
ttnphns's user avatar
  • 56.5k
51 votes
6 answers

Choosing variables to include in a multiple linear regression model

I am currently working to build a model using a multiple linear regression. After fiddling around with my model, I am unsure how to best determine which variables to keep and which to remove. My ...
cryptic_star's user avatar
  • 1,127
51 votes
5 answers

Why do we need multivariate regression (as opposed to a bunch of univariate regressions)?

I just browsed through this wonderful book: Applied multivariate statistical analysis by Johnson and Wichern. The irony is, I am still not able to understand the motivation for using multivariate (...
KarthikS's user avatar
  • 1,156
51 votes
2 answers

Multiple regression or partial correlation coefficient? And relations between the two

I don't even know if this question makes sense, but what is the difference between multiple regression and partial correlation (apart from the obvious differences between correlation and regression, ...
user34927's user avatar
  • 229
47 votes
2 answers

How well can multiple regression really "control for" covariates?

We’re all familiar with observational studies that attempt to establish a causal link between a nonrandomized predictor X and an outcome by including every imaginable potential confounder in a ...
half-pass's user avatar
  • 3,691
44 votes
4 answers

Significance contradiction in linear regression: significant t-test for a coefficient vs non-significant overall F-statistic

I'm fitting a multiple linear regression model between 4 categorical variables (with 4 levels each) and a numerical output. My dataset has 43 observations. Regression gives me the following $p$-...
Leo's user avatar
  • 2,584
43 votes
5 answers

How to derive the least square estimator for multiple linear regression?

In the simple linear regression case $y=\beta_0+\beta_1x$, you can derive the least square estimator $\hat\beta_1=\frac{\sum(x_i-\bar x)(y_i-\bar y)}{\sum(x_i-\bar x)^2}$ such that you don't have to ...
Saber CN's user avatar
  • 809
41 votes
3 answers

How to present results of a Lasso using glmnet?

I would like to find predictors for a continuous dependent variable out of a set of 30 independent variables. I am using Lasso regression as implemented in the glmnet package in R. Here is some dummy ...
jokel's user avatar
  • 2,763
36 votes
3 answers

How to tell the difference between linear and non-linear regression models?

I was reading the following link on non linear regression SAS Non Linear. My understanding from reading the first section "Nonlinear Regression vs. Linear Regression" was that the equation below is ...
mHelpMe's user avatar
  • 657
35 votes
2 answers

What does an Added Variable Plot (Partial Regression Plot) explain in a multiple regression?

I have a model of Movies dataset and I used the regression: ...
Abhishek Choudhary's user avatar
34 votes
4 answers

Importance of predictors in multiple regression: Partial $R^2$ vs. standardized coefficients

I am wondering what the exact relationship between partial $R^2$ and coefficients in a linear model is and whether I should use only one or both to illustrate the importance and influence of factors. ...
robert's user avatar
  • 746
33 votes
1 answer

Sandwich estimator intuition

Wikipedia and the R sandwich package vignette give good information about the assumptions supporting OLS coefficient standard errors and the mathematical background of the sandwich estimators. I'm ...
Robert Kubrick's user avatar
33 votes
1 answer

Bound for Arithmetic Harmonic mean inequality for matrices?

NOTE: This question has originally been posted in MSE, but it did not generate any interest. It was first posted there, because the question itself is a pure matrix-algebra question. Nevertheless, ...
Alecos Papadopoulos's user avatar
32 votes
1 answer

Geometric interpretation of multiple correlation coefficient $R$ and coefficient of determination $R^2$

I am interested in the geometric meaning of the multiple correlation $R$ and coefficient of determination $R^2$ in the regression $y_i = \beta_1 + \beta_2 x_{2,i} + \dots + \beta_k x_{k,i} + \...
Silverfish's user avatar
  • 22.9k
32 votes
4 answers

ANOVA vs multiple linear regression? Why is ANOVA so commonly used in experimental studies?

ANOVA vs multiple linear regression? I understand that both of these methods seem to use the same statistical model. However under what circumstances should I use which method? What are the ...
florian's user avatar
  • 561
32 votes
1 answer

How to calculate the prediction interval for an OLS multiple regression?

What is the algebraic notation to calculate the prediction interval for multiple regression? It sounds silly, but I am having trouble finding a clear algebraic notation of this.
Michael's user avatar
  • 683
32 votes
3 answers

How to deal with multicollinearity when performing variable selection?

I have a dataset with 9 continuous independent variables. I'm trying to select amongst these variables to fit a model to a single percentage (dependent) variable, ...
Julie's user avatar
  • 811
32 votes
1 answer

How incorrect is a regression model when assumptions are not met?

When fitting a regression model, what happens if the assumptions of the outputs are not met, specifically: What happens if the residuals are not homoscedastic? If the residuals show an increasing or ...
SpeedBirdNine's user avatar
32 votes
4 answers

What are variable importance rankings useful for?

I have become somewhat of a nihilist when it comes to variable importance rankings (in the context of multivariate models of all kinds). Often in the course of my work, I am asked to either assist ...
Matthew Drury's user avatar
31 votes
2 answers

Transforming variables for multiple regression in R

I am trying to perform a multiple regression in R. However, my dependent variable has the following plot: Here is a scatterplot matrix with all my variables (...
zgall1's user avatar
  • 869
31 votes
3 answers

Does the order of explanatory variables matter when calculating their regression coefficients?

At first I thought the order didn’t matter, but then I read about the gram-schmidt orthogonalization process for calculating multiple regression coefficients, and now I’m having second thoughts. ...
Ryan Zotti's user avatar
  • 6,297
30 votes
3 answers

What does "all else equal" mean in multiple regression?

When we do multiple regressions and say we are looking at the average change in the $y$ variable for a change in an $x$ variable, holding all other variables constant, what values are we holding the ...
EconStats's user avatar
  • 865
30 votes
2 answers

Why are p-values misleading after performing a stepwise selection?

Let's consider for example a linear regression model. I heard that, in data mining, after performing a stepwise selection based on the AIC criterion, it is misleading to look at the p-values to test ...
John M's user avatar
  • 2,097
30 votes
1 answer

The proof of shrinking coefficients using ridge regression through "spectral decomposition"

I have understood how ridge regression shrinks coefficients towards zero geometrically. Moreover I know how to prove that in the special "Orthonormal Case," but I am confused how that works in the ...
jeza's user avatar
  • 1,989
29 votes
1 answer

Why do we do matching for causal inference vs regressing on confounders?

I'm new to the area of causal inference. From what I understand, one of the main concerns that causal inference tries to address is the effect of confounders! For the sake of reference, let's denote ...
Ehsan Sh's user avatar
  • 455
29 votes
2 answers

Interpretation of betas when there are multiple categorical variables

I understand the concept that $\hat\beta_0$ is the mean for when the categorical variable is equal to 0 (or is the reference group), giving the end interpretation that the regression coefficient is ...
Renee's user avatar
  • 425
29 votes
2 answers

In what order should you do linear regression diagnostics?

In linear regression analysis, we analyze outliers, investigate multicollinearity, test heteroscedasticty. The question is: Is there any order to apply these? I mean, do we have to analyze outliers ...
halil's user avatar
  • 295
28 votes
4 answers

How to describe or visualize a multiple linear regression model

I'm trying to fit a multiple linear regression model to my data with couple of input parameters, say 3. \begin{align} F(x) &= Ax_1 + Bx_2 + Cx_3 + d \tag{i} \\ &\text{or} \\ F(x) &= (A\...
kris's user avatar
  • 413
27 votes
3 answers

How to model this odd-shaped distribution (almost a reverse-J)

My dependent variable shown below doesn't fit any stock distribution that I know of. Linear regression produces somewhat non-normal, right-skewed residuals that relate to predicted Y in an odd way (...
rolando2's user avatar
  • 12k
27 votes
3 answers

Can (should?) regularization techniques be used in a random effects model?

By regularization techniques I'm referring to lasso, ridge regression, elastic net and the like. Consider a predictive model on health care data containing demographic and diagnosis data where length ...
RobertF's user avatar
  • 5,562
26 votes
1 answer

Hat matrix and leverages in classical multiple regression

What is Hat matrix and leverages in classical multiple regression? What are their roles? And Why do use them? Please explain them or give satisfactory book/ article references to understand them.
1190's user avatar
  • 1,040
26 votes
1 answer

How to deal with high correlation among predictors in multiple regression?

I found a reference in an article that goes like: According to Tabachnick & Fidell (1996) the independent variables with a bivariate correlation more than .70 should not be included in ...
Ander's user avatar
  • 301
26 votes
1 answer

How to calculate p-value for multivariate linear regression

Software packages that calculate regressions sometimes also return p-values. I want to understand how to calculate this p-value by hand. Here's what I think I understand: I want to calculate the ...
Mars's user avatar
  • 1,088
25 votes
2 answers

What happens when I include a squared variable in my regression?

I start with my OLS regression: $$ y = \beta _0 + \beta_1x_1+\beta_2 D + \varepsilon $$ where D is a dummy variable, the estimates become different from zero with a low p-value. I then preform a ...
seini's user avatar
  • 381
25 votes
2 answers

What is the correct way to test for significant differences between coefficients?

I'm hoping someone can help straighten out a point of confusion for me. Say I want to test whether 2 sets of regression coefficients are significantly different from each other, with the following set ...
cashoes's user avatar
  • 523
24 votes
3 answers

How to split r-squared between predictor variables in multiple regression?

I have just read a paper in which the authors carried out a multiple regression with two predictors. The overall r-squared value was 0.65. They provided a table which split the r-squared between the ...
luciano's user avatar
  • 13.9k

2 3 4 5