Hot answers tagged multiple-regression
24
In regression, it is often recommended to center the variables so that the predictors have mean $0$. This makes it so the intercept term is interpreted as the expected value of $Y_i$ when the predictor values are set to their means. Otherwise, the intercept is interpreted as the expected value of $Y_i$ when the predictors are set to 0, which may not be a ...
18
You have come across a common belief. However, in general, you do not need to center or standardize your data for multiple regression. Different explanatory variables are almost always on different scales (i.e., measured in different units). This is not a problem; the betas are estimated such that they convert the units of each explanatory variable into ...
17
In addition to the remarks in the other answers, I'd like to point out that the scale and location of the explanatory variables does not affect the validity of the regression model in any way.
Consider the model $y=\beta_0+\beta_1x_1+\beta_2x_2+\ldots+\epsilon$.
The least squares estimators of $\beta_1, \beta_2,\ldots$ are not affected by shifting. The ...
17
The process is iterative, but there is a natural order:
You have to worry first about conditions that cause outright numerical errors. Multicollinearity is one of those, because it can produce unstable systems of equations potentially resulting in outright incorrect answers (to 16 decimal places...) Any problem here usually means you cannot proceed until ...
15
Methods of censored regression can handle data like this. They assume the residuals behave as in ordinary linear regression but have been modified so that
(Left censoring): all values smaller than a low threshold, which is independent of the data, (but can vary from one case to the other) have not been quantified; and/or
(Right censoring): all values ...
11
Note that the linearity assumption you're speaking of only says that the conditional mean of $Y_i$ given $X_i$ is a linear function. You cannot use the value of $R^2$ to test this assumption.
This is because $R^2$ is merely the squared correlation between the observed and predicted values and the value of the correlation coefficient does not uniquely ...
9
The difference is that rlm() fits models using your choice of a number of different $M$-estimators, while lm() uses ordinary least squares.
In general the $M$-estimator for a regression coefficient minimizes
$$ \sum_{i=1}^{n} \rho \left( \frac{Y_i - {\bf X}_{i} {\boldsymbol \beta}}{\sigma} \right) $$
as a function of ${\boldsymbol \beta}$, where $Y_i$ is ...
9
Synopsis
When the predictors are correlated, a quadratic term and an interaction term will carry similar information. This can cause either the quadratic model or the interaction model to be significant; but when both terms are included, because they are so similar neither may be significant. Standard diagnostics for multicollinearity, such as VIF, may ...
8
1) EDIT: Cardinal's comment below shows that the correct answer to the min $R^2$ question is $V$. Hence I'm deleting my "interesting", but ultimately incorrect, answer to that part of the OP's post.
2) The maximum $R^2$ is 1. Consider the following example, which fits your case.
x1 <- rnorm(100)
x2 <- rnorm(100)
y <- x1 + 2*x2
> ...
8
The general rule of thumb (based on stuff in Frank Harrell's book, Regression Modeling Strategies) is that if you expect to be able to detect reasonable-size effects with reasonable power, you need 10-20 observations per parameter (covariate) estimated. Harrell discusses a lot of options for "dimension reduction" (getting your number of covariates down to a ...
8
Becasue $Var(u|x)=E(u^2|x)-E^2(u|x)=E(u^2|x)-0^2=E(u^2|x)=E(u^2)=\sigma^2$, since $u$ has a zero conditional expectation. Here I used the fact that $u$ is independent of $x$. Therefore, (roughly speaking) each function of $u$ is independent of $x$ as well. So in particular $u^2$ is independent of $x$. This means that $E(u^2|x)=E(u^2)$.
7
Shifting/scaling variables will not affect their correlation with the response
To see why this is true, suppose that the correlation between $Y$ and $X$ is $\rho$. Then the correlation between $Y$ and $(X-a)/b$ is
$$ \frac{ {\rm cov}(Y,(X-a)/b) }{ {\rm SD}((X-a)/b) \cdot {\rm SD}(Y) } = \frac{ {\rm cov}(Y,X/b) }{ {\rm SD}(X/b) \cdot {\rm SD}(Y) } = ...
7
The reason is that you're testing two different hypotheses:
the Pearson correlation test is testing whether there is a non-zero correlation between the given predictor and the response variable, not taking into account the context supplied by the other predictors.
The $t$-test for the regression coefficient is testing whether that predictor has a ...
7
Adding duplicate data points will not even make the parameters estimable. Just think of it this way. It takes two points to fit a straight line and with two points you get a perfect fit. You have no information to estimate the variability in the determination of those points. Generally we assume that x is known and y is observed with some random error. ...
7
@Roland is correct that it's hard to say much without knowing what you're doing, substantively speaking. However, there are a few remarks we can still make. They fall into the categories: discovering why it's no good, making it better, and demonstrating improvement.
Diagnostics
R has good linear model diagnostics. Apply them, and read up enough to know ...
7
The multiple regression model controls for other sources of variability in the DV, whereas in the t-test, all of that variability is lumped into the error term. Thus, the t-test has lower statistical power to detect the effect. Under the assumption that the effect is real, however, the t-test would show 'significance' with a sample that was large enough.
...
7
Often the relationship between y and x is nonlinear. There are a variety of solutions. One solution is to add polynomial terms and the first one to look at is usually $x^2$. But you should first look at a scatterplot of x and y; you should also look at the residuals from the linear model without the quadratic term. But it turns out that many relationships ...
6
I prefer "solid reasons" for both centering and standardization (they exist very often). In general, they have more to do with the data set and the problem than with the data analysis method.
Very often, I prefer to center (i.e. shift the origin of the data) to other points that are physically/chemically/biologically/... more meaningful than the mean (see ...
6
My Answer: Those numbers are probably okay. What is the basis for this assertion? I simply had a look through the source material:
Fama, E. F. and French, K. R. (1993) Common risk factors in the returns on stock and bonds, Journal of Financial Economics, 33, 3–56
In particular, have a look at table 6 (on p24). You'll see that the numbers they get are ...
6
In general, you cannot interpret the coefficients from the output of a probit regression (not in any standard way, at least). You need to interpret the marginal effects of the regressors, that is, how much the (conditional) probability of the outcome variable changes when you change the value of a regressor, holding all other regressors constant at some ...
6
Number of subjects: 8803 Outcome Variable: Legal Need (Yes or No)
See the end, but I will deal with side issues first.
How to do multiple regression with ... excel?
The easiest way to do multiple regression in Excel is to install the Data Analysis Toolpack, which should have come with Excel but by default isn't installed.
I keep getting ...
6
"Dichotomous Predictor Variables", there are two ways to code dichotomous predictors: using the contrast 0,1 or the contrast 1,-1.
This is factually wrong. There is no limit to the number of ways they can be coded. Those two are merely the most common (indeed between them, almost ubiquitous), and probably the easiest to deal with.
I kind of ...
5
For two predictors, it is easy to write out the equation in algebraic form:
$R^2 = \frac{r^2_{x1,y} + r^2_{x2,y} - 2r_{x1,y}r_{x2,y}r_{x1,x2}}{1-r^2_{x1,x2}}$.
As pointed out by @gung, you also need to know the correlation between $x1$ and $x2$.
EDIT: Just a quick example (in R) to illustrate this equation:
set.seed(12873)
x1 <- rnorm(20)
x2 <- ...
5
Let $r_{1,2}$ equal the correlation between $X_1$ and $X_2$, $r_{1,Y}$ equal the correlation between $X_1$ and $Y$, and $r_{2,Y}$ the correlation between $X_2$ and $Y$. Then $R^2$ for the full model divided by $V$ equals
$$\left(\frac{1}{(1 - r_{1,2}^2)}\right) \left(1 - \frac{2 \cdot r_{1,2} \cdot r_{1,Y}}{r_{2,Y}} + \frac{U}{V}\right).$$
So $R^2$ for ...
5
I would like to suggest that this phenomenon (of a non-significant overall test despite a significant individual variable) can be understood as a kind of aggregate "masking effect" and that although it conceivably could arise from multicollinear explanatory variables, it need not do that at all. It also turns out not to be due to multiple comparison ...
5
Both @Henry, and @JDav are pointing you in the right direction (+1 to each). However, I'm very visual and it helps me if I can see how this works. In that respect, here's a quick plot in which the first variable is confounded with group membership. If the groups are ignored, the correlation coefficient is positive (as can be seen in the figure), but in a ...
5
What makes the most sense based on the source of the data?
We cannot answer this question for you, the computer cannot answer this question for you. The reason that we still need statisticians instead of just statistical programs is because of questions like this. Statistics is about more than just crunching the numbers, it is about understanding the ...
5
Linear regression can accommodate non-straight-line relationships between IVs and the DV through various transformations of variables, addition of polynomial terms and so on.
That is a model like
$y = b_0 + b_1x_1^2 + b_2x_1 + b_3x_3^5$
is a linear model. But a model such as
$y = b_0 + 2^{b_1x_1}$
is not.
If the data are really nonlinear, then the ...
5
Moran's I is a diagnostic statistic that can be used to detect spatial autocorrelation in the residuals of a regression, given that you have a weight matrix $\mathbf{w}$, with entries $w_{ij}$ representing distances between observation (residuals) $X_i$ and $X_j$. You can think of it as a spatially-weighted measure of correlation. Significance of the ...
5
From an epidemiologist's perspective:
Given exposure X and outcome Y, a key challenge is to deal with confounding of the X->Y relationship in order to make stronger inferences regarding causation. Instrumental variables (IVs) work because they are related to the X, and unrelated to Y, other than through their effects on X (i.e. X completely mediates the ...
Only top voted, non community-wiki answers of a minimum length are eligible
