63 votes

Geometric interpretation of multiple correlation coefficient $R$ and coefficient of determination $R^2$

If there is a constant term in the model then $\mathbf{1_n}$ lies in the column space of $\mathbf{X}$ (as does $\bar{Y}\mathbf{1_n}$, which will come in useful later). The fitted $\mathbf{\hat{Y}}$ is ...
user avatar
  • 21.2k
57 votes

What does negative R-squared mean?

$R^2$ can be negative, it just means that: The model fits your data very badly You did not set an intercept To the people saying that $R^2$ is between 0 and 1, this is not the case. While a negative ...
user avatar
  • 753
44 votes

Is a high $R^2$ ever useless?

Yes. The criteria for evaluating a statistical model depend on the specific problem at hand and aren't some mechanical function of $R^2$ or statistical significance (though they matter). The relevant ...
user avatar
44 votes

R-squared is equal to 81% means what?

As a matter of fact, this last explanation is the best one: r-squared is the percentage of variation in 'Y' that is accounted for by its regression on 'X' Yes, it is quite abstract. Let's try to ...
user avatar
39 votes
Accepted

What is the distribution of $R^2$ in linear regression under the null hypothesis? Why is its mode not at zero when $k>3$?

For the specific hypothesis (that all regressor coefficients are zero, not including the constant term, which is not examined in this test) and under normality, we know (see eg Maddala 2001, p. 155, ...
user avatar
32 votes
Accepted

Showing machine learning results are statistically irrelevant

You answered yourself: I made two additional models (mean and last sample) which often match or beat the RMSE of the RF and ANN models published in the paper. The mean model just takes the mean of ...
user avatar
  • 117k
31 votes

Relationship between $R^2$ and correlation coefficient

One way of interpreting the coefficient of determination $R^{2}$ is to look at it as the Squared Pearson Correlation Coefficient between the observed values $y_{i}$ and the fitted values $\hat{y}_{i}$....
user avatar
31 votes
Accepted

Is there any difference between $r^2$ and $R^2$?

Notation on this matter seems to vary a little. $R$ is used in the context of multiple correlation and is called the "multiple correlation coefficient". It is the correlation between the ...
user avatar
  • 21.2k
29 votes

What is the mathematical relationship between R2 and MSE?

Yes, allow me to elaborate. Recall that for some outcome $y_i \in \mathbb{R}, \forall i=1,2,..,n$ we define MSE and $\textrm{R}^2$ as \begin{equation} \textrm{MSE}(y, \hat{y} ) = \frac{1}{n} \sum_{i=1}...
user avatar
28 votes

What does it mean for a linear regression to be statistically significant but has very low r squared?

It means that you can explain a small portion of the variance in the data. For instance, you can establish that a college degree impacts salaries, but at the same time it's just a small factor. There ...
user avatar
  • 56.8k
27 votes
Accepted

R-squared in quantile regression

Koenker and Machado$^{[1]}$ describe $R^1$, a local measure of goodness of fit at the particular ($\tau$) quantile. Let $V(\tau) = \min_{b}\sum \rho_\tau(y_i-x_i'b)$ Let $\hat{\beta}(\tau)$ and $\...
user avatar
  • 263k
27 votes
Accepted

Adding a linear regression predictor decreases R squared

Could it be that you have missing values in Q that are getting auto-dropped? That'd have implications on the sample, making the two regressions not comparable.
user avatar
  • 12.3k
27 votes
Accepted

Is my model any good, based on the diagnostic metric ($R^2$/ AUC/ accuracy/ RMSE etc.) value?

This answer will mostly focus on $R^2$, but most of this logic extends to other metrics such as AUC and so on. This question can almost certainly not be answered well for you by readers at ...
user avatar
  • 13.2k
26 votes
Accepted

Difference between selecting features based on "F regression" and based on $R^2$ values?

TL:DR There won't be a difference if F-regression just computes the F statistic and pick the best features. There might be a difference in the ranking, assuming <...
user avatar
  • 3,461
24 votes

Is R-squared value appropriate for comparing models?

I think the crucial part to consider in answering your question is I'm trying to identify the best model to predict the prices of automobiles because this statement implies something about why you ...
user avatar
  • 1,491
23 votes

R-squared in quantile regression

The pseudo-$R^2$ measure suggested by Koenker and Machado (1999) in JASA measures goodness of fit by comparing the sum of weighted deviations for the model of interest with the same sum from a model ...
user avatar
  • 31.6k
23 votes

What is the distribution of $R^2$ in linear regression under the null hypothesis? Why is its mode not at zero when $k>3$?

I won't rederive the $\mathrm{Beta}(\frac{k-1}{2}, \, \frac{n-k}{2})$ distribution in @Alecos's excellent answer (it's a standard result, see here for another nice discussion) but I want to fill in ...
user avatar
  • 21.2k
23 votes

What does negative R-squared mean?

Neither answer so far is entirely correct, so I will try to give my understanding of R-Squared. I have given a more detailed explanation of this on my blog post here "What is R-Squared" Sum Squared ...
user avatar
23 votes

Why is $SST=SSE + SSR$? (One variable linear regression)

Adding and subtracting gives \begin{eqnarray*} \sum_{i=1}^n (y_i-\bar y)^2&=&\sum_{i=1}^n (y_i-\hat y_i+\hat y_i-\bar y)^2\\ &=&\sum_{i=1}^n (y_i-\hat y_i)^2+2\sum_{i=1}^n(y_i-\hat y_i)...
user avatar
23 votes
Accepted

Why is R Squared not a good measure for regressions fit using LASSO?

The goal of using LASSO is obtaining a sparse representation (of a predicted quantity) in the sense of not having many covariates. Comparing models with $R^2$ tends to favor models with lots of ...
user avatar
  • 53.6k
21 votes
Accepted

What is the problem with using R-squared in time series models?

Some aspects of the issue: If somebody gives us a vector of numbers $\mathbf y$ and a conformable matrix of numbers $\mathbf X$, we do not need to know what is the relation between them to execute ...
user avatar
21 votes
Accepted

Ridge regression in R with p values and goodness of fit

Although I recommended lm.ridge to you in response to an earlier question, you might consider the glmnet package as a better way ...
user avatar
  • 66.6k
21 votes

Difference between selecting features based on "F regression" and based on $R^2$ values?

I spent some time looking through the Scikit source code in order to understand what f_regression does, and I would like to post my observations here. The original ...
user avatar
  • 526
21 votes
Accepted

Can standardized $\beta$ coefficients in linear regression be used to estimate the $R^2$?

The geometric interpretation of ordinary least squares regression provides the requisite insight. Most of what we need to know can be seen in the case of two regressors $x_1$ and $x_2$ with response $...
user avatar
  • 294k
20 votes
Accepted

Why does statsmodels.api.OLS over-report the r-squared value?

This is not technically an error in statsmodels, rather it is because statsmodels.OLS does not add the intercept/constant term ...
user avatar
19 votes

R square in mixed model with random effects

The R package MuMIn also now has a function for calculating Nakagawa and Schielzeth's r-squared for mixed models. That is the function r.squaredGLMM() and you ...
user avatar
  • 406
18 votes
Accepted

Regressions. Why a and b explains more than a+b?

The reason is flexibility. Option 1: When you regress $X_1, X_2$ on $Y$ you are allowing the coefficients to be different. In other words, your regression equation is $Y = \alpha + \beta_1X_1 + \...
user avatar
  • 7,514
18 votes

Formula for 95% confidence interval for $R^2$

You can always bootstrap it: ...
user avatar
17 votes
Accepted

Should partial $R^2$ add up to total $R^2$ in multiple regression?

No. One way to understand partial $R^2$ for a given predictor is that it equals the $R^2$ that you would get if you first regress your independent variable on all other predictors, take the residuals, ...
user avatar
  • 95.8k
17 votes
Accepted

Linear regression what does the F statistic, R squared and residual standard error tell us?

The best way to understand these terms is to do a regression calculation by hand. I wrote two closely related answers (here and here), however they may not fully help you understanding your particular ...
user avatar
  • 5,292

Only top scored, non community-wiki answers of a minimum length are eligible