11
$\begingroup$

Sorry if this is really elementary stuff, but I'm a stats noob trying to wrap my head around this.

Intuitively, I'm imagining that a model with this information alone would be able to accurately predict the outcome, on average, in 10 tries. Is this one way of thinking about it or am I completely wrong?

$\endgroup$
14
$\begingroup$

We don't know.

Note that "variance explained" is a percentage, i.e., a relative measure. Your total sum of squares might be 100 and the residual sum of squares 90, yielding an $R^2$ of

$$ 1-\frac{SS_{\text{res}}}{SS_{\text{tot}}}=10\%,$$ since we have explained 10% of the variance. But we might also have a total SS of 1000 and a residual SS of 900, again for $R^2=10\%$.

And how well we can predict (in terms of mean squared error, MSE) depends on the residual sum of squares. So you see that the MSE could be any number.

The only thing we know is that by using the predictor, we can reduce the MSE by 10%. (And of course, $R^2$ is only an estimate based on the observations in a particular model. So the relationship may not hold for new data, especially if the model was data dredged.)

$\endgroup$
1
6
$\begingroup$

I love the answer by @StephanKolassa but wanted to emphasise a few points with a practical illustration — from a beginner's perspective, how we measure the accuracy in a regression model, and what Stephan means when he talks about the relative nature of $R^2$. People often like to think of predictions as basically "right" or "wrong", and there's a hint of that in your question as to whether we are "able to accurately predict the outcome, on average, in 10 tries". But "we get the answer right M times out of N" or "P% of our predictions are accurate" aren't really helpful or meaningful when you're performing a regression analysis to predict a continuous variable. I'm going to use the example of estimating people's height in centimetres, $y$, based on some personal data $x$ (this answer covers both simple and multiple regression, so interpret that as "several predictor variables $x_1, x_2, x_3, \dots x_k$" when that makes more sense).

If your prediction is $\hat y$ and the true value is $y$, you care about questions like "did I make an under- or overestimate, and how much by?" So the residual $y - \hat y$, also measured in centimetres, is a useful quantity; if it's positive you underestimated, negative you overestimated, and near to zero means you're about right. Big negative or positive values are both bad, in different ways, so one measure of the "overall badness" of your model's predictions is to square the residuals of your observations and add them up, $\sum (y_i - \hat y_i)^2$, to get the residual sum of squares, $SS_{res}$. Note this would be in square centimetres. That doesn't tell you how badly wrong your individual predictions are likely to be. One suitable metric is the mean squared error (MSE) found by sharing out $SS_{res}$ between your data points (with a small degrees of freedom adjustment that won't make much difference in large samples). You may also be interested in its square root, often called the standard error of the regression or residual standard error, and sometimes abbreviated SER. Despite its name, you're better to think of this like the standard deviation of the residuals (strictly speaking, it's an estimate of the standard deviation $\sigma$ of the regression's error or disturbance term $\varepsilon$). It shows, roughly speaking, "how much am I usually out by?" in the units of your original data, i.e. centimetres: intuitively you'd expect most of your predictions to be within two or three SERs of the true values, and indeed many to be within one SER.

However, you probably care how accurate a particular prediction is likely to be, and we can "personalise" this uncertainty based on the values of the predictors. If the predictors $x$ are close to their average values, your prediction $\hat y$ has less uncertainty (i.e. is likely to be nearer the true value of $y$) than for a data point with $x$ far from the average — there's additional uncertainty when you're extrapolating. Why? Your prediction $\hat y$ was based on estimating the mean value $\mathbb{E}(y|x)$ for the given $x$ values. Prediction errors occur partly because observed values of $y$ have random variation $\varepsilon$ above/below $\mathbb{E}(y|x)$ that your model cannot possibly account for. We generally assume $\varepsilon$ has a normal distribution and its variance $\sigma^2$ is the same at all values of $x$ (homoskedasticity), so this source of prediction error is equally bad everywhere. But another source of error is that your estimate of $\mathbb{E}(y|x)$ depends on estimated regression coefficients (slopes and intercept) and particularly in small samples, you'll have incurred some estimation error. When you're close to the mean value of your predictor $x$, your error in fitting the line isn't too bad, but as you extend the line further and further out, even a small error in your estimated slope $\hat \beta$ can result in your prediction $\hat y$ being badly off. Try holding a ruler to your regression line and wiggle it around slightly to represent uncertainty in its estimation; what other lines would look plausible if your data were slightly different, e.g. a couple of points deleted? Your predictions for the observed data will change a bit, but imagine extending your ruler out to make a prediction for a point a metre to your right — small wiggles of your ruler cause that prediction to jump up and down with great uncertainty. (You may notice the wigglability of your ruler seems to depend on $R^2$. If it's near one and the data points fit very close to a straight line, you can't deviate the ruler very far before it looks implausible. If it's near zero and the data points are scattered about all over the place, it feels like you can fit the ruler almost anywhere and the uncertainty in your predictions is much greater. We'll come back to this later.)

More scientific than ruler-wiggling, your statistical software can add confidence intervals around your regression line to show the uncertainty in $\mathbb{E}(y|x)$, i.e. where the line should be fit, and prediction intervals, also known as forecast intervals, to show how the likely error in your predictions changes as $x$ varies. For example, we expect only about 1 in 20 data points to lie outside a 95% prediction interval. In the simulated data set below, 3 out of 100 points lie clearly outside the 95% prediction limits (dashed red lines) and another 3 lie close to the boundary. You'll see these intervals trumpet outwards to show greater uncertainty as you move left or right from the mean $\bar x$: it's more visible for the confidence interval (shaded region), but if you count how many squares wide it is, the prediction interval widens too. So although the SER gives you an overview of how far out your predictions tend to be, you need to tweak it to find the uncertainty of a specific prediction with given $x$ values. But the SER is still useful, because these "personalised" uncertainty estimates scale with the SER — if you can halve the SER, you'll halve the width of your prediction intervals too. (The width of confidence intervals also scales with SER, but unlike prediction intervals, is roughly inversely proportional to the square root of the sample size. By collecting sufficient data you can make the uncertainty in your estimate of $\mathbb{E}(y|x)$ very small even if your model's fit is quite poor, i.e. $R^2$ is low, although it's easier — requires less data — if the fit is good. The uncertainty in your predictions would then be dominated by the random variation $\varepsilon$ about this mean.)

100 data points with 95% prediction interval

We've discussed what the "accuracy" or "badness" of our predictions might mean and how we might measure it, and sources of error in those predictions. Now let's consider what $R^2 = 0.9$ would tell us about them. In a sense, the answer is "nothing"! If you were interested in how many units (i.e. centimetres) wrong your height predictions might be, then you need to look at the SER (and for a specific data point, the prediction interval that depends on the SER and the $x$ values). You can have two regressions with identical $R^2$ but very different SERs. This is obvious since $R^2$ is a dimensionless quantity and doesn't scale with your data. Suppose you decided to change all your height data $y$ into metres and re-run the regression. You'll get exactly the same $R^2$ but all your predictions $\hat y$, the SER, and the prediction intervals are, numerically, a hundred times smaller/narrower. And because of the squaring effect, the residual sum of squares, $SS_{res}$, will be 10,000 times smaller! However, it's clear that your model and its accompanying predictions haven't got 100 or 10,000 times better. In that sense, it's useful that the $R^2$ is dimensionless and remains at 0.9. The $R^2$, SER and $SS_{res}$ are telling you things on fundamentally different scales. A give-away for this issue is that they were measured in different units, which motivated my choice of "height" for $y$.

$R^2$ is a relative measure because it's comparing the situation of using your regression model to a situation where you didn't make any use of $x$ to predict $y$. How can I predict someone's height if I know nothing about them?! The obvious answer is "just go with the average height". It's easy to prove that, using the ordinary least squares (OLS) approach, the optimal prediction is indeed the mean $\bar y$. This is equivalent to fitting a regression model, called the null model, which contains only the intercept term. You can tell your statistical software to do this, and check that it estimates the constant term as $\bar y$. Just how bad are the null model's predictions? Taking a similar approach to before, we can sum the squares of their errors, $\sum (y_i - \bar y_i)^2$, to obtain the total sum of squares, $SS_{tot}$. Intuitively, because we know the null model's predictions aren't very good — we were hampered by not being able to use the $x$ values to help us make personalised predictions and had to use a "one size fits all" approach — we'd expect $SS_{tot}$ to be a larger number than $SS_{res}$, since the latter is the sum of squared errors for predictions we'd expect to be superior. In fact, since by definition OLS regression minimises this sum of squares, $SS_{res}$ cannot be larger than $SS_{tot}$: even in the worst case, it's always an option to estimate the slope coefficients as zero, so the fitted model duplicates the null model and performs just as badly. Since the fitted model can't fare worse than the null model, it makes sense to treat the null model's performance as a baseline. When people define $R^2$ as the "proportion of total variation explained by your model", the "total variation" they're referring to is $SS_{tot}$. For $R^2 = 0.9$ it can be useful to flip things around and consider the "10% of total variation that remains unexplained by your model". The stuff your model still didn't get right, even though its predictions make use of the $x$ data, is represented by $SS_{res}$, and in this case we're saying it's 10% of $SS_{tot}$. Flipping the $R^2$ formula around, we have:

$$1 - R^2 = \frac{SS_{res}}{SS_{tot}} $$

You can think of $R^2 = 0.9$ as saying the "badness" of your predictions (as measured by the sum of the squares of their errors) is a mere 10% of the "badness" they'd have if you had just predicted everyone's height to be the mean. In other words, a 90% improvement on the null model. There's a theoretical upper limit of $R^2 = 1$ if your model's prediction make no errors and $SS_{res} = 0$. However, even a correctly-specified regression model which includes all relevant predictors will likely have some purely random variation $\varepsilon$ that you'll never be able to predict. Depending on how large the variance $\sigma^2$ of this random error is compared to the variance of $y$ (we get a formula for this later), you may max out well before 100% of variation can be explained. When predicting height, you can't legislate for measurement error; when predicting someone's wealth in ten years, you can't tell who'll win the lottery (literally or metaphorically). So while $R^2$ tells you proportionately how far down the road from $SS_{tot}$ (the worst) to zero (perfection) your model has been able to squeeze the $SS_{res}$ of its predictions by making use of the $x$ data, it doesn't tell you how far along that route is realistically attainable. In social sciences, when dealing with systems that are hard to model and include lots of random variation, $R^2 = 0.3$ can still be a "good" result, even though the residual sum of squares of your predictions is only 30% better than the null model.

How does $R^2$ relate to the accuracy/uncertainty of individual predictions? If we use MSE as our metric, it scales (compared to the null model) in a similar way to the sum of squared errors, except that due to its degrees-of-freedom correction we need to use the adjusted R-squared, $R^2_{adj}$. You can think of this as a "penalised" version of $R^2$, using a degrees-of-freedom adjustment to slightly reduce $R^2$ for every extra parameter (e.g. additional variable in a multiple regression) included in the model. The adjustment is only small if the sample size is large. Now, let's return to the scenario where we tried predicting height $y$ without use of any predictors, and resorted to the null model where we predict $\bar y$ for each individual. The MSE of the null model depends on how the true $y$ values are distributed about their mean $\bar y$, so is just the (sample) variance of the observed heights, $S^2_y$. We hope that by making use of the $x$ data, the MSE of our fitted model will be lower, although a certain proportion of variation will remain unexplained:

$$1 - R^2_{adj} = \frac{MSE}{S^2_y} $$

So another way of thinking about the "unexplained proportion" is that it's roughly the variance $\sigma^2$ of $\varepsilon$, divided by the variance of $y$ — i.e. what proportion of the variance of $y$ is due to the random error your model cannot capture? You may find that more intuitive than "sums of squares". By square-rooting and rearranging this equation, we find a handy formula for the standard error of the regression:

$$SER = S_y \sqrt{1 - R^2_{adj}} $$

So the SER is directly proportional to the sample standard deviation of $y$; this explains why the SER was scaled down by a factor of 100 when we switched the $y$ data from centimetres to metres. Note that $S_y$ depends only on the data, not our model. We could try to improve our model, by including or excluding predictor variables, or changing the functional form (e.g. including quadratic or cross terms) and this only affects the SER via the $\sqrt{1 - R^2_{adj}}$ factor. Suppose a new model increases $R^2_{adj}$ from 0.8 to 0.9; this results in a halving of the "proportion unexplained" from 0.2 to 0.1, but only reduces the SER by a factor of $\sqrt {0.5} \approx 0.707$. The width of our prediction intervals will narrow, but only by about 30%. Halving the prediction intervals would require the "proportion unexplained" to be reduced by a factor of 4, a much trickier feat of modelling. Drawing a larger sample (from the same population) doesn't help much, since the SER is just an estimate of $\sigma$, the standard deviation of $\varepsilon$, so we don't expect it to fall as $n$ increases. Larger $n$ does reduce the sampling error of our coefficient estimates, so that confidence intervals around the regression line become very narrow. The consequent reduction in extrapolation error means the prediction intervals no longer trumpet out, and instead their upper and lower limits run parallel to the regression line (illustrated in the graph below where the sample size has been increased to 1000). But even if we are increasingly confident of the true value of $\mathbb{E}(y|x)$ anywhere along our regression line, any observed value of $y$ includes an error term $\varepsilon$ which means it could easily lie $1.96\sigma$ above or below its predicted value. This uncertainty persists even if we take $n \to \infty$, and we can only narrow it down if we can chip away at the proportion of unexplained variance by increasing $R^2_{adj}$. The effects of mismeasured variables, omitted variables, and other sins all get lumped in to our residuals, and we may be able to strip some of that out by getting our hands on more accurately measured data, or correcting the specification of our model. But if what's left of $\varepsilon$ was fundamentally random and unpredictable, there's nothing we can do to improve our predictive accuracy further.

Prediction limits with n = 1000 run parallel to regression line

I hope this makes it clear why $R^2$ and $R^2_{adj}$ are related to the accuracy of your predictions, but can't tell you the likely magnitude of such errors. We should be somewhat cautious when we quantify uncertainty in predictions, e.g. by using prediction intervals, even if our $R^2$ looks "good". Much depends on the individuals you're making predictions for being drawn from the same distribution as the sample you fitted your model to. When making a prediction from $x$ values that are unusual for your sample, it's nice that prediction intervals account for the additional uncertainty due to extrapolating with a potentially erroneous slope — but the more fundamental problem is you cannot know if your model really applies in this region, even if a high $R^2$ shows it fitted well in the region you did have data. A classic example is how springs initially extend linearly with load, and a regression model will have excellent fit to data that only used loads well below the spring's elastic limit. It's physically clear the model would give inaccurate predictions when that limit is approached or exceeded, but without data there's no statistical way to determine how bad the errors might be. $R^2$ can also be misleadingly high if many irrelevant variables have been included in the regression. Including a variable for which the true regression slope is zero can never reduce $R^2$ and in practice almost always increases it, but obviously such an "improvement" in $R^2$ is not accompanied by improved accuracy of predictions. You get a bit of protection because prediction intervals depend on $R^2_{adj}$ and aren't so easily fooled by this kind of overfitting: due to the degrees-of-freedom penalty, the adjusted $R^2$ can fall when irrelevant predictors are included in the model, and if this happens then the SER increases and the prediction intervals widen. But it's better not to overfit your models in the first place :-) ... an obsession with making "better" models, with ever-improved $R-squared$, can be dangerous, and you certainly shouldn't expect ever-improved predictive accuracy from doing so.


Technical note

In simple linear regression, the formulas for the prediction and confidence intervals for a data point $(x_0, y_0)$ depend on the critical $t$-statistic with the residual degrees of freedom, $t_{res.df}^{crit}$ (pick from your statistical tables, depending on whether you want e.g. 95% or 99% coverage etc), the variance $S_x^2$ of $x$, sample size $n$, and the standard error of the regression:

$$\text{Confidence limits} = \hat y \pm t_{res.df}^{crit} \times SER \times \sqrt{\frac{1}{n} + \frac{(x_0 - \bar x)^2}{(n-1) S_x^2}} $$

$$\text{Prediction limits} = \hat y \pm t_{res.df}^{crit} \times SER \times \sqrt{\color{red}{1} + \frac{1}{n} + \frac{(x_0 - \bar x)^2}{(n-1) S_x^2}} $$

$R^2$ enters into the interval widths via the SER factor: messy-looking data with a low $R^2$ produces a higher SER and wider intervals, i.e. greater uncertainty in both your estimated conditional mean $\mathbb{E}(y|x)$ and consequent potential for error in your predictions. The $(x_0 - \bar x)^2$ term is why the intervals widen as you move away from $\bar x$. The red $\color{red}{1}$ in the prediction limit formula makes a big difference. As you increase the sample size, $n \to \infty$, the confidence limits tend to $\hat y \pm 0$ even if $R^2$ is poor and the SER is high, whereas the prediction limits tend to $\hat y \pm z^{crit} \times SER$. This is basically the $\pm 1.96 \sigma$ prediction error you can't shake off.

The story for multiple regression is similar. Let $\mathbf{x}_0$ be the row vector of predictors for the data point you want to make a prediction for. It should look like a row in the design matrix $\mathbf{X}$ (i.e. a 1 for the intercept term, then subsequent cells occupied by the variables). Then the formulas become:

$$\text{Confidence limits} = \hat y \pm t_{res.df}^{crit} \times SER \times \sqrt{\mathbf{x}_0 (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{x}_0^T} $$

$$\text{Prediction limits} = \hat y \pm t_{res.df}^{crit} \times SER \times \sqrt{\color{red}{1} + \mathbf{x}_0 (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{x}_0^T} $$

Since $SER = \sqrt MSE$, these formulas are often written with the mean squared error written inside the square root, rather than the standard error of the regression written as a factor outside it.

$\endgroup$
1
$\begingroup$

This may be off-topic, but some metrics, like the Area Under the ROC Curve (AUC) do provide one with the kind of information you describe in your post.

For example, if I measure the height of 100 men and 78 women and compute an AUC of 0.7 with men being taller on average, then this implies that if I tell you the height of a new person you can say with 70% probability of being correct the gender of the new person.

$\endgroup$
3
  • 1
    $\begingroup$ How do you interpret that given that the guess rate is 50%? $\endgroup$ – jsavn May 10 at 8:57
  • $\begingroup$ @jsavn I'm not sure i understand your question. Are you asking whether 70% guess rate is statistically better than a 50% guess rate? My preferred method for computing statistics on AUC, is to shuffle the labels and recompute the AUC many time (5000 or so) and then compare the data AUC to the shuffle distribution. $\endgroup$ – jerlich May 12 at 2:17
  • $\begingroup$ Yes, I think that's what I was getting at, and the test you suggest makes sense intuitively. $\endgroup$ – jsavn May 13 at 9:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.