R - Confused on Residual Terminology

Question

• Root mean square error
• residual sum of squares
• residual standard error
• mean squared error
• test error

I thought I used to understand these terms but the more I do statistic problems the more I have gotten myself confused where I second guess myself. I would like some re-assurance & a concrete example

I can find the equations easily enough online but I am having trouble getting a 'explain like I'm 5' explanation of these terms so I can crystallize in my head the differences and how one leads to another.

If anyone can take this code below and point out how I would calculate each one of these terms I would appreciate it. R code would be great..

Using this example below:

summary(lm(mpg~hp, data=mtcars))

Show me in R code how to find:

rmse = ____
rss = ____
residual_standard_error = ______  # i know its there but need understanding
mean_squared_error = _______
test_error = ________

Bonus points for explaining like i'm 5 the differences/similarities between these. example:

rmse = squareroot(mss)

Answer 1

As requested, I illustrate using a simple regression using the mtcars data:

fit <- lm(mpg~hp, data=mtcars)
summary(fit)

Call:
lm(formula = mpg ~ hp, data = mtcars)

Residuals:
    Min      1Q  Median      3Q     Max 
-5.7121 -2.1122 -0.8854  1.5819  8.2360 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 30.09886    1.63392  18.421  < 2e-16 ***
hp          -0.06823    0.01012  -6.742 1.79e-07 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 3.863 on 30 degrees of freedom
Multiple R-squared:  0.6024,    Adjusted R-squared:  0.5892 
F-statistic: 45.46 on 1 and 30 DF,  p-value: 1.788e-07

The mean squared error (MSE) is the mean of the square of the residuals:

# Mean squared error
mse <- mean(residuals(fit)^2)
mse
[1] 13.98982

Root mean squared error (RMSE) is then the square root of MSE:

# Root mean squared error
rmse <- sqrt(mse)
rmse
[1] 3.740297

Residual sum of squares (RSS) is the sum of the squared residuals:

# Residual sum of squares
rss <- sum(residuals(fit)^2)
rss
[1] 447.6743

Residual standard error (RSE) is the square root of (RSS / degrees of freedom):

# Residual standard error
rse <- sqrt( sum(residuals(fit)^2) / fit$df.residual ) 
rse
[1] 3.862962

The same calculation, simplified because we have previously calculated rss:

sqrt(rss / fit$df.residual)
[1] 3.862962

---

The term test error in the context of regression (and other predictive analytics techniques) usually refers to calculating a test statistic on test data, distinct from your training data.

In other words, you estimate a model using a portion of your data (often an 80% sample) and then calculating the error using the hold-out sample. Again, I illustrate using mtcars, this time with an 80% sample

set.seed(42)
train <- sample.int(nrow(mtcars), 26)
train
 [1] 30 32  9 25 18 15 20  4 16 17 11 24 19  5 31 21 23  2  7  8 22 27 10 28  1 29

Estimate the model, then predict with the hold-out data:

fit <- lm(mpg~hp, data=mtcars[train, ])
pred <- predict(fit, newdata=mtcars[-train, ])
pred
 Datsun 710     Valiant  Merc 450SE  Merc 450SL Merc 450SLC   Fiat X1-9 
   24.08103    23.26331    18.15257    18.15257    18.15257    25.92090 

Combine the original data and prediction in a data frame

test <- data.frame(actual=mtcars$mpg[-train], pred)
    test$error <- with(test, pred-actual)
test
            actual     pred      error
Datsun 710    22.8 24.08103  1.2810309
Valiant       18.1 23.26331  5.1633124
Merc 450SE    16.4 18.15257  1.7525717
Merc 450SL    17.3 18.15257  0.8525717
Merc 450SLC   15.2 18.15257  2.9525717
Fiat X1-9     27.3 25.92090 -1.3791024

Now compute your test statistics in the normal way. I illustrate MSE and RMSE:

test.mse <- with(test, mean(error^2))
test.mse
[1] 7.119804

test.rmse <- sqrt(test.mse)
test.rmse
[1] 2.668296

---

Note that this answer ignores weighting of the observations.

Answer 2

The original poster asked for an "explain like I'm 5" answer. Let's say your school teacher invites you and your schoolmates to help guess the teacher's table width. Each of the 20 students in class can choose a device (ruler, scale, tape, or yardstick) and is allowed to measure the table 10 times. You all are asked to use different starting locations on the device to avoid reading the same number over and over again; the starting reading then has to be subtracted from the ending reading to finally get one width measurement (you recently learned how to do that type of math).

There were in total 200 width measurements taken by the class (20 students, 10 measurements each). The observations are handed over to the teacher who will crunch the numbers. Subtracting each student's observations from a reference value will result in another 200 numbers, called deviations. The teacher averages each student's sample separately, obtaining 20 means. Subtracting each student's observations from their individual mean will result in 200 deviations from the mean, called residuals. If the mean residual were to be calculated for each sample, you'd notice it's always zero. If instead we square each residual, average them, and finally undo the square, we obtain the standard deviation. (By the way, we call that last calculation bit the square root (think of finding the base or side of a given square), so the whole operation is often called root-mean-square, for short; the standard deviation of observations equals the root-mean-square of residuals.)

But the teacher already knew the true table width, based on how it was designed and built and checked in the factory. So another 200 numbers, called errors, can be calculated as the deviation of observations with respect to the true width. A mean error can be calculated for each student sample. Likewise, 20 standard deviation of the error, or standard error, can be calculated for the observations. More 20 root-mean-square error values can be calculated as well. The three sets of 20 values are related as sqrt(me^2 + se^2) = rmse, in order of appearance. Based on rmse, the teacher can judge whose student provided the best estimate for the table width. Furthermore, by looking separatelly at the 20 mean errors and 20 standard error values, the teacher can instruct each student how to improve their readings.

As a check, the teacher subtracted each error from their respective mean error, resulting in yet another 200 numbers, which we'll call residual errors (that's not often done). As above, mean residual error is zero, so the standard deviation of residual errors or standard residual error is the same as the standard error, and in fact, so is the root-mean-square residual error, too. (See below for details.)

Now here is something of interest to the teacher. We can compare each student mean with the rest of the class (20 means total). Just like we defined before these point values:

• m: mean (of the observations),
• s: standard deviation (of the observations)
• me: mean error (of the observations)
• se: standard error (of the observations)
• rmse: root-mean-square error (of the observations)

we can also define now:

• mm: mean of the means
• sm: standard deviation of the mean
• mem: mean error of the mean
• sem: standard error of the mean
• rmsem: root-mean-square error of the mean

Only if the class of students is said to be unbiased, i.e., if mem = 0, then sem = sm = rmsem; i.e., standard error of the mean, standard deviation of the mean, and root-mean-square error the mean may be the same provided the mean error of the means is zero.

If we had taken only one sample, i.e., if there were only one student in class, the standard deviation of the observations (s) could be used to estimate the standard deviation of the mean (sm), as sm^2~s^2/n, where n=10 is the sample size (the number of readings per student). The two will agree better as the sample size grows (n=10,11,...; more readings per student) and the number of samples grows (n'=20,21,...; more students in class). (A caveat: an unqualified "standard error" more often refers to the standard error of the mean, not the standard error of the observations.)

Here are some details of the calculations involved. The true value is denoted t.

Set-to-point operations:

• mean: MEAN(X)
• root-mean-square: RMS(X)
• standard deviation: SD(X) = RMS(X-MEAN(X))

INTRA-SAMPLE SETS:

• observations (given), X = {x_i}, i = 1, 2, ..., n=10.
• deviations: difference of a set with respect to a fixed point.
• residuals: deviation of observations from their mean, R=X-m.
• errors: deviation of observations from the true value, E=X-t.
• residual errors: deviation of errors from their mean, RE=E-MEAN(E)

INTRA-SAMPLE POINTS (see table 1):

• m: mean (of the observations),
• s: standard deviation (of the observations)
• me: mean error (of the observations)
• se: standard error of the observations
• rmse: root-mean-square error (of the observations)

INTER-SAMPLE (ENSEMBLE) SETS:

• means, M = {m_j}, j = 1, 2, ..., n'=20.
• residuals of the mean: deviation of the means from their mean, RM=M-mm.
• errors of the mean: deviation of the means from the "truth", EM=M-t.
• residual errors of the mean: deviation of errors of the mean from their mean, REM=EM-MEAN(EM)

INTER-SAMPLE (ENSEMBLE) POINTS (see table 2):

• mm: mean of the means
• sm: standard deviation of the mean
• mem: mean error of the mean
• sem: standard error (of the mean)
• rmsem: root-mean-square error of the mean

Answer 3

I also feel all the terms are very confusing. I strongly feel it is necessary to explain why we have these many metrics.

Here is my note on SSE and RMSE:

First metric: Sum of Squared Errors (SSE). Other names, Residual Sum of Squares (RSS), Sum of Squared Residuals (SSR).

If we are in optimization community, SSE is widely used. It is because it is the objective in optimization, where the optimization is

$$\underset{\beta}{\text{minimize}} ~ \|X\beta-y\|^2$$

And the residual/error term is $e=X\beta-y$, and $\|e\|^2=e^Te$, which is called Sum of Squared Errors (SSE).

Second Metric: Root-mean-square error (RMSE). Other names, root-mean-squares deviation.

RMSE is

$$ \|\frac 1 {\sqrt N} ({X\beta-y}) \|= \sqrt{\frac 1 N e^Te} $$

where $N$ is number of data points.

Here is why we have this metric in addition to SSE we talked above. The advantage of RMSE metric is that it is more "normalized". Specifically, SSE will be depending on the amount of the data. The MSE would not depend on the amount of the data, but the RMSE also expresses the error in the same units as $y$.