Notation
$y_i$ is observation $i$ of some response variable $Y$.
$\hat{y}_i$ is the value of $y_i$ predicted by the regression.
$\bar{y}$ is the average of all observations of the response variable.
$$ y_i-\bar{y} = (y_i - \hat{y_i} + \hat{y_i} - \bar{y}) = (y_i - \hat{y_i}) + (\hat{y_i} - \bar{y}) $$
$$( y_i-\bar{y})^2 = \Big[ (y_i - \hat{y_i}) + (\hat{y_i} - \bar{y}) \Big]^2 = (y_i - \hat{y_i})^2 + (\hat{y_i} - \bar{y})^2 + 2(y_i - \hat{y_i})(\hat{y_i} - \bar{y}) $$
$$ \sum_i ( y_i-\bar{y})^2 = \sum_i(y_i - \hat{y_i})^2 + \sum_i(\hat{y_i} - \bar{y})^2 + 2\sum_i\Big[ (y_i - \hat{y_i})(\hat{y_i} - \bar{y}) \Big]$$
$$ = SSRes + SSReg + Other $$
When $Other = 0$, as we have in linear regression, then $SSRes$ is a perfectly reasonable measure of what strikes me as the real value of interest: $SSReg$. As one decreases, the other increases, so we can get a strong model fit (high $SSReg$) by minimizing $SSRes$.
However, $Other \ne 0$ in general, such as in nonlinear regressions. A popular nonlinear regression these days is a neural network. While neural nets may be most used for classification problems, they are perfectly reasonable to use in regression problems. In neural network regressions, I have seen $MSE$ as the loss function. For instance, sklearn's MLPRegressor
uses SSRes as the loss function (same $argmin$ as $MSE$).
Minimizing $SSRes$ misses the $Other$ term! The $SSRes$ could be very small, yet there could be a major contribution from the $Other$ term that shows the regression model not to be good.
I've tried it out in Python, using some code I found on Stack Overflow for MLPRegressor
. That $Other$ term definitely doesn't drop to zero.
from sklearn.neural_network import MLPRegressor
import numpy as np
import random
random.seed(2019)
x = np.arange(0.0, 1, 0.001).reshape(-1, 1)
y = np.sin(2 * np.pi * x).ravel()
nn = MLPRegressor(hidden_layer_sizes=(100,), activation='relu')
n = nn.fit(x, y)
train_y_pred = n.predict(x)
Other = (train_y_pred - np.mean(y) ) * (y - train_y_pred)
sum(Other)
Questions
What is the reason for using $SSRes$ or $MSE$ loss when there is that $Other$ term?
This might be more philosophy or perhaps not so different from the first question, but am I off-base to claim that $SSReg$ is the real value of interest and that we use $SSRes$ as a proxy because we're used to minimizing loss rather than maximizing gain?
Code for linear regression:
import numpy as np
from sklearn.linear_model import LinearRegression
import random
random.seed(2019)
X = np.random.normal(10,1,100).reshape(-1, 1)
X = np.sin(X).reshape(-1, 1)
e = np.random.normal(0,0.25,100).reshape(-1, 1)
y = X + e
y.reshape(-1, 1)
reg = LinearRegression().fit(X, y)
y_pred = reg.predict(X)
resid = y - y_pred
Other = (y_pred - np.mean(y) )* (y - y_pred)
sum(Other)