4
$\begingroup$

One fact about linear regression is that the predictions and residuals are orthogonal. In other words:

$$ \sum_{i=1}^N \hat{y}_i (y_i-\hat{y}_i) = 0 $$

In nonlinear regression, this fails to be the case.$^{\dagger}$

That does not make sense to me. I've simulated what happens and confirmed that the nonlinear regression lacks orthogonal residuals and predictions, but it still is not intuitive, particularly for an approach with a neural network.

enter image description here

The neural network above does some feature engineering to find three features to feed into a linear regression, but the neural network is a nonlinear regression, since there would be a ReLU activation function in the hidden layer acting on the red, blue, and yellow parameters. However, if I got lucky and guessed the features in the hidden layer, then I could call my regression linear.

Those seem like the same model to me, yet one would be a linear regression with orthogonal residuals and predictions and one would be a nonlinear regression that lacks orthogonal residuals and predictions.

What gives?

A few links to threads that discuss this lack of orthogonality:

regression - does R2 only apply to measure linear regression performance?

Is R-squared truly an invalid metric for non-linear models?

$^{\dagger}$I am not sure if it can hold for a nonlinear regression, but it at least does not have to hold for a nonlinear regression.

Improve this question
$\endgroup$
1
  • $\begingroup$ orthogonality is a fundamentally linear notion: it's the inner product between the residuals and predictions (and inner products are the same thing as linear functions when both exist). I don't see any reason why a general nonlinear model would impose orthogonality necessarily, though if the set of linear models lies within the span of the nonlinear model, it is possible that it could if this leads to it achieving minimal MSE. $\endgroup$
    – John Madden
    Commented May 31, 2023 at 0:58

1 Answer 1

Reset to default
1
$\begingroup$

It appears this is due to the instability of the hidden features and the dynamics of gradient descent with these shifting features.

I did some experiments fitting with a small network with 3 hidden neurons, fitting one dimensional nonlinear data.

If you freeze the first layer, and only run gradient descent on the last layer, then the residuals go to zero: enter image description here

Here I froze the first layer at 5000 epochs. The red line is plotting $\sum_{i=1}^N \hat{y}_i (y_i-\hat{y}_i)$ for the NN's predictions; the purple shows the same sum calculated with linear regression on the hidden features. The other lines show the activations of the hidden nodes and how they shift around until I freeze them.

If you just run forever, the residuals will go to zero if the hidden features stop shifting around: enter image description here

Code here

Improve this answer
$\endgroup$
1
  • $\begingroup$ do you think this might just be because when you fix all but the last hidden layer, we are simply doing linear regression with a different set of features? $\endgroup$
    – John Madden
    Commented May 31, 2023 at 0:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.