One fact about linear regression is that the predictions and residuals are orthogonal. In other words:

$$ \sum_{i=1}^N \hat{y}_i (y_i-\hat{y}_i) = 0 $$

In nonlinear regression, this fails to be the case.$^{\dagger}$

That does not make sense to me. I've simulated what happens and confirmed that the nonlinear regression lacks orthogonal residuals and predictions, but it still is not intuitive, particularly for an approach with a neural network.

The neural network above does some feature engineering to find three features to feed into a linear regression, but the neural network is a nonlinear regression, since there would be a ReLU activation function in the hidden layer acting on the red, blue, and yellow parameters. However, if I got lucky and guessed the features in the hidden layer, then I could call my regression linear.

Those seem like the same model to me, yet one would be a linear regression with orthogonal residuals and predictions and one would be a nonlinear regression that lacks orthogonal residuals and predictions.

What gives? A few links to threads that discuss this lack of orthogonality:

regression - does R2 only apply to measure linear regression performance?

Is R-squared truly an invalid metric for non-linear models?

$^{\dagger}$I am not sure if it can hold for a nonlinear regression, but it at least does not have to hold for a linear regression.

One fact about linear regression is that the predictions and residuals are orthogonal. In other words:

$$ \sum_{i=1}^N \hat{y}_i (y_i-\hat{y}_i) = 0 $$

In nonlinear regression, this fails to be the case.$^{\dagger}$

That does not make sense to me. I've simulated what happens and confirmed that the nonlinear regression lacks orthogonal residuals and predictions, but it still is not intuitive, particularly for an approach with a neural network.

The neural network above does some feature engineering to find three features to feed into a linear regression, but the neural network is a nonlinear regression, since there would be a ReLU activation function in the hidden layer acting on the red, blue, and yellow parameters. However, if I got lucky and guessed the features in the hidden layer, then I could call my regression linear.

Those seem like the same model to me, yet one would be a linear regression with orthogonal residuals and predictions and one would be a nonlinear regression that lacks orthogonal residuals and predictions.

What gives?

A few links to threads that discuss this lack of orthogonality:

regression - does R2 only apply to measure linear regression performance?

Is R-squared truly an invalid metric for non-linear models?

$^{\dagger}$I am not sure if it can hold for a nonlinear regression, but it at least does not have to hold for a nonlinear regression.