As far to my knowledge r-squared should not be used in non-linear regression setup. Not only might the r2 be too high, but also the interpretation as the variance explained by the model might no longer be valid.
But looking at the mean squared error, it seems to be very similar. But it is said everywhere that mean squared error can be used in non-linear regression model. So my question is: is mse valid or should it also be considered not valid for non-linear regression?
There are two common reasons why square loss (such as $MSE$) is popular.
Large misses are brutally punished. If your residual is $1$, your squared residual is $1$, but if your residual is $2$, your squared residual is $4$, and if your residual is $10$, your squared residual is $100$. By increasing the residual by a factor of ten, the square loss penalty is increased by a factor of $100$.
If we assume Gaussian error terms, the least squares solution is equivalent to maximum likelihood of the regression parameters.
(If others comment about other reasons why square loss is popular, I can edit to include them, but these are the ones that come to mind quickly.)
The way to compare which model does a better job of minimizing square loss is to look at which model minimizes such a value, so of course there is a sense in which $MSE$ is a legitimate performance metric.
A typical criticism of $R^2$ is that it can be driven arbitrarily high by overfitting, and this criticism is legitimate. However, measures of square loss like $MSE$ and $RMSE$ suffer from the same issue. If $R^2 = 1$ then $RMSE = 0$ and $MSE=0$.
A remedy for this is to do out-of-sample testing, such as the cross validation for which this Stack is named. While we might be able to drive $R^2$ up to $1$ by including features that are unrelated to the outcome and give a regression model that fits the noise instead of the signal (something like playing connect-the-dots with the scatterplot), out-of-sample performance will be poor when this is the case, hence the appeal of out-of-sample testing in machine learning.
There is an out-of-sample $R^2$:
$$ R^2_{out} = 1 - \dfrac{n_{out}MSE_{out}}{\sum_{i = 1}^{n_{out}}\big(y_i - \bar y_{in}\big)^2}\\ n_{out}\text{: Number of observations in the out-of-sample data}\\ MSE_{out}\text{: Mean of the squared residuals for the out-of-sample predictions}\\ \bar y_{in}\text{: Mean of the in-sample response variable} $$
Notice that the subscripts in that equation indicate out-of-sample numbers, except for the mean $\bar y_{in}$.
To understand why, consider what in-sample $R^2$ measures: a comparison of the model under consideration compared to a model that naïvely guesses the pooled mean of $y$ every time in its attempt to model the conditional mean. It makes sense to consider such a model to be the baseline when we test out-of-sample. If we cannot do better than just guessing $\bar y_{in}$ every time, we have done a poor job of modeling the conditional mean.
The denominator term of $R^2_{out}$ is constant for a given test set, regardless of the model. Consequently, maximizing $R^2_{out}$ is equivalent to minimizing $MSE$.
REGARDING THE NOTEBOOK you linked in the comments, you made at least two mistakes.
A quadratic relationship does not preclude linear modeling. Indeed, $y_i = \beta_0 +\beta_1x_i +\beta_2x_i^2 +\epsilon_i$ is a linear model and would give quite a good fit to your parabolic scatterplot. A nonlinear regression would be something like a neural network with $ReLU$ activation functions in order to fit the parabola, but you could do the same with the first plot.
You are comparing models of different data. The critical part of my argument is that the denominator term in the $R^2$ equation, either in-sample or out-of-sample, is constant. If you change that value, then you can get situations where lower $MSE$ corresponds to lower $R^2$.
I quote two sentences from Applied Linear Statistical Models (Kutner) that are relevant for the question.
p.525: the error sum of squares SSE and the regression sum of squares SSR do not necessarily sum to the total sum of squares SSTO. Consequently, the coefficient of multiple determination R2 = SSR/SSTO is not a meaningful descriptive statistic for non-linear regression.
p.528: For nonlinear regression, MSE is not an unbiased estimator of sigma2, but the bias is small when the sample size is large.
So unlike linear regression, the inference procedure for nonlinear regression based on MSE is only approximate, but it is good enough for large samples.
