2
$\begingroup$

These are error's empirical distribution for XGB, RF and kNN, the last one have taken on another dataset.

enter image description here

Neither of them is normally distribuited but they all are symmetric. None of used algorithms have even used MSE optimum, for example, both XGB and RF make a greedy approach for it due to being decision-tree-based and kNN uses euclidean distance which has nothing to do with MSE because that is not even a error-based estimation, my guess is that happens due to quadratic-based methods ignore error's signal but I can't link that to symmetry in probability density sense.

Improve this question
$\endgroup$
1
  • $\begingroup$ I do wonder what quadratic estimation you have in mind that isn't equivalent to MSE. Remember that MSE, SSE, and RMSE all have the same argmin (and the same as the argmax of $R^2$), so I consider all to be equivalent loss functions (in some sense). $\endgroup$
    – Dave
    Sep 18, 2021 at 5:32

1 Answer 1

Reset to default
1
$\begingroup$

NO

For a counterexample, fit an OLS regression with an exponential error term.

$$ Y=X\beta+\epsilon\\ \epsilon_i\overset{iid}{\sim}exp(1) $$

When you do the OLS fit, which uses square loss, you will get asymmetric residuals.

(There’s an annoying issue where $exp(1)$ has a mean of $1$ instead of $0$. You can resolve this if you do a simulation in software like R by subtracting $1$ from every error value. That isn’t an exponential distribution but a shifted exponential, but the asymmetric residuals will be present.)

Here is a simulation demonstrating my counterexample.

set.seed(2021)
N <- 1000
x <- seq(0, 10, 10/(N - 1))
e <- rexp(N, 1) - 1 # subtract 1 so E(e) = 0
y <- x + e
plot(x, y)
L <- lm(y ~ x)
hist(resid(L))
plot(L)

The histogram of the residuals and first two plots of the model L show the marked asymmetry of the residuals.

enter image description here

enter image description here

enter image description here

Improve this answer
$\endgroup$
7
  • $\begingroup$ You're sure, I think the error should've got a distribution lying in $(-a,a)$ to OLS do what I supposed it'd do. $\endgroup$
    – Davi Américo
    Sep 18, 2021 at 0:11
  • $\begingroup$ Why do you say that? $\endgroup$
    – Dave
    Sep 18, 2021 at 1:21
  • $\begingroup$ It was as I thought, I've simulated a non-symmetric distribution with symmetric interval then I used that as error and I've generated y=x^2+e, after I trained it by random forest via MSE impurity and I've gotten symmetric residuals even though error not being. I'm going to send you my script. $\endgroup$
    – Davi Américo
    Sep 18, 2021 at 4:32
  • $\begingroup$ import numpy as np from sklearn.ensemble import RandomForestRegressor from matplotlib import pyplot as plt k=1/(np.exp(1)-np.exp(-1)) u=np.random.uniform(size=500) e=np.log((u+k*np.exp(-1))/k)##symetric_support-non_symmetric_distribution sample plt.hist(e) x=np.random.uniform(30,70,size=500) y=x**2+e plt.scatter(x,y) m=RandomForestRegressor().fit(x.reshape(-1,1),y) ye=m.predict(x.reshape(-1,1)) r=ye-y plt.hist(r)###run it after run before rows if youre using spyder, residuals arent symmetric even though error is $\endgroup$
    – Davi Américo
    Sep 18, 2021 at 4:40
  • $\begingroup$ @DaviAmérico Please see my edit. While you may have demonstrated that the residuals can be symmetric despite an asymmetric error term, your question was looking for a counterexample to the claim that square loss produces symmetric errors, which I give. $\endgroup$
    – Dave
    Sep 18, 2021 at 4:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.