5
$\begingroup$

I'm trying to use the quantreg package to fit an exponential curve.

Here is a reproductible example. IRL I have much more complex data with outliers, that's why I prefer not using nls which is not robust to outliers.

library(quantreg)
library(ggplot2)

x = 1:100
set.seed(42)
y = 500*exp(-0.02*x) +rnorm(100, 0, 5 )
df = data.frame(cbind(x,y))
plot(df)

formula =  y ~ k * exp(b*x) 
qr_exp = nlrq(formula,
                   data = df,
                   start = list(k = 600, b = -0.01),
                   tau = .50,
                   nlrq.control(maxiter=1000))
summary(qr_exp)
sum(qr_exp$m$resid())
[1] -26.52373

I expected to have sum(qr_exp$m$resid()) around 0 since tau = 0.5but the value is negative which means the model tend to overestimate the real values.

As you can see I have sum of the residual is closer to 0 with tau= 0.47

formula =  y ~ k * exp(b*x) 
qr_exp = nlrq(formula,
              data = df,
              start = list(k = 600, b = -0.01),
              tau = .47,
              nlrq.control(maxiter=1000))
summary(qr_exp)
sum(qr_exp$m$resid())
[1] -4.467781

I don't really understand why !

Is it because there could be an infinite number of solution and so no garantee of having as much negative residual than positive residual ?

If yes what is the best solution if this is very important for me to:

  • minimise Least absolute deviation and not least square deviation (not robust with outliers)
  • have balanced residual?

Could it make sense to add a small portion of L2 penalty to have something balanced ? (see Huber loss)

$\endgroup$
5
$\begingroup$

You are conducting a median regression (tau=0.5) on asymmetrically distributed data. Here is a simpler example to show what is going on.

Suppose your asymmetric data are lognormal:

set.seed(1)
xx <- rlnorm(100,0,1)

Then what you are doing amounts to finding the median of your data.

median(xx)
[1] 1.121518

Now, the median minimizes the sum of absolute errors. It will not minimize the sum of "raw" errors:

sum(xx-median(xx))
[1] 52.74494

If you want a value that minimizes the sum of "raw" errors, you need to take the mean:

mean(xx)
[1] 1.648967
sum(xx-mean(xx))
[1] -9.992007e-15

So: if it is important to you that your fit yields zero average error, you will need to run an ordinary OLS regression. Which will, of course, be sensitive to outliers. (You incidentally found that the conditional mean is equal to the conditional 47% quantile. But that of course won't minimize absolute deviations.)

There is no way to have both minimal absolute deviation and balanced residuals if your distribution is asymmetric. You can of course find a tradeoff between median and OLS regression, perhaps by taking an average, or by regularizing in some other way (lasso, ridge regression, elastic net).

$\endgroup$
3
  • $\begingroup$ Thanks @stephan for this answer. It clarifies the problem. I'm just not sure to udnerstand when you say OLS can help me to have a zero average user. If I remember well I will have a zero average of squared residual but not zero average of the residual itself ! $\endgroup$
    – psql
    Oct 27 '15 at 18:09
  • $\begingroup$ Would it make sense to minimise the sum of the residuals itself ? (no squared or absolute value). Or just a bit of suqared residual so the model doesn't just give the mean as a predictor For information I'm thinking about this because I want my prediction to have the same expectancy as the real values. $\endgroup$
    – psql
    Oct 27 '15 at 18:21
  • $\begingroup$ OLS will yield a zero average error, and a minimal squared error. (The squared error won't be zero unless you have a perfect fit, since all errors will have a positive contribution because of the squaring.) Try a few examples! If you minimize the sum of residuals, you'll just get an astronomically high "fit", so all errors are astronomically negative... If you want an unbiased prediction (the expected value), you should do OLS and getting the conditional mean. Do look at regularization, which gives you higher bias but lower variance, for reduced overall expected error. $\endgroup$ Oct 27 '15 at 19:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.