10
$\begingroup$

I'm using the quantreg package to make a regression model using the 99th percentile of my values in a data set. Based on advice from a previous stackoverflow question I asked, I used the following code structure.

mod <- rq(y ~ log(x), data=df, tau=.99)    
pDF <- data.frame(x = seq(1,10000, length=1000) ) 
pDF <- within(pDF, y <- predict(mod, newdata = pDF) )

which I show plotted on top of my data. I've plotted this using ggplot2, with an alpha value for the points. I think that the tail of my distribution is not being considered sufficiently in my analysis. Perhaps this is due to the fact that there are individual points, that are being ignored by the percentile type measurement.

One of the comments suggested that

The package vignette includes sections on nonlinear quantile regression and also models with smoothing splines etc.

Based on my previous question I assumed a logarithmic relationship, but I'm not sure if that is correct. I thought I could extract all the points at the 99th percentile interval and then examine them separately, but I'm not sure how to do that, or if that is a good approach. I would appreciate any advice on how to improve identifying this relationship.

enter image description here

Improve this question
$\endgroup$
6
  • $\begingroup$ There are a couple of good questions on the site already talking about transforming data like this, see stats.stackexchange.com/q/1444/1036 or stats.stackexchange.com/q/298/1036 $\endgroup$
    – Andy W
    Mar 31, 2011 at 14:35
  • $\begingroup$ Can you update the plot to add the conditional median? this seems to me more like a quantile crossing problem than a data transformation problem... $\endgroup$
    – user603
    Jul 13, 2011 at 17:53
  • $\begingroup$ @user603 What do you mean by the conditional median? (I searched online but am not sure how to calculate it) $\endgroup$
    – celenius
    Jul 13, 2011 at 18:46
  • $\begingroup$ tau=0.5 in the rq() function. $\endgroup$
    – user603
    Jul 25, 2011 at 15:30
  • 1
    $\begingroup$ If your goal is specifically to estimate the conditional 99th percentile, I'd vote for nonlinear quantile regression (of some sort--I don't know the R packages well), since it doesn't sound like you know the true functional form. I still wasn't clear to me from your previous question what the actual goal is, though, so I would reiterate the comment on your previous question from Spacedman Jan 4 at 17:01 $\endgroup$
    – David M Kaplan
    Sep 12, 2011 at 16:07

1 Answer 1

Reset to default
1
$\begingroup$

All models are wrong, but some are useful (George Box). You are forcing a logrithmic shape to your fitted curve, and honestly it doesn't look that bad. The fit is poor at the tail because there are less points there; the two parameters you have allowed will fit the bulk of the data. In other words, on a log scale, that tail isn't far enough away from the bulk of your data to provide leverage. It doesn't have to do with the quantile nature of the regression; OLS would also disregard those points (especially on the log scale).

It's pretty easy to allow for some more non-linearity. I'm partial to natural splines, but again, all models are wrong:

library(splines)
mod <- rq(y ~ ns(log(x), df=6), data=df, tau=.99)

The quantreg package has some special hooks for monotonic splines if that's of concern to you.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.