6
$\begingroup$

I've got the following sets of data (x and y scales on each chart are identical):

enter image description here

The eye picks out an upward-sloping floor trendline on each chart that runs upward from left to right, hugging the bottom of the bulk of blue dots, with a few outliers below it. And that trendline is different for each chart.

That's easy enough for the eye to pick out. But is there a robust way to do it algorithmically?

I've considered writing an objective function that calculates the residuals from a trend line, discards (say) 1% of points that are further below the trend line, and then weights residuals so that points above the line all contribute very little to the objective function, and points below the line contribute a lot, to get the line to hug the bottom of the bulk. And then find the trendline that minimises that objective function.

Are there more robust methods?

$\endgroup$
8
$\begingroup$

I think linear quantile regression would be close to what you want. This fits a line so that the predicted value for each x value is close to the chosen quantile of the response conditional on x.

Here's an R package: http://cran.r-project.org/web/packages/quantreg/index.html

For example, you could try a 1% quantile, and see if that avoids the outliers. You can adjust the quantile you choose until it looks about right.

If you want to be more principled about deciding where the outliers start, I think you'll need to make some more assumptions about your data distribution.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ +1 This was my first thought as well. It seems to me that this is fairly principled. $\endgroup$ – Wayne Aug 25 '13 at 15:18
  • $\begingroup$ Well said from amother DavidR $\endgroup$ – IrishStat Aug 25 '13 at 15:22
  • $\begingroup$ Thanks, linear quantile regression does a pretty decent job. I found the MatLab quantreg package, and that's given me something pretty plausible. $\endgroup$ – 410 gone Aug 26 '13 at 15:06
  • 1
    $\begingroup$ In principle and in practice linear quantile regression could mean predictors other than straight x, e.g. cubic splines or fractional polynomials in x. $\endgroup$ – Nick Cox Aug 27 '13 at 17:57
  • $\begingroup$ Yes, I agree @NickCox... you can just feed transformed x's into your quantile regression. If you have enough data, you can also pretty easily do a "poor man's" non-parametric quantile regression: bin up the x's into groups, as though you were making a histogram, and find the quantile you're looking for in each bin. Then you get a piecewise constant nonparametric quantile regression line. $\endgroup$ – DavidR Aug 27 '13 at 18:14
0
$\begingroup$

Is there only 1 trend line ? Probably not . You have outlying points below the "visual trend line at the bottom" . How will these be "ignored" so as to capture the dominant floor trend line that the eye sees and not be influenced by them. To detect tehm and reduce their influence one would want to simultaneously detect BOTH trend line(s) and pulses that are inconsistent with the trend(s). If you can reduce your xy observations as you said and then post the reduced set , I might take a shot at this using the only piece of commercially avaialable software that I know that deals with trend detection while considering ARIMA structure and pulses. As far as I know nothing free is available and of course the heuristics to duplicate the human eye are not disclosable.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

perhaps try running this custom-built code in the R language..if you've never used R, check http://twotorials.com/ to get started ;)

# create eight sets of a thousand random x values
x1 <- rnorm( 1000 , mean = 1 )
x2 <- rnorm( 1000 , mean = 2 )
x3 <- rnorm( 1000 , mean = 3 )
x4 <- rnorm( 1000 , mean = 4 )
x5 <- rnorm( 1000 , mean = 5 )
x6 <- rnorm( 1000 , mean = 1 )
x7 <- rnorm( 1000 , mean = 1 )
x8 <- rnorm( 1000 , mean = 3 )

# create eight sets of a thousand random y values
y1 <- rnorm( 1000 , mean = 1 )
y2 <- rnorm( 1000 , mean = 2 )
y3 <- rnorm( 1000 , mean = 3 )
y4 <- rnorm( 1000 , mean = 4 )
y5 <- rnorm( 1000 , mean = 5 )
y6 <- rnorm( 1000 , mean = 5 )
y7 <- rnorm( 1000 , mean = 3 )
y8 <- rnorm( 1000 , mean = 5 )

# combine all of these values into two vectors
x <- c( x1 , x2 , x3 , x4 , x5 , x6 , x7 , x8 )
y <- c( y1 , y2 , y3, y4 , y5 , y6 , y7 , y8 )

# this distribution looks like your example distribution
plot( x , y )

# along the x axis, figure out some reasonable intervals to "bin" the data

# let's say you want one bin per 100 points..
num.bins <- length( x ) / 100

# figure out what quantiles to cut your bins at
quantile.probs <- seq( 0 , 1 , length.out = num.bins )

# slice up your `x` data into that many equal bins
bin.cutpoints <- quantile( x , quantile.probs )

# now let's look at just the first bin #

# positions within the first bin
first.bin <- which( bin.cutpoints [ 1 ] <= x & x < bin.cutpoints [ 2 ] )

# x midpoint between first two cutpoints
first.midpoint <- as.numeric( bin.cutpoints [ 1 ] + ( bin.cutpoints [ 2 ] - bin.cutpoints [ 1 ] ) / 2 )

first.midpoint

# since you wanted to discard 1% of all points, choose the 1% quantile cutoff point
one.percent.cutoff <- round( quantile( 1:length( y[ first.bin ] ) , 0.01 ) )

# find the point at the edge of the first percentile within this bin
first.percentile <- sort( y[ first.bin ] )[ one.percent.cutoff ]

# and there's your `y` value
first.percentile


# create two empty vectors to start storing values
low.x <- NULL
low.y <- NULL

# repeat this process for all bins:
for ( i in 2:length( bin.cutpoints ) ){

    this.bin <- which( bin.cutpoints [ i - 1 ] <= x & x < bin.cutpoints [ i ] )

    this.midpoint <- as.numeric( bin.cutpoints [ i ] + ( bin.cutpoints [ i ] - bin.cutpoints [ i ] ) / 2 )

    low.x <- c( low.x , this.midpoint )

    # since you wanted to discard 1% of all points, choose the 1% quantile cutoff point
    one.percent.cutoff <- round( quantile( 1:length( y[ this.bin ] ) , 0.01 ) )

    # find the point at the edge of the first percentile within this bin
    first.percentile <- sort( y[ this.bin ] )[ one.percent.cutoff ]

    # and there's your `y` value
    first.percentile

    low.y <- c( low.y , first.percentile )
}

# plot your original points
plot( x , y , main = 'one bin per hundred points' )

# RE-plot the points that had the second-lowest y value within each "bin"
# so you can see exactly what line you're best-fitting
points( low.x , low.y , col = "red" , pch = 19 )

# draw your line of best fit
abline( lm( low.y ~ low.x ) )

here's the result.. enter image description here

notice it is sensitive to the size of each bin..

enter image description here

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Interesting ! Does this distinguish multiple trend lines or does it assume 1 trend. How does it deal with the third row third column example from the OP. Do you have to manually ignore the anomalies after the visually obvious floor trend line ? $\endgroup$ – IrishStat Aug 25 '13 at 15:21
  • $\begingroup$ Thanks for this. Please add a disclosure about your relationship with the site you've linked to - otherwise, your answer risks being deleted as spam. $\endgroup$ – 410 gone Aug 26 '13 at 14:10
  • $\begingroup$ i anthony damico hereby disclose my relationship with twotorials.com. also, linear quantile regression is the right answer ;) $\endgroup$ – Anthony Damico Aug 27 '13 at 13:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.