I am working with the following data (see picture)

I am trying to predict future Y axis values from the X values. As you can tell from the picture and as I discovered, the prediction is going to be horrible based on the data alone.

I am now trying to go row by row through the data and classify each point as significant or nonsignificant and see if I can find anything interesting in the classification. I want my algorithm something like this:

  1. Take absolute value of every element in both Y axis and X axis data columns

  2. Iterate through Y axis column and find the top 5% largest values

  3. Iterate through again marking those top 5% or maybe 1% (Will fiddle with this) with a flag

  4. Plot only the top 5% and see if the linear model (or some model) has a better R^2

  5. Perform some sort of classification analysis against the two flags, significant and non significant (I really need help and suggestions with this step)

    enter image description here

The two things I need are a pointer on are

  • How to step element by element through data, making modifications along the way
  • What kind of classification analysis exists that might be helpful, goal being to predict which classification future sampled elements will fall into.

1 Answer 1


You don't step element by element through data in R. Well, you can, but you need to be a very patient person.

The R way would be to calculate the quantile you want and subset the data.


DF <- data.frame(x=runif(1e3),y=rlnorm(1e3))

#95 % quantile
q95 <- quantile(DF$y,0.95)

#plot data subset

#fit linear model to data subset
fit <- lm(y~x,data=DF[DF$y>q95,])

Can't help you with the statistical issue ...

  • $\begingroup$ Hey this helped a lot. I really needed some help with the basic subset commands. $\endgroup$ Jun 11, 2013 at 19:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.