# Iterating through rows and filtering data in R

I am working with the following data (see picture)

I am trying to predict future Y axis values from the X values. As you can tell from the picture and as I discovered, the prediction is going to be horrible based on the data alone.

I am now trying to go row by row through the data and classify each point as significant or nonsignificant and see if I can find anything interesting in the classification. I want my algorithm something like this:

1. Take absolute value of every element in both Y axis and X axis data columns

2. Iterate through Y axis column and find the top 5% largest values

3. Iterate through again marking those top 5% or maybe 1% (Will fiddle with this) with a flag

4. Plot only the top 5% and see if the linear model (or some model) has a better R^2

5. Perform some sort of classification analysis against the two flags, significant and non significant (I really need help and suggestions with this step)

The two things I need are a pointer on are

• How to step element by element through data, making modifications along the way
• What kind of classification analysis exists that might be helpful, goal being to predict which classification future sampled elements will fall into.

You don't step element by element through data in R. Well, you can, but you need to be a very patient person.

The R way would be to calculate the quantile you want and subset the data.

Example:

set.seed(42)
DF <- data.frame(x=runif(1e3),y=rlnorm(1e3))
plot(y~x,data=DF)

#95 % quantile
q95 <- quantile(DF$y,0.95) #plot data subset plot(y~x,data=DF[DF$y>q95,])

#fit linear model to data subset
fit <- lm(y~x,data=DF[DF\$y>q95,])
abline(fit)