# How to identify inflection point from list of results and a graph with no given formula?

I have df1

> df1
V1      V2      V3      V4   V5     V6       V7  V8
1   1   EFNB2   EFNB2   EFNB2   72  0.667  0.05980 YES
2   2    AOC2    AOC2    AOC2  108  0.609  0.13000 YES
3   3     FOS     FOS     FOS  142  0.554  0.19300 YES
4   4   CCNE1   CCNE1   CCNE1  167  0.530  0.25700 YES
5   5    CBX4    CBX4    CBX4  275  0.448  0.26700 YES
6   6    CRY1    CRY1    CRY1  343  0.400  0.29000 YES
7   7 TNFAIP6 TNFAIP6 TNFAIP6  600  0.194  0.18500  NO
8   8    SSX3    SSX3    SSX3  641  0.171  0.18900  NO
9   9    WNT2    WNT2    WNT2 1211 -0.173 -0.08030  NO
10 10   FGF18   FGF18   FGF18 1296 -0.225 -0.09130  NO
11 11    PIM2    PIM2    PIM2 1386 -0.273 -0.09780  NO
12 12   PLCL1   PLCL1   PLCL1 1636 -0.415 -0.16700  NO
13 13     MX2     MX2     MX2 1676 -0.437 -0.12400  NO
14 14   ABCA6   ABCA6   ABCA6 1720 -0.457 -0.07980  NO
15 15 HSPA12A HSPA12A HSPA12A 1788 -0.501 -0.04190  NO
16 16 TNFSF10 TNFSF10 TNFSF10 1944 -0.855  0.00155  NO


I am supposed to iterate through RES column and find the point of inflection and mark everything from there with YES -CORE_ENRICHMENT-. P-value is identified and according to my very little knowledge in statistics that we need to set the second derivative of f(x) to 0 in order to identify it. Is there anything that I am missing or a formula that might help me to understand how can I identify it? I need to implement a program in R while I am really unable to understand the problem.

Edit: Based on User603 suggestion: I modified the code as follow, and plotted this plot:

df1 <- read.table("file.XLS", header = FALSE)
mlearn <- df1$V7 #plot of RES: plot(mlearn, ylim = c(-0.2, 0.3), type = 'p', pch = 16) #plot of RES^1: #plot of the smoothed values of RES: lines(fdata.deriv(mlearn, nderiv = 0, method = "bspline", class.out = 'fd', nbasis = 6), col = 'blue') #the first derivative of the smoothed values of RES lines(fdata.deriv(mlearn, nderiv = 1, method = "bspline", class.out = 'fd', nbasis = 6), col = 'red') legend('topleft', col = c('blue', 'red'), lty = c(1, 1), legend = c('Smoothed curve', 'First derivative of smoothed curve')) abline(h = 0)  1- If I were to identify IP from the graph, it should be something around 5, however, I've been told by my doctor that the other crossing point around 12 should be an IP as well. I used this code to check the condition: All redline points cross 0. unlins <- RedLine_points$data[1, 1:16]
ifelse(diff(sign(unlins))==-2|diff(sign(unlins))==2 , mlearn, NA)


However, when I run the code, what returns is two points: First point is the point before the IP around 5 which I believe is wrong. 2nd point is exactly the point around 12.

How can I enhance my code to return the exact IPs. I need a general formula as I am working on 18 other files.

Update: This is the modified code suggested

install.packages("matrixStats")
install.packages('fda.usc')
library(matrixStats)
library(fda.usc)
mlearn <- df1$V7 rc_container <- fdata.deriv(mlearn, nderiv = 1, method = "bspline", class.out = 'fdata', nbasis = 6) red_curve <- cbind(rc_container$argvals, t(rc_container$data)) red_curve <- data.frame(red_curve) colnames(red_curve) <- c('x', 'y'); plot(red_curve$x, red_curve$y * 30, type = 'l', col = 'red') diff_rc <- cbind(2:length(red_curve$y)-1 ,
red_curve$y[-c(length(red_curve$y))],
red_curve$y[-c(1)]) x_loc <- red_curve$x[diff_rc[which(rowProds(diff_rc[, 2:3]) < 0), 1]] #return index "index"
mtest       <- phoneme$learn[1, test_sample] #think of this as RES^1 #plot of RES: plot(mlearn, ylim = c(-5, 20), type = 'p', pch = 16) #plot of RES^1: points(mtest$argvals, mtest$data, pch = 8) #plot of the smoothed values of RES: lines(fdata.deriv(mlearn, nderiv = 0, method = "bspline", class.out = 'fd', nbasis = 6), col = 'blue') #plot of 30 times (so it is easily visible) #the first derivative of the smoothed values of RES lines(fdata.deriv(mlearn, nderiv = 1, method = "bspline", class.out = 'fd', nbasis = 6) * 30, col = 'red') #plot of the smoothed values of RES^1: lines(fdata.deriv(mtest, nderiv = 0, method = "bspline", class.out = 'fd', nbasis = 6), col = 'blue') #the first derivative of the smoothed values of RES^1 lines(fdata.deriv(mtest, nderiv = 1, method = "bspline", class.out = 'fd', nbasis = 6) * 30, col = 'red') legend('topleft', col = c('blue', 'red'), lty = c(1, 1), legend = c('Smoothed curve', 'First derivative of smoothed curve')) abline(h = 0)  So in this example the original data ($\text{RES}$) are the dots (ordered by frequencies). The stars show another sample of the same distribution ($\text{RES}^1$). The blue line is the result of applying a scatter plot smoother to the entries of ($\text{RES}$) (the dotted blue line is the same but based on$\text{RES}^1$). The red curve is the (explicit) first derivative of the blue curve (multiplied by 30 so it is visible on the same scale). Here too, the dotted red line is constructed like the red one but using$\text{RES}^1$instead of$\text{RES}$. On the red line, there is a single localization where the red curve crosses 0 and is positive to the left and negative to the right of that point (around frequency = 100). This will be the estimate of$\text{IP}_0$based on$\text{RES}$($\text{IP}(\text{RES})$). You can see that this estimate is close to the maximum values of$\text{RES}$but doesn't quiet match it. This is because the blue line tries to dampen (or smooth) the idiosyncratic features of ($\text{RES}$). Because of this dampening, the estimate of$\text{IP}_0$you would have obtained from$\text{RES}^1$is not that different from the one you obtained using$\text{RES}$. In this case$\text{IP}(\text{RES}^1)$is just a little to the right of$\text{IP}(\text{RES})$. Your estimate of the true inflection point you get in this way is less moved around by the idiosyncrasies of the individual samples you use to compute it and will be a more reliable predictor of the true$\text{IP}_0$. # Edit: Code to find the points where the red curve crosses 0. This is an adaptation of the code posted by @josliber here. library(matrixStats) rc_container <- fdata.deriv(mlearn, nderiv = 1, method = "bspline", class.out = 'fdata', nbasis = 6) red_curve <- cbind(rc_container$argvals, t(rc_container$data)) red_curve <- data.frame(red_curve) colnames(red_curve) <- c('x', 'y'); plot(red_curve$x, red_curve$y * 30, type = 'l', col = 'red') pos <- head(cumsum(rle(red_curve$y >= 0)$lengths), -1) x_loc <- (red_curve$x[pos] + red_curve$x[pos+1]) / 2 c("positive", "negative")[head(rle(red_curve$y >= 0)$values + 1, -1)] abline(v = x_loc) abline(h = 0)  The script above finds all interval where the first derivative of the blue curve (the red curve) crosses 0 (since you now want both local minimae and local maximae). # Edit2: I am now adapting the code to run on the OP's specific example. According to the OP, the variable of interest is 'V7'. library(fda.usc) a1 <- read.table('op_data.txt', header = FALSE) mlearn <- a1[, 'V7'] plot(mlearn, type = 'p', pch = 16) lines(fdata.deriv(mlearn, nderiv = 0, method = "bspline", class.out = 'fd', nbasis = 6), col = 'blue') lines(fdata.deriv(mlearn, nderiv = 1, method = "bspline", class.out = 'fd', nbasis = 6), col = 'red', ylab = 'First derivative of smoothed mlearn') abline(h = 0)  Now, it remains to find the points where the red curve crosses 0. Using the code above, we get: rc_container <- fdata.deriv(mlearn, nderiv = 1, method = "bspline", class.out = 'fdata', nbasis = 6) red_curve <- cbind(rc_container$argvals, t(rc_container$data)) red_curve <- data.frame(red_curve) colnames(red_curve) <- c('x', 'y'); plot(red_curve$x, red_curve$y, type = 'l', col = 'red') pos <- head(cumsum(rle(red_curve$y >= 0)$lengths), -1) x_loc <- (red_curve$x[pos] + red_curve$x[pos+1]) / 2 c("negative", "positive")[head(rle(red_curve$y >= 0)$values + 1, -1)]  So the localizations of changes in sign in the first derivative of the smoothed curve are: x_loc  which in your case are$x=\{4.5, 12.5, 15.5\}$. If we look at the signs of the red curve between these intervals: c("negative", "positive")[head(rle(red_curve$y >= 0)\$values + 1, -1)]


we see that the blue line is increasing between indexes 1 and 4.5 then decreasing between indexes 4.5 and 12.5 and then increasing again between indexes 12.5 and 16.

• Thank you so much for the fascinating explanation. I wasn't able to find a similar answer on the internet for days. I think that I pretty much understand your explanation. I am trying to study the R code you provided and see how can I implement a similar one to my problem. I might need your help here again. Thank you! Oct 13, 2017 at 21:05
• No problem, all the functions I used are open source and documented and easy to use and install, see the fda.usc package. There are some ancillary aspects I left out to make the answer more readable, but you should be able to find them addressed on cross validated ('how to select the parameters of a bspline?'). All in all this is fairly standard stuff;) Oct 14, 2017 at 14:52
• I have modified my question above @user603 I only used RES samples for now as I don't have the whole one. I'm still a bit confused Oct 15, 2017 at 8:35
• I appreciate your help. now, I should check the change of signs for all y_loc (prev and next)points. Since there was 5 points returned of y_loc and I need to make sure which makes IP. Is that correct? Oct 30, 2017 at 12:10
• @MarJamil: I updated my code. We only need to consider pairs of succesive points where the sign of the first is not the same as the sign of the second. The changes are in the '#Edit' part only. Check if it helps. Can you update your question to use the new code? Oct 30, 2017 at 13:33