Investigate correlation conditional on a threshold

Question

I have 3 variables in my data set.

(i) My gut feel says variable1 and variable2 are correlated, only when variable3 >= threshold3. What is the technique I can use to see if this holds true, to statistical significance (simple regression based/machine learning)?

eg. Taking variables in Stock Market, I think stock price move is correlated to Price Momentum over the last 3o minutes, but only when Volatility Index is above a certain threshold. I do not know the threshold value and also do not know if the statement, that the above correlation is significant only when the threshold is met, holds. Looking for a technique that I can use to formally investigate.

(ii) The other scenario: My gut feel says variable1 and variable2 are correlated, after variable2 >= threshold2. What would be the technique to investigate this?

eg. Similar to the above example, when the correlation between two variables is significant only when one of the 2 variables is above a threshold in its space, looking to investigate if that is true and also estimate the threshold.

Answer 1

You could investigate your data visually with a conditioning plot, in R this is named coplot. There is an example in Best method to visualize large interaction between two factors or run example(coplot) within R.

Say your tree variables are $x,y,z$ and you suspect the relationship between $y$ and $x$ depends on the values of $z$. Then you would use the following command in R:

coplot(y ~ x | z, data=yourdataframe)

assuming your variables are in data frame yourdataframe.

Below an example of an conditioning plot produced by R:

The R code used was:

data(swiss)
with(swiss, coplot(Fertility ~ Education | Agriculture))

To read the plot: The order of the subplots is from bottom and from left, so the plot corresponding to the lowest vales of variable Agriculture is bottom left. Other examples of coplots can be found here: What is the physical significance of cumulative correlation coefficient? and Can I analyze or model a conditional correlation?

As for your second scenario: Here there is only two variables, so a usual scatterplot will do. You can add some nonparametric smooth to the plot, for a more formal approach see How to use segmented package to fit a piecewise linear regression with one breakpoint? and search this site for the tag piecewise-linear.