Pearson correlation in data set with “jump”

I'm trying to calculate the correlation between the condition number of a finite elements matrix and the coarseness of the mesh that it represents. However, when trying to calculate the Pearson correlation coefficient between the two, my data shows that at a very particular value of the parameter I use to quantify coarseness, the condition number of the matrix jumps up by quite a bit. I suspect this is due to the triangular elements of the matrix becoming very elongated, thus making the matrix ill-conditioned. However, there clearly is some sort of link between the two parameters, which doesn't show up clearly when calculating the correlation.

My question is "How can I adjust the correlation calculation so as to still clearly indicate that there is such a link?"

To make all of this a bit clearer, here is a plot that shows what is going on quite well:

• Welcome to Cross Validated. To be clear, your question is not about the underlying issue (the structure of the matrix) but rather just about ideas on how to more descriptively present the situation? If it is the former, then I suspect you might want to post additional information about the structure of the matrix. If it is the latter, is there any reason why you wouldn't simply look at the data on each side of the "cliff" separately (in the framework of a linear regression, that would be equivalent to using a dummy)? – Candamir Nov 10 '18 at 20:10
• You're right, my question is not about the structure of the matrix, but about how I can correctly calculate a linear regression on this type of data. I thought about just cutting it up into two pieces, but that feels like cheating the math and cheating the math is not really a good idea I think... – Peiffap Nov 10 '18 at 20:21
• @Candamir is right. Creating two regressions would be appropriately adapting your analysis to the realities inherent in the data. Not doing so would be trying to shoehorn two obviously distinct relationships into one equation. But slightly different on the surface than creating two regressions, and perhaps more tolerable to you, would be fitting an interaction term (@Candamir's "dummy") into a single regression. – rolando2 Nov 10 '18 at 21:41
• Sounds good to me! I'm not sure as to how to do so though, but if you or @Candamir could do that for me, I'll accept it as answer! – Peiffap Nov 10 '18 at 21:44

1 Answer

I don't know how to do this in Python but I can explain you how to do it in theory. (If anything of the below is unclear, I suggest you google "regression dummy variable".)

Your regression currently looks like this:

$$y = \beta_0 + \beta_1x + \epsilon$$

(In your specific case, k(A) is $$y$$ and clscale is $$x$$.) Next you create the dummy variable $$d$$, which is 0 if clscale is smaller than the threshold value (it seems to be somewhere between 30 and 35) and 1 else. Next you run the following regression:

$$y = \beta_0 + \beta_1d+\beta_2x+\beta_3(x*d)+\epsilon$$

Now, based on whether $$d$$ is 0 or 1, your coefficient estimates can be interpreted as follows:

• In the segment where clscale is below the threshold value, $$\hat{\beta_0}$$ is the intercept of the regression line and $$\hat{\beta_2}$$ is the slope of the regression line. Because $$d$$ is 0 in this part of the sample, the terms with $$\beta_1$$ and $$\beta_3$$ just drop away.
• In the segment where clscale is above the threshold value, $$(\hat{\beta_0}+\hat{\beta_1})$$ is the intercept of the regression line (because $$d=1$$ means that $$\beta_0+\beta_1d=\beta_0+\beta_1$$) and $$(\hat{\beta_2}+\hat{\beta_3})$$ is the slope of the regression line (because $$d=1$$ means that $$\beta_2x+\beta_3(x*d)=(\beta_2+\beta_3)x$$).

You could now use these four numbers (intercept and slope, 2x each) and use them to plot the two regression lines. You will see that they fit each of the two "groups" much better than your current regression line.

• Yes, this looks like it would be very easy to implement, and it's quite close to what I was thinking of doing... Thanks a lot! – Peiffap Nov 10 '18 at 23:09