I have two large datasets from nonlinear control systems. I am trying to correlate the error with the control action taken.
If the error is plotted as fx and the control action as fy we end up with this:
We can see that fx from t = 0 to t = 1244 is decreasing. It is hard to see in the figure, but from t = 1245 to t = 9500 the error is slightly increasing towards 0 but never actually achieving it. On the other hand, fy is always decreasing.
There are changes in every single observation from fx and fy but they are very subtle.
The correlation between fx and fy is very different from Pearson and Spearman's coefficients. Since, for the most part, fx increases and fy decreases, the latter coefficient produced a more expected value than Pearson's. Pearson: 0.998, Spearman: -0.699.
Just out of curiosity, Kendall tau is -0.788.
I decided then to do a scatter plot with the correlations. Points being (x, y) from fx and fy respectively:
The pearson correlation is so close to being perfectly linear that it is actually causing me to question the logic used above to explain the expected negative value and go for it.
I've read this question many times before asking this one up. I decided to do so because I'm not so sure why would Pearson's coefficient produce such a value when there is so much data indicating a negative trend.
I would rather ask more than just a single question for this:
Q1: Am I doing/assuming something wrong?
Q2: Should I stick with my logic and use Spearman's or consider Pearson's instead?
Edit: I've made a histogram of both error (fx) and control action (fy) observations. The majority of occurrences is during the stage where the error is negative and increasing and the control action is positive and decreasing. The data is really skewed, so there is a really significant concentration of points in the left tail of the Pearson plot. Is that perhaps the reason why I should use Spearman?
Pearson's left tail where lies the highest concentration of points: