I have a dataset of 2 variables that should be heavily correlated.
There are some underlying reasons why this set has an R^2 of only 0.620 when modeled in a simple Linear Regression; the independent variable data has 'bled' into itself by the way it has been produced.
What I'd like is to see if I can use autocorrelation to help find some errors I can propagate into the independent variable to resolve some of these issues but being quite new to these methods I'm unsure of how to implement them.
I've tried a moving 2-day average on the independent variable that I feel is causing the issue. This did increase the R^2 which adjusted to about 0.787 after averaged but I'd like to do much better than that.
I've been using this Python script below to help me understand autocorrelation better.
Here is the dataset I've been working on including the 2-day averaging: Test Set for West Data
What I understand so far is that autocorrelation can find a lag term to fit the least error between a set using self-similarity. What I need to figure out is how to find these errors for the length of my independent variable.
Below is my first attempt at using this script to lag my series but because it's a ML algorithm the prediction errors calculated are only the length of the test set. I'm sure it's due to my lack of understanding but how can I use these errors and propagate them into my original length 67 independent variable set to help fit my model better?
Above is the graph of the independent variable's predicted values (red) using autocorrelation on the test set (blue).