How to properly utilize lag and errors in Time Series modelling I have a dataset of 2 variables that should be heavily correlated.
There are some underlying reasons why this set has an R^2 of only 0.620 when modeled in a simple Linear Regression; the independent variable data has 'bled' into itself by the way it has been produced.
What I'd like is to see if I can use autocorrelation to help find some errors I can propagate into the independent variable to resolve some of these issues but being quite new to these methods I'm unsure of how to implement them.
I've tried a moving 2-day average on the independent variable that I feel is causing the issue. This did increase the R^2 which adjusted to about 0.787 after averaged but I'd like to do much better than that.
I've been using this Python script below to help me understand autocorrelation better.
Here is the dataset I've been working on including the 2-day averaging:
Test Set for West Data
What I understand so far is that autocorrelation can find a lag term to fit the least error between a set using self-similarity. What I need to figure out is how to find these errors for the length of my independent variable.
Below is my first attempt at using this script to lag my series but because it's a ML algorithm the prediction errors calculated are only the length of the test set. I'm sure it's due to my lack of understanding but how can I use these errors and propagate them into my original length 67 independent variable set to help fit my model better?

Above is the graph of the independent variable's predicted values (red) using autocorrelation on the test set (blue).
 A: Since the data has a seasonality of 5, we used seasonality of 5.
Here is a plot of the Y and X showing a strong positive relationship and some visually obvious outliers.
The ACF/PACF looks as follows:


Here is the forecast.

Here is the model. The model includes the causal plus 4 seasonal dummies and 11 outliers and a change in the seasonality for periods 2 and 3 that went down severely beginning at period 37 and 58.

Here are the residual ACF/PACF


Here is the dataset cleansed of the outliers

A: Since the data is frequency of 5, our default is daily data so I reran suppressing the automatic search for day of the week dummies.
We now have a lag of the X variable.   9 outliers were identified and a level shift higher beginning at period 38 and forward.

A: Forcing intercept equals zero (i.e. "0 +" in the regression formula) helps in reaching higher R^2. Moreover, West.Demands acf plot may suggest a regressor at lag = -2.

test_ds <- read.csv("test.csv", header = TRUE, stringsAsFactors = FALSE)
test_ds$Date <- as.Date(test_ds$Date, format=c("%Y-%m-%d"))

ll <- nrow(test_ds)-1
west_counts_lag <- c(NA, test_ds$West.Counts[1:ll])
ll2 <- ll - 1
west_demands_lag2 <- c(NA, NA, test_ds$West.Demands[1:ll2])

lm_res <- lm(West.Demands ~ 0 + West.Counts + west_counts_lag + west_demands_lag2, data= test_ds)
summary(lm_res)

Call:
lm(formula = West.Demands ~ 0 + West.Counts + west_counts_lag + 
    west_demands_lag2, data = test_ds)

Residuals:
   Min     1Q Median     3Q    Max 
-42281  -2333   1324   4915  13886 

Coefficients:
                  Estimate Std. Error t value Pr(>|t|)    
West.Counts       75.39004    6.17358  12.212  < 2e-16 ***
west_counts_lag   59.09928    7.51237   7.867  6.1e-11 ***
west_demands_lag2 -0.07648    0.03980  -1.922   0.0591 .  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 8295 on 63 degrees of freedom
  (2 observations deleted due to missingness)
Multiple R-squared:  0.9747,    Adjusted R-squared:  0.9735 
F-statistic: 809.8 on 3 and 63 DF,  p-value: < 2.2e-16

Diagnostic plots:




