I am just trying to understand what the relationship is between a normal multiple/simple regression vs. multiple/simple regression when the variables are differenced.

For example, I am analyzing the relationship between deposit balance ($Y_T$) vs. market rates ($R_T$) If I run a simple linear regression, the correlation is negative and pretty significant (around -.74) However, if I take the log and difference of the dependent variable and the difference of the independent variable, so my equation is now $d\, \ln(Y_T)$ being regressed with $d\, R(T)$, my correlations and R^2's are not significant at all ($R^2 = .004$).

I was just wondering whether this low $R^2$ even mean anything? Does it mean my model isn't a good fit, or do I ignore the $R^2$ when I am looking at differenced data? I know from the data there is a significant correlation between the original two variables, yet for my model I need to look at the variables differenced, so just wondering how to go about this.


The simple version is that any two variables that tend to change in one direction over time will appear to be correlated, whether there's any connection between them or not. Consider the following variables:

time = seq(from=1, to=100, by=1)
x  = .5 + .3*time +        rnorm(100)
y1 =  3 + .3*time +        rnorm(100)
y2 =  7 + .1*time + .8*x + rnorm(100)

$x$ is just a function of time, as is $y1$. $y2$ is a function of both time and $x$. The point is to recognize from the code that there really is a relationship between $x$ and $y2$, and that there is no relationship between $x$ and $y1$. Now look at the following figure, all three lines look awfully similar, don't they?

enter image description here

In fact, the $R^2$ value for the relationship between $x$ and $y1$ is 98%, and the $R^2$ for $x$ and $y2$ is 99%. But we know that there's no real relationship between $x$ and $y1$, whereas there is between $x$ and $y2$, so how do we differentiate the real from mere appearance? That's where differencing comes in. For any two of the variables, since they both tend to go up over time, that's not very informative, but given that one goes up by some specific amount, does that tell us how much the other goes up? Differencing allows us to answer that question. Note the following two figures, scatterplots I made after differencing all three variables.

enter image description here

enter image description here

Here, we clearly see that knowing something about how much $x$ went up tells us something about how much $y2$ goes up ($R^2=.43$), but that this is not the case for $x$ and $y1$ ($R^2=.07$). So the answer to your question is that you should ignore the correlations amongst your original variables and look at the differenced variables. Given that your $R^2$ is .004, I would say there's no actual relationship.

Some other points: In the figures, I make a point of noting that these are simultaneous changes. There's nothing wrong with that, and it follows from the way I set up the problem, but usually people are interested in effects at some lag. (That is, change in one thing at one point in time leads to change in something else later.) Second, you mention taking the log of one of your series. Taking the log simply switches your data from levels to rates. And thus, when you difference, you are looking at changes in rates rather than changes in levels. That's very common, but I didn't include that element in my demonstration; it's orthogonal to the issues I discussed. Lastly, I want to acknowledge that time series data are often more complicated than my demonstration lets on. A comprehensive overview would require a book length treatment, but @Charlie's answer does a good job of succinctly pointing out some of the complexities that I left out.


@gung offers a nice answer, but I want to offer a few caveats to what you are suggesting.

Differencing is mostly used to combat the problem of unit roots, for example, when the process is AR(1) with a correlation coefficient of 1. Differencing can be used effectively to remove a linear time trend when the error term is white noise (in particular, it exhibits no serial correlation), as @gung shows above. But, if the error term has serial correlation with a correlation coefficient less than 1 in absolute value, using differencing to remove a linear time trend produces errors with a very complicated structure. It is difficult to obtain accurate standard errors and make valid inferences in this case.

As a result, it is best to test for a unit root first and, if one is detected, to fix that via differencing. Next, check for a linear time trend. Fix this problem by detrending. Without doing the latter, you are open to the omitted variables-type problem that @gung nicely illustrates.

  • 1
    $\begingroup$ +1 This is a nice complement to my answer. I tried to keep my answer simple and intuitive. However, it is certainly true that there are more complexities than I discussed, and that they can be very important. I should have acknowledged that in my final paragraph. Thanks for keeping me honest. $\endgroup$ – gung - Reinstate Monica Dec 10 '11 at 19:39

When the objective is to form/identify the relationship between two or more series , one might need to filter the stationary X variable in order to transform it to noise. This is a two step process , the differencing required and the ARMA structure . To retain objectivity and to avoid Model Specification Bias one should not assume the filter but rather construct that filter using the autocorrelative nature of the stationary X series. Then one takes the Y series and applies whatever differencing operators are necessary to make it stationary and then apply the previously developed filter to the stationary Y . This procedure has one and only one objective and that is to identify the relationship between between Y and X. One should never jump to conclusions about the required differencing operators, the ARMA filter and the relationship between the variables unless one is an econometrician who knows the model before they observe the data or if you speak directly to the almighty. Careful analysis regarding the normality of the errors requirement are necessary to believe any statistical test that may be computed. Computation of the F tests/ T tests is necessary but not sufficient. In summary I suggest that you pursue the subject of "How to Identify a Transfer Function Model" . Others and I have addressed this subject a number of times. If you wish you might peruse some of the answers to questions that have the tag "time series" attached to them. As Yogi said "You can observe a lot by simply reading / watching". Sometimes nice and simply answers can lead you astray and potentially overcomplicated/conservative answers like mine might require you to develop a better understanding of modelling time series data. As was once said "Toto, we are not in Kansas (i.e. cross-sectional data) anymore !"


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.