6
$\begingroup$

I am using Stata and used the regress command and received $p$ values that indicate the regressor is statistically significant.

However, after plotting the residuals, I noticed there was clearly an autocorrelation (which was confirmed by the Breusch-Godfrey test). I then used the prais ..., corc command to correct it, but got the result that now my regressor is statistically insignificant.

What does this mean? Is my regression useless?

$\endgroup$
2
  • 7
    $\begingroup$ Isn't it interesting how as an applied scientist, the more you know about statistics, the more difficult it is to get a publishable statistical result? I often wonder about what effect this has on what kinds of people get jobs/tenure. But anyways, I would try to find a way of presenting both results that sounds interesting and maybe even can challenge some existing beliefs, e.g. "At first glance, it looks like X has an effect on Y as previously reported in the literature(?), but actually we found that alleged relationship vanishes when properly accounting for autocorrelation". $\endgroup$ Commented Apr 7 at 19:19
  • $\begingroup$ A related puzzle in the opposite direction: stats.stackexchange.com/questions/643152 $\endgroup$ Commented Apr 8 at 9:31

2 Answers 2

10
$\begingroup$

One possibility is that both your dependent and independent variables are related to time. This is the source of many humorous correlations such as: Ice cream sales go up when sharks attack or children's vocabulary is related to their weight.

I don't think we can say, on the information given, whether this is what's going on here. But you can investigate it by looking at the relationship of your variables to time.

$\endgroup$
1
6
$\begingroup$

Autocorrelation Influences

My answer is a bit of a two-parter, the second part largely being more important than the first. First, a bit about autocorrelation...

Consider the following data that I just simulated in R, which has a substantial autocorrelation:

enter image description here

If we look at each "zig" or "zag" in this plot, we see generally that each original value is heavily weighting where the next point will go. If there was no relationship we would instead have something completely random like this:

enter image description here

One can see that autocorrelation then implies a heavy dependence between data points, and will consequently have a large bearing on the $p$ values. Leaving them unaccounted for will provide only biased results, and thus it is no surprise that your $p$ values change in some way based of that modeling decision.

On Reporting Null Effects

This isn't an annoyance with you, but this statement really drives me crazy because of how often I see it:

What does this mean? Is my regression useless?

I blame this on how badly most methods courses are taught in grad school, but there are two major issues with this statement.

  1. Your analysis doesn't increase/decrease in importance due to statistical significance.
  2. Getting an unexpected result is not a failure.

To the first point, under the NHST paradigm, a $p$ value simply states the probability of a given estimate given the null hypothesis is true. If a $p$ value is higher than some arbitrary alpha cutoff, this means we cannot reject the null hypothesis, but neither can we accept the null in lieu of this information. One suspends judgement until further research is committed to answering those questions. The null hypothesis anyway is sometimes called the "dull hypothesis", because most people do not believe there can be an absolutely zero effects anyway.

For the second point, unexpected results give you a unique opportunity: a surprising result is an interesting result! What often confuses us should motivate us to find out more. Why was my data not to my expectation? Was it because there is perhaps an angle I haven't investigated? Was it because of sample-level characteristics? These are the sorts of questions one should really be asking when this happens.

As an example of how even "useless" results can be interesting, the inventor of the Post-It note was originally testing what was supposed to be an adhesive that should have been stronger than any other before it. But to his dismay, the objects which he applied the adhesive to could be easily removed. After a lot of brainstorming, this later led to the invention of Post-It notes when considering paper-based applications like bookmarks.

Finally, I strongly suggest that you not only plot your data, but also look at the coefficients, effect size, etc. I sincerely doubt your predictors have no association with the outcome. This is the danger of $p$ values. People get so blinded by them that that don't actually investigate whether or not there is a positive/negative association and by how much. This is equally, if not more important, than $p$ values alone.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.