1
$\begingroup$

General question:

What does it mean when the results of an OLS regression (or fixed effects regression) are unstable (i.e., the coefficients change and become significant or insignificant) considering different time periods or more specifically extending the observed time/adding more observations?

My specific case:

I am interested in the length similarity between a set of texts after a treatment (also a text). For this I take the absolute difference between the length of the treatment text and the texts before and after the treatment. The absolute difference is the dependent variable and my independent variables are if a treatment exists or not (let's call it 'After'), how many texts there are as a control and the entity and time fixed effects since I have panel data. That is, a negative coefficient would imply that the texts after the treatment are more similar to the treatment text than those before it. However, my results are highly unstable, that is the significance of the coefficent of the variable 'After' changes for different time periods. For example, the coefficent is insignificant if I only consider the last month before and first one after the treatment. Then when I compare the last month before and the second month after the treatment (i.e., leave out the first month after the treatment considering it may take time until the treatment becomes effective) the coefficent is positive and significant. When I add another month so that I compare the last two months before the treatment and month 2 and 3 after the treatment, the coefficient is negative and insignficant. But extending it again by looking at the last three month before the treament and month 2,3, and 4 the coefficient is again negative but highly significant.

Edit:

This is the equation which I specified in R:

m1 = feols(length_diff ~ after +text_volume +i(month) +i(relative_month) | id, text_data, vcov = ~id)

(The feols function is from the fixest package.) Length_diff refers to the absolute difference between the length (word count) of a text i for an entity r in month t and the treatment text for entity r, after refers to if there was a treatment or not at the time of the submission of text i, text_volume to the volume of text in month t to account for the differences in volume after and before the treatment, month to the month t in which the text i was submitted and relative_month to the months from the start of the text submission for an entity until the submission month of text i.

This is the output I get for the first month after and before the treatment:

The variables 'month::9', 'month::14' and three others have been removed because of collinearity (see $collin.var).
> etable(m1)
                                  m1
Dependent Var.:          length_diff
                                    
after                -0.4976 (1.093)
text_volume         -0.0110 (0.0138)
month = 2          -8.542*** (1.300)
month = 3          -6.001*** (1.029)
month = 7            -0.7695 (1.013)
month = 8             0.2051 (1.313)
month = 12          14.19*** (1.013)
month = 13         -18.46*** (1.346)
month = 15           -9.523* (4.291)
month = 16           -8.375* (3.678)
month = 17          -12.05** (3.585)
month = 18           -5.742. (3.142)
month = 19            3.547. (1.862)
month = 20         -10.60*** (2.480)
month = 21          -6.891** (2.315)
month = 22         -6.961*** (1.506)
month = 23            0.7266 (1.979)
month = 24             2.164 (1.883)
month = 25             4.480 (4.733)
month = 26             2.973 (3.113)
month = 27            7.699* (3.513)
month = 28            7.883. (4.247)
month = 29            7.332. (3.951)
month = 30           -0.0957 (4.259)
month = 31            -9.234 (5.653)
month = 32             1.639 (1.259)
month = 33          11.30*** (1.680)
month = 36           -9.039* (3.645)
month = 37           -6.076. (3.271)
month = 38             2.057 (2.126)
month = 39           -4.474* (1.770)
month = 41             7.141 (9.976)
month = 42            -12.47 (9.736)
month = 43            -3.051 (9.728)
month = 44            -1.422 (9.379)
month = 45            -7.406 (9.475)
month = 46           -8.600* (3.325)
month = 47          -7.265** (2.125)
month = 48         -4.086*** (1.055)
relative_month = 1    -1.716 (1.678)
relative_month = 2  -5.651** (1.619)
Fixed-Effects:     -----------------
id                               Yes
__________________ _________________
S.E.: Clustered            by:    id
Observations                   5,353
R2                           0.93937
Within R2                    0.00890
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

This is the output I get for the last month before and the second month after the treatment:

The variables 'month::11', 'month::36' and two others have been removed because of collinearity (see $collin.var).
> etable(m1)
                                  m1
Dependent Var.:          length_diff
                                    
after               20.23*** (2.014)
text_volume         -0.0002 (0.0094)
month = 3          -12.77*** (1.152)
month = 4          -27.64*** (2.396)
month = 5          -31.33*** (2.686)
month = 8           21.41*** (2.656)
month = 9           12.62*** (2.091)
month = 10             1.969 (1.464)
month = 13          204.2*** (19.49)
month = 14          180.2*** (20.67)
month = 15          205.3*** (19.23)
month = 16          171.7*** (18.73)
month = 17          154.3*** (17.50)
month = 18          142.6*** (17.02)
month = 19          130.2*** (16.03)
month = 20          134.9*** (15.64)
month = 21          117.7*** (13.66)
month = 22          98.73*** (11.91)
month = 23          94.45*** (10.64)
month = 24          90.51*** (9.757)
month = 25          80.60*** (8.765)
month = 26          74.31*** (7.799)
month = 27          61.79*** (7.249)
month = 28          57.95*** (6.188)
month = 29          46.78*** (5.209)
month = 30          42.08*** (4.602)
month = 31          31.43*** (4.787)
month = 32          36.95*** (2.183)
month = 33          23.58*** (2.014)
month = 34          21.77*** (1.764)
month = 35          14.17*** (3.372)
month = 37          134.4*** (12.37)
month = 38          126.8*** (14.59)
month = 39          120.1*** (10.37)
month = 40          115.7*** (9.989)
month = 41          99.87*** (8.591)
month = 42          97.03*** (7.588)
month = 43          75.69*** (8.110)
month = 44          66.74*** (6.886)
month = 45          60.77*** (6.325)
month = 46          51.26*** (5.655)
month = 47          43.70*** (4.138)
month = 48          30.48*** (3.580)
month = 49          24.77*** (2.649)
month = 50          17.37*** (3.168)
relative_month = 1    -2.519 (2.670)
relative_month = 2    4.064* (1.802)
relative_month = 3 -6.810*** (1.460)
Fixed-Effects:     -----------------
id                               Yes
__________________ _________________
S.E.: Clustered            by:    id
Observations                   5,476
R2                           0.94506
Within R2                    0.00993
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Here is a small simulated sample:

sample <- data.frame(id=c(1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4), after=c(0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1), month=c(15,15,15,15,15,15,15,15,15,15,15,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,17,17,17,17,17,17,17,17,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,22,22,22,22,22,22,22,22,22,22,22,22,22,22,25,25,25,25,25,25,25,25,25,25,25,25,25,25,25,25,25,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,26,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,27,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,3,3,3,3,3,3,3,3,3,3,3,3,3), relative_month=c(0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2,2,2,2), length_diff=c(206,207,206,199,182,207,163,135,200,203,207,165,201,189,203,197,159,130,184,198,208,208,207,203,201,184,206,181,203,201,204,198,203,115,186,188,200,156,120,107,128,103,92,50,88,107,43,107,136,134,93,97,123,132,124,124,75,39,101,83,130,135,69,135,107,132,78,127,124,134,126,100,124,124,117,135,64,127,120,116,130,131,127,101,134,125,138,98,34,125,138,122,98,134,10,86,124,119,90,81,132,77,139,118,105,115,63,135,135,129,135,131,119,131,122,127,131,86,126,65,118,129,137,135,137,128,75,75,138,124,70,107,101,125,86,99,86,15,130,133,126,128,137,98,101,85,145,129,48,106,35,106,35,127,135,128,54,88,136,107,120,123,136,122,131,37,117,136,132,124,55,109,134,131,133,144,106,112,116,122,124,169,170,135,165,119,151,138,167,175,161,176,164,173,151,150,172,174,173,172,174,68,162,156,177,160,149,126,175,150,165,171,164,174,153,173,5,125,87,54,174,169,173,159,174,167,168,173,167,27,157,179,175,94,157,169,97,29,158,171,156,170,163,158,112,155,170,173,167,116), text_volume=c(11,11,11,11,11,11,11,11,11,11,11,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,19,8,8,8,8,8,8,8,8,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,21,28,28,28,28,28,28,28,28,28,28,28,28,28,28,28,28,28,28,28,28,28,28,28,28,28,28,28,28,14,14,14,14,14,14,14,14,14,14,14,14,14,14,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,17,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,14,14,14,14,14,14,14,14,14,14,14,14,14,14,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,13,13,13,13,13,13,13,13,13,13,13,13,13))
$\endgroup$
2
  • $\begingroup$ This is a little hard to follow. Could you clarify it by adding the equations for the models you fitted, and perhaps a sample of your data & the model output? A simulated example would do if you don't feel like sharing the data. $\endgroup$
    – mkt
    Aug 20, 2023 at 10:10
  • 1
    $\begingroup$ @mkt I added my equation, my model output and also a small sample that is similar to my data. I hope this makes it clearer. I would be happy about any suggestions :) $\endgroup$
    – Mina
    Aug 20, 2023 at 11:37

1 Answer 1

1
$\begingroup$

What does it mean when the results of an OLS regression (or fixed effects regression) are unstable (i.e., the coefficients change and become significant or insignificant)

It can mean two things:

  • The analysis as a whole is insignificant and noisy, and the observed variations in the parameters are dominated by noise.

    (and occasionally you get a significant parameter, potentially you better analyse the significance for all times together, https://en.m.wikipedia.org/wiki/Multiple_comparisons_problem).

  • The model parameters are not constant in time.

    (and the variations in significance can be because of the parameters are occasionally small and close to zero or the noise being variable, a plot of parameters in time with additional standard errors might show better what is happening).

It is not restricted to these two cases and possibly more is happening. For example your analysis can have a varying bias.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.