6
$\begingroup$

I would appreciate your help with this problem. I am trying to replicate this study which is about finding the optimal level of bank capital. All this is done using Stata. The only difference is that I have quarterly data, and the authors have semi-annual data.

We have a problem with the following regression:

Beta=alpha(i) + X(i,t-1)b + u(it)

X(i,t-1) represents a very much documented important predictor of beta. The process of estimation is the following:

  1. Perform a unit-root test to make sure beta and X do not have a spurious link.

    We performed the test and we reject the H0, therefore all good up to here.

  2. Perform the regression using OLS, Fixed Effects and Random Effects. When we run the regressions, following every single step indicated in the study, we get that X is not significant. Not even at the 80% level! For any of the three methods. (Please see from the end of page 13 to page 15 for more details about the regression).

Our panel data looks something like this:

Panel data

According to what is indicated in the paper, this is the regression we run:

. xtset bank quarters, quarterly

panel variable:  bank (strongly balanced)
time variable:  quarters, 1998q1 to 2013q1
delta:  1 quarter

--

. xtreg beta ib(first).year LD.leveragefull, fe vce(cluster bank)

And this is what we get:

    Fixed-effects (within) regression               Number of obs      =       295
Group variable: bank                            Number of groups   =         5

R-sq:  within  = 0.2965                         Obs per group: min =        59
       between = 0.0002                                        avg =      59.0
       overall = 0.1387                                        max =        59

                                                F(4,4)             =         .
corr(u_i, Xb)  = 0.0000                         Prob > F           =         .

                                   (Std. Err. adjusted for 5 clusters in bank)
------------------------------------------------------------------------------
             |               Robust
        beta |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        year |
       1999  |  -.2352546   .0679795    -3.46   0.026    -.4239961   -.0465131
       2000  |  -.3410705   .1256874    -2.71   0.053    -.6900346    .0078936
       2001  |  -.2187538   .1399625    -1.56   0.193    -.6073519    .1698443
       2002  |  -.2628397    .094347    -2.79   0.050     -.524789   -.0008903
       2003  |  -.1413272    .050201    -2.82   0.048    -.2807076   -.0019469
       2004  |  -.2506764   .0697621    -3.59   0.023     -.444367   -.0569858
       2005  |  -.3280419   .1220112    -2.69   0.055    -.6667993    .0107156
       2006  |  -.2976486   .0996758    -2.99   0.040     -.574393   -.0209043
       2007  |  -.1901107   .1236727    -1.54   0.199    -.5334811    .1532598
       2008  |  -.1407684   .0840705    -1.67   0.169    -.3741854    .0926487
       2009  |  -.0503666   .0505586    -1.00   0.376    -.1907398    .0900066
       2010  |  -.0764699    .057999    -1.32   0.258    -.2375011    .0845613
       2011  |  -.0035166   .0720702    -0.05   0.963    -.2036155    .1965823
       2012  |   .0006204   .0492506     0.01   0.991     -.136121    .1373619
       2013  |   .3996657   .0604211     6.61   0.003     .2319099    .5674215
             |
leveragefull |
         LD. |   .0047472   .0068057     0.70   0.524    -.0141484    .0236428
             |
       _cons |   .5461694    .069342     7.88   0.001     .3536452    .7386935
-------------+----------------------------------------------------------------
     sigma_u |  .29632318
     sigma_e |  .21633914
         rho |   .6523096   (fraction of variance due to u_i)
------------------------------------------------------------------------------

The authors of the study, on the other hand, get very significant values for leveragefull (X), and this makes absolute sense with the theory and practice.

So the question is, what am I doing wrong? What can be the problem? How can I fix either the data or the regression in order to increase the significance of the results?

$\endgroup$
1
  • 1
    $\begingroup$ I wonder if quarterly data is right. Why won't you try with t-2, it equals the lag noticed in the authors study. $\endgroup$
    – user28332
    Jul 22 '13 at 16:05
4
$\begingroup$

One thing you can do is look at the effect sizes and build a confidence interval around them. “Very” significant coefficients do not necessarily represent strong effects and test results depend a lot on the sample size. Therefore, a failure to reject the null hypothesis is not as such evidence that your results are inconsistent with the previous study (Andrew Gelman regularly puts it that way: “the difference between significant and non-significant is not itself significant”).

Basically the more data you have, the more confidence you can have in the fact that a given coefficient is different from 0 but, again, that's not a measure of the strength of the relationship between the variables. If the confidence intervals overlap or the coefficient or effect size measures are similar but one is not significant, it simply means that one of the studies had less power. If that's the case, one way to “fix” it is simply to collect more data.

Also, I know nothing of your field and I am not sure to follow exactly what you are doing but generally speaking one published study would not be enough to convince me that something definitely should be significant, no matter what the relevant p value was. If you have strong theoretical reasons to expect it to be that's another story but you should not overestimate the reproducibility of many published results (see for example John Ioannidis's publications).

$\endgroup$
2
  • $\begingroup$ Thank you for your answer @Gaël. I must admit I'm not so sure how to implement your recommendation in the first paragraph, but I have found this Do you think the lack of other explanatory variables could be the reason of my problem? Thanks. $\endgroup$ Jul 20 '13 at 21:32
  • $\begingroup$ It certainly can, if the model is not the same, any comparison is even more difficult. $\endgroup$
    – Gala
    Jul 20 '13 at 22:42
2
$\begingroup$

After reading more I found out that the values are not significant because the fixed effects model does not work well with the data. I have a high heterskedasticity, which is not fixed by transforming the data into its log form. So far, the OLS with Robust Error adjustment for heteroskedasticity is what is giving us the best results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.