5
$\begingroup$

I have a question in the area of meta research. I have a dataset that consists of regression data of several economics papers. More explicitly, I have the values of the regression coefficients, the standard errors, the sample sizes and the significance levels.

My research question is: Do the effect sizes get smaller over the years? This has to do with the fact that by now we have Big Data which yields smaller effect sizes but more significance.

Unfortunately, I don't know if the coefficients are standardized. Is there a method or a formula that allows to pool the different regression coefficients and somehow measure their magnitude over the years? I know that usually one would use Cohen's Kappa in meta research in order to compare effect sizes. But since this is no test between two samples but rather a regression, Cohen's Kappa is not applicable.

This is what I have so far: Considering that I have the unstandardized coefficients and standard errors, I have the respective t-values by dividing the two values. These values have to be the same as for the quotient of the standardized coefficients and standard errors:

$\frac{b_1}{se_{x_1}} = t = \frac{\hat{b_1}}{\hat{se_{x_1}}}$

Also, the standardized coefficient can be derived by the unstandardized coefficient and the sample deviations of y and x_1 (which are unfortunately unknown):

$\hat{b_1}= b \frac{\sigma_{x_1}}{\sigma_{y}}$

Lastly, I know that the mean of the standardized variables is 0 and the standard deviation is 1. Any ideas on how to proceed or on a new approach are welcome! Thanks in advance!

By surprise I found a paper that might actually answer my own question: Doucouliagos wrote something about a partial correlation coefficient here.

It is calculated the following way:

$r = \frac{t}{\sqrt{t^2 + df}},$

where df are the degrees of freedom and $t$ is the t-value. Since I sometimes have the t-value and in other cases I can estimate it via dividing the coefficient by the standard error, and since I have also the observations as an estimate for $df$, I can calculate this $r$. What do you think about it? Would this be a good measure to check if the effect sizes have increased in time?

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ +1 Beware: there are plenty of reasons to expect effect sizes to decline over time even without the possibility of collecting larger datasets. There are some special subtleties with regression: except for associations that are truly linear across the full range of values of explanatory variables, the coefficients will not even be comparable from one study to the next. This will especially be the case for standardized coefficients. Coefficients won't be comparable, either, unless either (a) the regressor is orthogonal to all others or (b) is invariably included with the same other regressors. $\endgroup$
    – whuber ♦
    Commented Aug 11, 2020 at 18:52
  • $\begingroup$ @whuber Thank you for your answer! Which other reasons for expecting effect size to decline? $\endgroup$
    – Irazall
    Commented Aug 11, 2020 at 22:06
  • $\begingroup$ I added some thoughts that I already have gone through. Maybe that helps... $\endgroup$
    – Irazall
    Commented Aug 11, 2020 at 22:06
  • 2
    $\begingroup$ Effect size will decline when, for instance, initial studies are exploratory and small: only large effects will get published and chances are they are larger than the true effects (due to publication bias). Larger follow-up studies ought to converge downwards towards the true effects. In other words, observed effect size in publications focusing on results with low p-values (which is the norm) is confounded with study size and power. $\endgroup$
    – whuber ♦
    Commented Aug 12, 2020 at 13:02

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.