1
$\begingroup$

I estimated a simple regression model twice in a randomly split sample of about $n = 800$ cases (calibration sample: $n_A = 400$; validation sample: $n_B = 400$).

I then tested whether the regression coefficients ($\beta$) obtained from OLS estimation in each separate sample differed significantly (for instance, $\beta_A = .60$ and $\beta_B = .40$). For some coefficients this is the case for others it is not.

I now look for a sound interpretation of the result that some of the coefficients differed significantly.

Could it be that there is a set of unobserved variables not included in the model that cause the differences in the coefficients between both samples? Do you have a examples where other people had similar findings?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Sorry, I'm not sure what you mean by "nearly impossible." Are you suggesting that, given random sampling, it would be "nearly impossible" to get different betas? $\endgroup$
    – user78229
    Jul 8, 2016 at 12:49
  • $\begingroup$ I am sorry for the confusion. I meant that an interpretation might be easier by knowing the theoretical model that was used in the research project. I just removed this sentence from my question. $\endgroup$
    – phx
    Jul 8, 2016 at 12:52
  • 2
    $\begingroup$ There are those that would argue in favor of an underlying theoretical model, Wooldridge in his Econometric Analysis of Cross Section and Panel Data is one of them. With that as background, omitted variables could be an issue. I prefer a different, simpler explanation which is that any observed differences are largely attributable to the vagaries of finite data samples. One assumption that is rarely discussed in the literature but is almost always worth examining concerns the stability of the parameter estimates across a series of random forest mini-models. $\endgroup$
    – user78229
    Jul 8, 2016 at 13:03
  • 1
    $\begingroup$ This would expand your two samples into many. The RFs could be bootstrapped or jacknifed but the variability in the parameters would be well summarized by a coefficient of variation. $\endgroup$
    – user78229
    Jul 8, 2016 at 13:04
  • 1
    $\begingroup$ How many coefficients do your models have? How do you set up your significance test (correct for multiple comparisons)? (And I frequently face unstable models - with clustered data, e.g. some 1000s of rows coming from say 10ish patients x 1000 variates) $\endgroup$
    – cbeleites unhappy with SX
    Jul 9, 2016 at 11:01

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.