4
$\begingroup$

Approach 1: I regress $Y$ on $X$, obtaining estimates $\hat{\beta}$ and its covariance matrix $S$.

Approach 2: I first partition $X$ into $X_1, X_2$ using some method (e.g. k-means). Then I separately regress $Y$ on $X_1$ and $X_2$, obtaining results $\hat{\beta_1}, S_1, \hat{\beta_2}, S_2$.

Did approach 2 improve over approach 1?

I've been looking at the determinant of the covariance matrix, aka generalized variance, as a scalar measure of model performance (intuitively I want my estimates to have low variances). Since smaller samples naturally lead to higher-variance estimates, is there a way to take that into account?

$\endgroup$
4
  • 1
    $\begingroup$ No, it did not improve your model in any meaningful way: andrewgelman.com/2017/06/19/… $\endgroup$
    – mkt
    Jul 4, 2017 at 12:44
  • $\begingroup$ If the actual relationship is piece-wise linear, then surely splitting my sample at the juncture would lead to better estimates? I'm looking for a quantitative metric to use, not asking a philosophical question. $\endgroup$
    – scip
    Jul 4, 2017 at 12:47
  • 1
    $\begingroup$ Yes, but in that case fitting a linear regression was wrong in the first place, you should have fit a piecewise regression instead. So it's an unfair comparison that you are describing. Plotting the data first should let you choose if this is necessary. Or you may have strong a priori expectation for a discontinuity of some sort. $\endgroup$
    – mkt
    Jul 4, 2017 at 12:52
  • 2
    $\begingroup$ Also, whether intended as such or not, your question does get at important statistical (perhaps philosophical) concepts. I think it's worth considering them to see why this is misguided. But if all you want is a general quantitative metric of model performance, prediction error on a hold-out dataset is almost always good. $\endgroup$
    – mkt
    Jul 4, 2017 at 12:54

1 Answer 1

3
$\begingroup$

Turning my comments above into an answer:

Splitting data into subsets and fitting models to each subset cannot improve your models, parameter estimates or inferences provided your original model is properly specified. As Andrew Gelman puts it here, while considering the case of data being sampled multiple times from the same process:

From a Bayesian standpoint, the result is the same, whether you consider all the data at once, or stir in the data one-third at a time. The problem would come if you make intermediate decisions that involve throwing away information, for example if you take parts of the data and just describe them as statistically significant or not.

As @whuber points out in the comments below, the assumption that the original model (fit to the full dataset) is specified properly is crucial. If splitting the dataset and fitting models to the subsets leads to improvement, it would be evidence that the original model was misspecified. I should note that there are, of course, other ways of identifying model misspecification.

$\endgroup$
10
  • 1
    $\begingroup$ It's difficult to see how such a blanket statement could generally be true. The question seems to ask how to compare a more complex model to a simpler one in which it is nested. Would you throw away all the machinery of model selection in this case under the assumption that it's impossible for any data to conform better to the more complicated model? Counterexamples are easy to come by... . I am not gainsaying Gelman: his remarks are predicated on the assumption that the data are an exchangeable sample from the same process. $\endgroup$
    – whuber
    Aug 21, 2018 at 20:34
  • $\begingroup$ @whuber I don't think we've interpreted the question in the same way. It seems to focus on potential gains from partitioning a dataset and fitting models to subsets vs. fitting a single model to the whole dataset simultaneously. I don't think such gains are possible unless the model for the whole dataset is misspecified. If there's a question here about complex vs. simple models in general, I've entirely missed it. $\endgroup$
    – mkt
    Aug 22, 2018 at 5:32
  • 1
    $\begingroup$ I see splitting a model into subsets and performing a separate fit on each one as constituting a more complex model. As an example, suppose the model is split on the value of a binary explanatory variable: the procedure is tantamount to including that variable and all its interactions, as well as allowing for limited heteroscedasticity (the error variances in the two halves may differ). $\endgroup$
    – whuber
    Aug 22, 2018 at 11:14
  • 1
    $\begingroup$ I won't disagree with that! But it looks like we have established that your statement "... cannot improve your models" is not correct, at least not without substantial elaboration concerning your assumptions. $\endgroup$
    – whuber
    Aug 22, 2018 at 12:44
  • 1
    $\begingroup$ @whuber Agreed - thanks for this discussion. I have edited to elaborate regarding those assumptions. $\endgroup$
    – mkt
    Aug 22, 2018 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.