If one has multiple similar dependent variables for which one wants to fit linear regressions

y1 = c11*x1 + c10 + e1
y2 = c21*x2 + c20 + e2
y3 = c31*x3 + c30 + e3

one can estimate the regressions separately for y1, y2, y3 or one can pool the data for y1, y2, y3 and estimate a single regression. This forces c11=c21=c31 and c10=c20=c30. A middle ground is to estimate the regressions separately but with penalties on the variance of [c11 c21 c21] and [c10 c20 c30]. The larger the penalties, the closer one gets to a pooled regression.

Does this model have a name? Should one use cross-validation to determine the sizes of the variance penalties?

  • $\begingroup$ Could you explain what you mean by "variance" in this context? Would that refer to the sampling variance of the coefficient estimates or would it simply be the mean squared deviation of any three-vector relative to the mean of its components? $\endgroup$ – whuber Jun 24 '19 at 12:52
  • $\begingroup$ @whuber -- the latter, the variance of the N-vector, where N is the number of regressions being estimated $\endgroup$ – Fortranner Jun 24 '19 at 14:35
  • $\begingroup$ Thank you. Please note that the "middle ground" does not reduce to a pooled regression as penalties increase, because it's a different model: the pooled regression assumes all error variances are the same, whereas the separate regressions allow the error variances to differ from one group to another. $\endgroup$ – whuber Jun 24 '19 at 14:54

Here's an approach I might take, which relies on ridge regression.

First, pool all the data, then create dummy variables $d_1$ to $d_3$ to represent which sample a given row comes from. Then, run a ridge regression with the following coefficients:

$y = c_{\_1}x + c_{11}d_1x + c_{21}d_2x + c_{31}d_3x + c_{\_0} + c_{10}d_1 + c_{20}d_2 + c_{30}d_3 + e$

In this model, the $c_{.1}$ coefficients represent the difference between the overall slope $c_{\_1}$ and the sample-specific slopes. If these are regularized to zero, then the samples will just have the overall slope. The $c_{.0}$ coefficients represent the difference between the overall intercept $c_{\_0}$ and the sample-specific intercepts. A ridge regression puts a penalty on the $l_2$ norm (i.e., the variance) of the coefficients. We put the penalty on all the coefficients except $c_{\_1}$ and $c_{\_0}$ because we're concerned with regularizing the deviations from these overall coefficients rather than the overall coefficients themselves.

So, our ridge regression loss function looks like this:

$L =\frac{1}{n}\sum_{i}^n{(y - c_{\_1}x - c_{11}d_1x - c_{21}d_2x - c_{31}d_3x - c_{\_0} - c_{10}d_1 - c_{20}d_2 - c_{30}d_3)^2} + \lambda (c_{11}^2+c_{21}^2+c_{31}^2+c_{10}^2+c_{20}^2+c_{30}^2)$

You choose the values of the coefficients that minimize $L$ for a given value of $\lambda$, which you can choose using a variety of methods. Finally, you can get the sample-specific slopes and intercepts by adding the overall slope and intercept to the sample-specific slope and intercept for each sample.

| cite | improve this answer | |
  • $\begingroup$ (+1) Just out of curiosity: Wouldn't lasso regression be useful because here because the coefficients can get reguarized to 0? As far as I know, the coefficients in rdige will never be exactly zero, right? $\endgroup$ – COOLSerdash Jun 24 '19 at 8:33
  • 1
    $\begingroup$ Yes, lasso would definitely work here too. I only mentioned ridge because OP asked to minimize the variance of the coefficients. For readers: for lasso, replace the sum of the squared coefficients with the sum of the absolute values of the coefficients. $\endgroup$ – Noah Jun 24 '19 at 16:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.