I have two data sets labeled $\alpha$ and $\beta$, with corresponding models that I want to fit to. $$ \textbf{Y}_{\alpha} = \textbf{X}_{\alpha}\cdot\vec{\theta}_{\alpha} + \epsilon \\ \textbf{Y}_{\beta} = \textbf{X}_{\beta}\cdot\vec{\theta}_{\beta} + \epsilon $$

Here the form of the design matrix $\textbf{X}$ is the same for both $\beta$ and $\alpha$. I expect $\theta_{\alpha}$ to be different from $\theta_{\beta}$. However, I know that data sets $\alpha$ and $\beta$ are highly correlated. I want to take advantage of this correlation to simultaneously determine $\theta_{\alpha}$ and $\theta_{\beta}$. Is there a way to do this?

  • $\begingroup$ You need to give some context. What does this variables represent? In what way are they correlated? The residuals? .... $\endgroup$ Feb 3, 2020 at 12:55
  • $\begingroup$ What about pooling the data and then fitting a mixed-effects model with random effects for the groups identified by data set? $\endgroup$
    – ulfelder
    Feb 3, 2020 at 13:53
  • $\begingroup$ To answer @kjetilbhalvorsen it is difficult to explain what the variables represent. Let's just say I am trying to apply linear models to two very similar (but different) quantities computed from Monte Carlo Simulations. Any concern about autocorrelation has already been addressed through bootstraping Monte-Carlo samples. $\endgroup$ Feb 3, 2020 at 18:35
  • $\begingroup$ To answer @ulfelder, I am not sure exactly what you mean. I have never heard of a mixed-effects model. I proposed a solution below. Let me know what you think. $\endgroup$ Feb 3, 2020 at 18:35

1 Answer 1


I think I have come up with a solution.

I think I can group the data in the following way

$$ \vec{\textbf{y}}= \begin{pmatrix} \textbf{Y}_{\alpha}\\\textbf{Y}_{\beta} \end{pmatrix} = \begin{pmatrix} \textbf{X}_{\alpha} & 0\\ 0 & \textbf{X}_{\beta} \end{pmatrix}\cdot \begin{pmatrix} \theta_{\alpha} \\ \theta_{\beta} \end{pmatrix} $$

where $\vec{\textbf{y}}$ = $\{y_{\alpha_1}, ...,y_{\alpha_{N}}, y_{\beta_1},...,y_{\beta_M}\}$

Then my covariance matrix $\Sigma$ is an $(N+M)\times (N+M)$ matrix.

So the optimal fit parameters are

$$ \begin{pmatrix} \theta_{\alpha} \\ \theta_{\beta} \end{pmatrix}^{\text{best}} = \Bigg( \begin{pmatrix} \textbf{X}_{\alpha} & 0\\ 0 & \textbf{X}_{\beta} \end{pmatrix}^{\dagger} \Sigma^{-1} \begin{pmatrix} \textbf{X}_{\alpha} & 0\\ 0 & \textbf{X}_{\beta} \end{pmatrix} \Bigg)^{-1} \begin{pmatrix} \textbf{X}_{\alpha} & 0\\ 0 & \textbf{X}_{\beta} \end{pmatrix}^{\dagger}\Sigma^{-1}\vec{\textbf{y}} $$

Constructing projection matrices to get $\theta_{\alpha}$ or $\theta_{\beta}$ are easy to see.

  • $\begingroup$ That assumes you know $\Sigma$ which is usually not the case. Getting an estimate of $\Sigma$ standardly involves putting restrictions on the covariance structure. So I think @kjetils question: In what way are they correlated? remains relevant. But yes Generalized Least Squares is an estimation method. $\endgroup$ Feb 15, 2020 at 13:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.