# How to simultaneously fit multiple correlated linear models

I have two data sets labeled $$\alpha$$ and $$\beta$$, with corresponding models that I want to fit to. $$\textbf{Y}_{\alpha} = \textbf{X}_{\alpha}\cdot\vec{\theta}_{\alpha} + \epsilon \\ \textbf{Y}_{\beta} = \textbf{X}_{\beta}\cdot\vec{\theta}_{\beta} + \epsilon$$

Here the form of the design matrix $$\textbf{X}$$ is the same for both $$\beta$$ and $$\alpha$$. I expect $$\theta_{\alpha}$$ to be different from $$\theta_{\beta}$$. However, I know that data sets $$\alpha$$ and $$\beta$$ are highly correlated. I want to take advantage of this correlation to simultaneously determine $$\theta_{\alpha}$$ and $$\theta_{\beta}$$. Is there a way to do this?

• You need to give some context. What does this variables represent? In what way are they correlated? The residuals? .... Feb 3, 2020 at 12:55
• What about pooling the data and then fitting a mixed-effects model with random effects for the groups identified by data set? Feb 3, 2020 at 13:53
• To answer @kjetilbhalvorsen it is difficult to explain what the variables represent. Let's just say I am trying to apply linear models to two very similar (but different) quantities computed from Monte Carlo Simulations. Any concern about autocorrelation has already been addressed through bootstraping Monte-Carlo samples. Feb 3, 2020 at 18:35
• To answer @ulfelder, I am not sure exactly what you mean. I have never heard of a mixed-effects model. I proposed a solution below. Let me know what you think. Feb 3, 2020 at 18:35

I think I have come up with a solution.

I think I can group the data in the following way

$$\vec{\textbf{y}}= \begin{pmatrix} \textbf{Y}_{\alpha}\\\textbf{Y}_{\beta} \end{pmatrix} = \begin{pmatrix} \textbf{X}_{\alpha} & 0\\ 0 & \textbf{X}_{\beta} \end{pmatrix}\cdot \begin{pmatrix} \theta_{\alpha} \\ \theta_{\beta} \end{pmatrix}$$

where $$\vec{\textbf{y}}$$ = $$\{y_{\alpha_1}, ...,y_{\alpha_{N}}, y_{\beta_1},...,y_{\beta_M}\}$$

Then my covariance matrix $$\Sigma$$ is an $$(N+M)\times (N+M)$$ matrix.

So the optimal fit parameters are

$$\begin{pmatrix} \theta_{\alpha} \\ \theta_{\beta} \end{pmatrix}^{\text{best}} = \Bigg( \begin{pmatrix} \textbf{X}_{\alpha} & 0\\ 0 & \textbf{X}_{\beta} \end{pmatrix}^{\dagger} \Sigma^{-1} \begin{pmatrix} \textbf{X}_{\alpha} & 0\\ 0 & \textbf{X}_{\beta} \end{pmatrix} \Bigg)^{-1} \begin{pmatrix} \textbf{X}_{\alpha} & 0\\ 0 & \textbf{X}_{\beta} \end{pmatrix}^{\dagger}\Sigma^{-1}\vec{\textbf{y}}$$

Constructing projection matrices to get $$\theta_{\alpha}$$ or $$\theta_{\beta}$$ are easy to see.

• That assumes you know $\Sigma$ which is usually not the case. Getting an estimate of $\Sigma$ standardly involves putting restrictions on the covariance structure. So I think @kjetils question: In what way are they correlated? remains relevant. But yes Generalized Least Squares is an estimation method. Feb 15, 2020 at 13:11