Suppose we have two feature matrices, $X_1$ and $X_2$, with response variables $Y_1$ and $Y_2.$ Where $X_1$ and $X_2$ have the same feature columns, but distinct observations. Furthermore, $Y_1$ and $Y_2$ are continuous responses measured on different scales (i.e. the underlying distribution of $Y_1$ is different than the underlying distribution of $Y_2$). Suppose there exists a function $f_Y$ such that $Y_1 \approx f_Y(Y_2)$.

Can we combine the datasets $(X_1, Y_1)$ and $(X_2, Y_2)$ to increase the number of observations we have for regression?

For example, for the mappings $$ \begin{matrix} X_1 & \underset{(1)}{\rightarrow} & Y_1 \\ & & \downarrow\tiny(Y) \\ X_2 & \underset{(2)}{\rightarrow} & Y_2 \\ \end{matrix} $$ One could use $\underset{(1)}{\rightarrow}$ and $\underset{(2)}{\rightarrow}$ to learn $\downarrow\tiny(Y)$ and then regress using transformed $Y_2$ values to get:

$$ \left(\begin{matrix} X_1 \\ X_2 \\ \end{matrix}\right) \rightarrow \left(\begin{matrix} Y_1 \\ f_Y(Y_2) \\ \end{matrix}\right). $$

As an example of this problem in R, say we have the simulated data:

x1 = matrix(rnorm(500), nrow = 50)
x2 = matrix(rnorm(600), nrow = 60)
b1 = matrix(c(1,2,3, rep(0,7)), ncol = 1)
b2 = 2*(b1)^2
y1 = x1 %*% b1 + rnorm(50)
y2 = x2 %*% b2 + rnorm(60)

We can build simple linear regressions lm(y1~x1) and lm(y2~x2) to learn b1 and b2, but suppose we can learn a more powerful predictor if we combine the datasets.

How do we combine these data?

  • $\begingroup$ Maybe write another equation of Y1 = f(Y2) and then use SUR? $\endgroup$
    – Dayne
    Aug 28 '19 at 12:07
  • $\begingroup$ You haven't any information at all that could be used to estimate the mapping of dependent variables. To get some, you will need to make some assumptions; that is, create an explicit model of the responses. This is not a matter of combining data. $\endgroup$
    – whuber
    Aug 28 '19 at 13:47
  • $\begingroup$ @whuber do you think we could learn something from the crossed predictions $X2 \underset{(1)}{\rightarrow}$ and $X1 \underset{(2)}{\rightarrow}$? $\endgroup$ Aug 28 '19 at 14:33
  • $\begingroup$ Yes, you might--but it is hard to see how you could improve either prediction using data from the other one without postulating some model about how they might be related. $\endgroup$
    – whuber
    Aug 28 '19 at 14:52
  • $\begingroup$ @whuber I see, so it's probably best to scale the response variables in a reasonable fashion, say percent rank, as opposed to trying to learn the "true" mapping. $\endgroup$ Aug 28 '19 at 15:40

The introduction to this paper by Bastani gives some good references for solving this type of problem.

Multitask learning combines data from multiple related predictive tasks to train similar predictive models for each task. It does this by using a shared representation across tasks (Caruana 1997). Such representations typically include variable selection (i.e., enforce the same feature support for all tasks in linear or logistic regression, Jalali et al. 2010, Meier et al. 2008), kernel choice (i.e., use the same kernel for all tasks in kernel regression, Caruana 1997), or intermediate neural net representations (i.e., use the same weights for intermediate layers for all tasks in deep learning, Collobert and Weston 2008). Transfer learning specifically focuses on learning a single new task by transferring knowledge from a related task that has already been learned (see Pan et al. 2010 for a survey).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.