Discerning between two different linear regression models in one sample

Question

Suppose I observe a sample $(y_i,x_i)$, $i=1,...,n$. Suppose that I know the following:

$y_i=\alpha_0+\alpha_1x_i+\varepsilon_i$, $i \in J\subset\{1,...,n\}$

$y_i=\beta_0+\beta_1x_i+\varepsilon_i$, $i \in J^c$

where $\varepsilon_i$ are i. i. d. and $J$ is not known in advance. Is it possible to estimate $\alpha_0,\alpha_1,\beta_0,\beta_1$? Or at least test the hypothesis that $J=\varnothing$?

If $J$ is known the problem is very easy to solve. Going through all the subsets is not feasible, since we have $2^n$ possible combinations. If we assume $J=\{1,...,k\}$ with unknown $k=1,...,n$, it is the classical change-point problem, for which many tests are available. I suspect that this maybe ill-posed problem, so I wanted to check before trying to solve it.

Here is a simple illustration of the problem:

N <- 200
s1 <- sample(1:N,N %/% 2)
s2 <- (1:N)[!(1:N) %in% s1]

x <- rnorm(N)
eps <- rnorm(N)

ind <- 1:N

y <- rep(NA,N*T)
y[ind %in% s1] <- 2+0.5*x[ind %in% s1]+eps[ind %in% s1]/5
y[ind %in% s2] <- 1+1*x[ind %in% s2]+eps[ind %in% s2]/5
y

sal1 <- ind %in% s1

plot(x, y)
points(x[sal1], y[sal1], col=2)
abline(2, 0.5, col=2)
abline(1, 1)

Graphically it is more or less obvious that we have two different models. Maybe it is possible to use some classification or data-mining techniques for solving this problem?

Answer 1

You need to model the observations as a mixture model. Define:

$p$ as the probability that a sample belongs to the first data generating process.

Thus, the density function of $y_i$ is given by:

$f(y_i|-) \sim p f_1(y_i|-) + (1-p) f_2(y_i|-)$

where

$f_1(.)$ is the density that arises because of the first data generating process and

$f_2(.)$ is the density that arises because of the second data generating process.

You can then either use maximum likelihood (see for example the EM algorithm) or bayesian approaches to estimate the model.

Answer 2

The first hit on Rseek with keywords "mixture regression" brings up the flexmix package, which does what you want. I seem to recall that there were other packages for this as well.