linear regression inappropriate when there are multiple underlying groups

Question

In this scenario, one group has a clear linear relationship between x and y. Another group doesn't. However, there is no way to differentiate them in the data. In this case, performing a simple linear regression on the data results in a clear trend in the residual plot. A simple linear regression isn't appropriate in this case and transformations of the variables is meaningless right?

Here's my R code to simulate this scenario, in order to show what I'm talking about:

    num_obs = 100
    beta_0  = 0
    beta_1  = 1
    sigma   = 2
    set.seed(1)
    epsilon = rnorm(n = num_obs, mean = 0, sd = sigma)
    x_vals = seq(from = 0, to = 10, length.out = num_obs)#group 1
                 # (linear relationship between x and y)
    y_vals = beta_0 + beta_1 * x_vals + epsilon # group 1
    
    x_vals = c(x_vals,rep(c(0:10),10))   #add group 2  (no 
             # relationship between x and y)
    y_vals = c(y_vals, rep(0:9, each=11))  #add group 2
    
    sim_fit = lm(y_vals ~ x_vals)
    coef(sim_fit)
    plot(y_vals ~ x_vals)
    abline(sim_fit)
    
    plot(sim_fit$fitted.values, sim_fit$residuals) #residual plot
                         # shows clear trend

Here's the data and the best fit line:

Answer 1

You seem to have two groups, with different regressions within each group, but then you have lost the group id's. If that is the situation, one approach is to do clustering and regression simultaneously. One way to do that is via an EM algorithm (Expectation Maximization). There are at least two R packages implementing this, mixreg and flexmix, and I will make a small illustration using the first one, and the data you simulated.

So, starting after running the code in your post,

  library(mixreg)
  df <- data.frame(x=x_vals, y=y_vals)
  mixmod <- mixreg::mixreg(df$x, df$y, ncomp=2)

A plot produced by the package is

which clearly shows the two distinct regression lines. References for the methods can be found in the package documentation.