I have a computer generated data and when I plot the raw data, I see the following plot.
As a quick and dirty method, I applied linear model to this data and when I plot the predicted vs residual, I got expanding flannel plot which shows that the variance is not random. Therefore, I used BoxCoxTran
method in caret
package to transform the input variable. I got this message : Lambda could not be estimated; no transformation is applied
. The studentized Breusch-Pagan test still shows the p-value to be < 2^-16, which confirms the heteroscedasticity problem in the data.
When I plotted predicted value vs residual, I got flannel shaped scatter plot.
I tried transforming the idependent variable (taking sqrt, log, second degree polynomial), but nothing helped.
What other methods should I try to create a better classifier?
Edit: After having clue from the responses below, I created a new variable and then tried to create a model. I finally got relatively better model. The residue plot can be seen below.
new_col = Y/(X+0.1)
df$new_col = new_col
model_lm <- lm(Y ~ new_col^2, data = df, na.action=na.exclude)
My new question is: For unknown data, how can I get the response?