1
$\begingroup$

I have a computer generated data and when I plot the raw data, I see the following plot.

Raw data plot. Y is response and X is independent variable

As a quick and dirty method, I applied linear model to this data and when I plot the predicted vs residual, I got expanding flannel plot which shows that the variance is not random. Therefore, I used BoxCoxTran method in caret package to transform the input variable. I got this message : Lambda could not be estimated; no transformation is applied. The studentized Breusch-Pagan test still shows the p-value to be < 2^-16, which confirms the heteroscedasticity problem in the data.

When I plotted predicted value vs residual, I got flannel shaped scatter plot. Residual plot

I tried transforming the idependent variable (taking sqrt, log, second degree polynomial), but nothing helped.

What other methods should I try to create a better classifier?

Edit: After having clue from the responses below, I created a new variable and then tried to create a model. I finally got relatively better model. The residue plot can be seen below.

new_col = Y/(X+0.1)
df$new_col = new_col
model_lm <- lm(Y ~ new_col^2, data = df, na.action=na.exclude)

My new question is: For unknown data, how can I get the response?

Residual plot

$\endgroup$
3
  • $\begingroup$ Weighted least squares? $\endgroup$ – SmallChess Apr 23 '17 at 12:20
  • $\begingroup$ These data can be handled beautifully with linear models--and probably do not need transformation or weighting. Plot $y/x$ and proceed from there: that should make it obvious what is going on and it will reveal important details of the smaller variations in the data that are presently obscured. $\endgroup$ – whuber Apr 23 '17 at 14:13
  • $\begingroup$ This appears to be repeated-measures data. I suggest you look into mixed effects models. In fact, you data looks exactly like what I usually sketch when I explain mixed effects models to our PhD students. $\endgroup$ – Roland Apr 25 '17 at 7:41
1
$\begingroup$

Depending on your goals there are different methods that do not assume homoscedasticity. Two such are quantile regression and regression trees.

$\endgroup$
2
  • 1
    $\begingroup$ The heteroscedasticity in this plot has such an obvious structure that it would be a mistake to work around it. This is clearly a collection of lines through the origin. Another way to put it is that there is an extremely strong correlation structure to these data. Quantile regression will not cope with that. A regression tree would make short work of it indeed, provided one were to introduce an appropriate variable--namely, one that depends monotonically on $y/x$. $\endgroup$ – whuber Apr 23 '17 at 14:15
  • $\begingroup$ @whuber, please see my edit. As suggested by you I am able to find relatively good model. For test data (When response is not known), how this model will work? Thanks for your response. $\endgroup$ – learner Apr 25 '17 at 3:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.