3
$\begingroup$

I am getting the below plot for my data,

enter image description here

and relevant summary is as below:-

Coefficients:
                        Estimate Std. Error t value Pr(>|t|)    
(Intercept)             -682.709     33.469   -20.4   <2e-16 ***
df.keep$X                947.343      5.123   184.9   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 2431 on 18863 degrees of freedom
Multiple R-squared:  0.6445,    Adjusted R-squared:  0.6445 
F-statistic: 3.419e+04 on 1 and 18863 DF,  p-value: < 2.2e-16
all.mod <- lm(df.keep$Y~df.keep$X)
plot(df.keep$X,df.keep$Y,col = "blue", xlab = "x", ylab = "Y",
     main = "Y against X")
abline(lm(df.keep$Y~df.keep$x))

If I use log, sqrt, 1/y^2, 1/y on the Y variable the R squared value reduces significantly and the abline plot still shows the long range of Y values. Would it be possible to use any kind of transformation that gives a better plot?

The 0 value in X axis is crucial in my case because when X is 0 it is very important for me to infer on Y

enter image description here

$\endgroup$
2
  • $\begingroup$ Your last sentence is unclear. Do you mean "When x=0, Y must be 0" or something else? $\endgroup$
    – Glen_b
    Sep 12 '16 at 5:50
  • 1
    $\begingroup$ This looks like fake data. Is this homework? $\endgroup$
    – Glen_b
    Sep 12 '16 at 6:05
3
$\begingroup$

It looks like you're trying to fit a linear model on simulated time series data. The variance keeps on increasing continuously and this looks odd. For heteroskedastic data the variance would change but not increase continuously. This appears to be simulating a random walk.

A linear model will only be able to give you estimates of the mean. Irrespective of any transformation you apply when trying to scale down the variance when fitting the model, the variance of the residuals would keep increasing.


Edited after additional information on the data set:

So I'm inferring that each record represents a vehicle's total distance covered from start date. You're trying to estimate the total distance covered in the future months.

I would suggest the following changes:

  1. Represent each month's data point not as total distance covered, but as incremental distance covered in that month.
  2. If you can, add other features such as geographical area (e.g. vehicles delivering supplies in different regions may have to drive further or nearer than other geographical areas), or vehicle type, etc. these may help you obtain a better estimate of distance driven next month.
  3. You could also cluster the vehicles as lightly-driven, medium, heavily driven and use that as a categorical variable.

Try other features you may find informative and which make sense to the problem you're trying to solve.

$\endgroup$
3
  • $\begingroup$ @Glen_b Each point in the X axis is age in months (i.e. 0 months means less than 30 days, 1 month means 30 days from birth etc) The Y point is kilometers covered by the vehicle. So ideally as the vehicle passes through 0,1,2,... n months the distance covered keep increasing until the vehicle gets calibrated. That is why at 0 (i.e. intercept) it is also important for me to infer the kilometers covered by the vehicles. This is not homework data. I have tried to remove outliers by taking the 90th quantile for every age (i.e. 0,1,2,3,..,11) $\endgroup$ Sep 12 '16 at 7:00
  • $\begingroup$ After 90th quantile I noticed a sudden jump in kilometers for some vehicle which I do not need to encounter. When I apply lm and abline I got the above charts. Hence was seeking for some expert suggestions. I did try various transformations on Y (kilometers) but it did not help @mark999 $\endgroup$ Sep 12 '16 at 7:00
  • $\begingroup$ @ARIMITRAMAITI By putting your reply to me under another person's answer (rather than under your question or under my answer), I was not notified of it (since I had not participated in discussion under that answer). It's pure luck I stumbled upon it at all. I will add some comments to my answer. $\endgroup$
    – Glen_b
    Sep 12 '16 at 9:56
4
$\begingroup$

Response to comments under the other answer (which changes things substantively):

Each point in the X axis is age in months (i.e. 0 months means less than 30 days, 1 month means 30 days from birth etc) The Y point is kilometers covered by the vehicle. So ideally as the vehicle passes through 0,1,2,... n months the distance covered keep increasing until the vehicle gets calibrated.

If you have many observations on each vehicle then you should not be fitting an ordinary regression model -- the data for a single vehicle are dependent for two different reasons. Firstly, you're observing the same vehicle, which leads to dependence (repeated measures), but secondly you're looking at cumulative totals. This makes the values highly dependent.

  1. You should be looking at how much was added each time. This removes the effect of the cumulation (and may also remove much of the fanning effect).

  2. You still need to account for the dependence between observations on the same vehicle in some way, either via a model than accounts for them as repeated measures or via time series models.

  3. The values you're observing will necessarily be positive (or at least non-negative); this will tend to lead to skewness, and may make models designed for a normal response unsuitable. You might need something like a generlaized linear (mixed) model, or you might need to consider a transformed model, for example.

That is why at 0 (i.e. intercept) it is also important for me to infer the kilometers covered by the vehicles.

If I understand (and I am still not sure what it is you're trying to find out here), I think the information you're after should be in each month's incremental data

I have tried to remove outliers by taking the 90th quantile for every age (i.e. 0,1,2,3,..,11)

If you're after averages this may not be suitable (and even if outliers really are something you need to remove rather than model, I doubt this is the best approach) ... but I'm confused are you going across vehicles when you do this? I am not at all sure that makes sense.

I have a feeling we're dealing with an XY problem here -- that instead of explaining the underlying problem you're trying to solve, you're focused on asking about your attempt at a solution. You may be better writing a new question clearly describing the underlying problem more explicitly.

[Original answer removed since in light of the new information, it's no longer relevant]

$\endgroup$
1
  • $\begingroup$ @mark I agree that's a possible interpretation (and partly why I asked for clarification) in which case most of what I have written would be of no help; but if that is what the OP means there are several very problematic issues with the post as it stands. We can only wait on the OP's clarification to be certain, but I felt confident enough that this was likely the intent that I went ahead. $\endgroup$
    – Glen_b
    Sep 12 '16 at 6:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.