1
$\begingroup$

I am trying to make "immediate" prediction of bikes availabilities (which means 30 minutes to 2 hours prediction from the last observation).

In my regression features, I found it clear that I had to put the "bikes_before" feature, that is the value of the last observation. To handle the lack of periodicity of the observation, I input another feature "timediff" that represents the time difference between the observation and the immediate last observation.

I expected that a good model would less take into account the "bikes_before" value when "timediff" is high, and in this case would take more into account the other features (weather, hour...). But I trained a gradient boosted trees model that gave me a ~1 bike RMSE (not surprising) and it gave a almost exclusive (~95%) feature importance to bike_before. When I artificially changed the timediff to a very high value for prediction, the model still predicts a value really close to the bikes_before value.

My question is : how can I make my model understand that "bikes_before" is a good information when timediff is low but not a good one when it is higher ?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ What exactly are your data? Have you thought of constructing a model that reflects the physics, economics, and geography of borrowing vehicles? $\endgroup$
    – whuber ♦
    Commented Apr 16, 2017 at 22:01
  • $\begingroup$ @whuber: I'm guessing the OP is looking at the bike sharing example on Kaggle $\endgroup$
    – Cliff AB
    Commented Apr 17, 2017 at 3:36
  • 2
    $\begingroup$ @CLiff The OP would still need to specify the relevant information in the question. $\endgroup$
    – Glen_b
    Commented Apr 17, 2017 at 5:55
  • 1
    $\begingroup$ @whuber My data are weather data + date data (hour, day of the week) + number of bikes on the 3 closest stations. $\endgroup$
    – camille.chanial
    Commented Apr 17, 2017 at 13:39

1 Answer 1

Reset to default
1
$\begingroup$

The problem you have is, that variable timediff doesn't affect the value of conditional mean by much:

$$E(\hat{y}|x_1,...,x_p) \approx E(\hat{y}|x_1,...,x_p,x_{timediff}) $$

You said bikes_before is the most predictive variable. This is to be expected. In all likelihood the expected value of bikes_after is equal to the value of bikes_before. What changes with time or timediff is variability around this expected value. And this variability changes with timediff. Intuitively this is easy to understand: if we observe 7 bikes on a stand in time $t$, there will probably still be 7 bikes there in a minute or two. But in two hours the number could easily be completely different like 2 or 12 bikes. The variability changes but the expected value probably doesn't. In two hours the average number of bikes on the stand is probably pretty close to 7.

The problem is regression and tree models are meant to model expected values. When timediff gets larger accuracy of your prediction will get worse. To get around this problem you can regress a chosen variance measure on timediff (and other variables) and use this information in setting the confidence regions for predictions. Variability of prediction is a direct measure of quality of prediction. Don't forget to validate results with training data.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for your insight! As a beginner in machine learning, I don't really get how I can put your suggestion in practice. Should I select my data so that timediff varies on a large range (from 0 to 60 i.e. 1 hour for example?) ? $\endgroup$
    – camille.chanial
    Commented Apr 18, 2017 at 8:45
  • 1
    $\begingroup$ I think what @Vivaldi is suggesting is that you imagine a scatter plot: on the x-axis you write down the timediff and on the y-axis you plot all differences of bikes_before to bikes_now. You should see that with increasing timediff, the plot becomes more 'spread out'. Now you make a new table with two columns: col1: timediff col2: variance of (bikes_before - bikes_now) for all observations with the respective timediff. Then you make another (linear or something) regression and use this as an additional information 'how certain' your prediction is... $\endgroup$
    – Fabian Werner
    Commented Apr 18, 2017 at 8:58
  • 1
    $\begingroup$ Another possibility would be to train two different models: one for 'small' timediff's and one different model (possibly different features as well) for 'large' timediffs. You should use a method like the variance above in order to see what 'large' and 'small' could be. However, since you have already observed that including timediff does not make any difference I suspect that the rest of the features do not help for 'large' timediffs. Hence, you will need to include more/other/more descriptive features in the model for large timediffs. $\endgroup$
    – Fabian Werner
    Commented Apr 18, 2017 at 9:00
  • 1
    $\begingroup$ Think about it like that: what do you expect to influence the amount of bikes when you already know that your prediction is going to be 'bad' because the time diff is large? Then add these features... $\endgroup$
    – Fabian Werner
    Commented Apr 18, 2017 at 9:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.