1
$\begingroup$

I am trying to make "immediate" prediction of bikes availabilities (which means 30 minutes to 2 hours prediction from the last observation).

In my regression features, I found it clear that I had to put the "bikes_before" feature, that is the value of the last observation. To handle the lack of periodicity of the observation, I input another feature "timediff" that represents the time difference between the observation and the immediate last observation.

I expected that a good model would less take into account the "bikes_before" value when "timediff" is high, and in this case would take more into account the other features (weather, hour...). But I trained a gradient boosted trees model that gave me a ~1 bike RMSE (not surprising) and it gave a almost exclusive (~95%) feature importance to bike_before. When I artificially changed the timediff to a very high value for prediction, the model still predicts a value really close to the bikes_before value.

My question is : how can I make my model understand that "bikes_before" is a good information when timediff is low but not a good one when it is higher ?

$\endgroup$
  • 1
    $\begingroup$ What exactly are your data? Have you thought of constructing a model that reflects the physics, economics, and geography of borrowing vehicles? $\endgroup$ – whuber Apr 16 '17 at 22:01
  • $\begingroup$ @whuber: I'm guessing the OP is looking at the bike sharing example on Kaggle $\endgroup$ – Cliff AB Apr 17 '17 at 3:36
  • 2
    $\begingroup$ @CLiff The OP would still need to specify the relevant information in the question. $\endgroup$ – Glen_b Apr 17 '17 at 5:55
  • 1
    $\begingroup$ @whuber My data are weather data + date data (hour, day of the week) + number of bikes on the 3 closest stations. $\endgroup$ – camille.chanial Apr 17 '17 at 13:39
1
$\begingroup$

The problem you have is, that variable timediff doesn't affect the value of conditional mean by much:

$$E(\hat{y}|x_1,...,x_p) \approx E(\hat{y}|x_1,...,x_p,x_{timediff}) $$

You said bikes_before is the most predictive variable. This is to be expected. In all likelihood the expected value of bikes_after is equal to the value of bikes_before. What changes with time or timediff is variability around this expected value. And this variability changes with timediff. Intuitively this is easy to understand: if we observe 7 bikes on a stand in time $t$, there will probably still be 7 bikes there in a minute or two. But in two hours the number could easily be completely different like 2 or 12 bikes. The variability changes but the expected value probably doesn't. In two hours the average number of bikes on the stand is probably pretty close to 7.

The problem is regression and tree models are meant to model expected values. When timediff gets larger accuracy of your prediction will get worse. To get around this problem you can regress a chosen variance measure on timediff (and other variables) and use this information in setting the confidence regions for predictions. Variability of prediction is a direct measure of quality of prediction. Don't forget to validate results with training data.

$\endgroup$
  • $\begingroup$ Thank you for your insight! As a beginner in machine learning, I don't really get how I can put your suggestion in practice. Should I select my data so that timediff varies on a large range (from 0 to 60 i.e. 1 hour for example?) ? $\endgroup$ – camille.chanial Apr 18 '17 at 8:45
  • 1
    $\begingroup$ I think what @Vivaldi is suggesting is that you imagine a scatter plot: on the x-axis you write down the timediff and on the y-axis you plot all differences of bikes_before to bikes_now. You should see that with increasing timediff, the plot becomes more 'spread out'. Now you make a new table with two columns: col1: timediff col2: variance of (bikes_before - bikes_now) for all observations with the respective timediff. Then you make another (linear or something) regression and use this as an additional information 'how certain' your prediction is... $\endgroup$ – Fabian Werner Apr 18 '17 at 8:58
  • 1
    $\begingroup$ Another possibility would be to train two different models: one for 'small' timediff's and one different model (possibly different features as well) for 'large' timediffs. You should use a method like the variance above in order to see what 'large' and 'small' could be. However, since you have already observed that including timediff does not make any difference I suspect that the rest of the features do not help for 'large' timediffs. Hence, you will need to include more/other/more descriptive features in the model for large timediffs. $\endgroup$ – Fabian Werner Apr 18 '17 at 9:00
  • 1
    $\begingroup$ Think about it like that: what do you expect to influence the amount of bikes when you already know that your prediction is going to be 'bad' because the time diff is large? Then add these features... $\endgroup$ – Fabian Werner Apr 18 '17 at 9:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.