# Taking last observation into account in a regression

I am trying to make "immediate" prediction of bikes availabilities (which means 30 minutes to 2 hours prediction from the last observation).

In my regression features, I found it clear that I had to put the "bikes_before" feature, that is the value of the last observation. To handle the lack of periodicity of the observation, I input another feature "timediff" that represents the time difference between the observation and the immediate last observation.

I expected that a good model would less take into account the "bikes_before" value when "timediff" is high, and in this case would take more into account the other features (weather, hour...). But I trained a gradient boosted trees model that gave me a ~1 bike RMSE (not surprising) and it gave a almost exclusive (~95%) feature importance to bike_before. When I artificially changed the timediff to a very high value for prediction, the model still predicts a value really close to the bikes_before value.

My question is : how can I make my model understand that "bikes_before" is a good information when timediff is low but not a good one when it is higher ?

• What exactly are your data? Have you thought of constructing a model that reflects the physics, economics, and geography of borrowing vehicles? – whuber Apr 16 '17 at 22:01
• @whuber: I'm guessing the OP is looking at the bike sharing example on Kaggle – Cliff AB Apr 17 '17 at 3:36
• @CLiff The OP would still need to specify the relevant information in the question. – Glen_b Apr 17 '17 at 5:55
• @whuber My data are weather data + date data (hour, day of the week) + number of bikes on the 3 closest stations. – camille.chanial Apr 17 '17 at 13:39

The problem you have is, that variable timediff doesn't affect the value of conditional mean by much:
$$E(\hat{y}|x_1,...,x_p) \approx E(\hat{y}|x_1,...,x_p,x_{timediff})$$
You said bikes_before is the most predictive variable. This is to be expected. In all likelihood the expected value of bikes_after is equal to the value of bikes_before. What changes with time or timediff is variability around this expected value. And this variability changes with timediff. Intuitively this is easy to understand: if we observe 7 bikes on a stand in time $t$, there will probably still be 7 bikes there in a minute or two. But in two hours the number could easily be completely different like 2 or 12 bikes. The variability changes but the expected value probably doesn't. In two hours the average number of bikes on the stand is probably pretty close to 7.
The problem is regression and tree models are meant to model expected values. When timediff gets larger accuracy of your prediction will get worse. To get around this problem you can regress a chosen variance measure on timediff (and other variables) and use this information in setting the confidence regions for predictions. Variability of prediction is a direct measure of quality of prediction. Don't forget to validate results with training data.