I'm experimenting with catboost, predicting electricity demand from temperature, timeofday, dayofweek and if a day is a public holiday or not (and a few other continuous and categorical columns but they're the key ones).

I'm having issues with the holiday category. The feature importance of this value is very low and it's pretty much ignoring it. Demand on a holiday is usually very low, but there aren't many holidays in a year. Generally with linear regression a holiday category will still have a high importance as it locally reduces the mean squared error considerably on these days (demand is usually pretty low on a holiday). But this isn't happening using catboost. I'm wondering if there's anything I can do to boost the weight put on this feature?

I've tried a True / False category (i.e is or isn't a public holiday) and I've tried specific holiday names (Christmas Day, Easter Monday etc.) but in either case the predicted demand on public holidays has very high error.

To give some more info:

I have the following columns

TimeOfDay (1-96 representing 15 minute intervals) DayOfWeek (Monday - Sunday) MonthOfYear (Jan-Dec) Holiday (Easter, Christmas Day etc) Temperature

I'm training with 1 years data, cross validating (not using the catboost cross validation) with 5 cuts from a 2nd year of data. I don't have much data.

The model picks up on the temperature and time of day / day of week cycle fairly well. But it doesn't differentiate between holidays and non-holidays. Are trees poor at predicting low frequency events - there's only 10 in the training set and when you look at (MonthOfYear,DayOfWeek,Holiday) the data doesn't cover the space at all well (holidays tend to be clustered on Mondays around the first few months of the year.

  • $\begingroup$ Can you give more details? A straw thought: If you are using, in some way, cross-validation in the process, many folds will completely miss the holidays ... $\endgroup$ – kjetil b halvorsen Dec 17 '18 at 9:20
  • $\begingroup$ Your hunch was essentially correct. The site operational policy changed, in the train set the site was closed on public holidays and in the validation set it was closed. $\endgroup$ – David Waterworth Dec 18 '18 at 2:40

I figured out the issue, kjentil b halvorsen pointed me in the correct direction. The behaviour had changed right about where I split the data into train and validate.

So in order to rectify I've had to reclassify my list of holidays, looking at each day and identifying if the site was trading or not on the holiday (I'm modelling commercial building electricity load) and then update the list on a site by site basis. As I mentioned in the comment this makes forecasting slightly difficult but I'll make the assumption that if next years holidays are the same as this years (I think what's happening is the owners have updates the control system, programming holidays in so aircon / lighting etc is now shutting down but it wasn't in the past)

  • $\begingroup$ so you had to stratify on holiday, to assure that they were in each fold? $\endgroup$ – EngrStudent - Reinstate Monica Dec 18 '18 at 2:47
  • $\begingroup$ @EngrStudent I had to re-label the holidays, as for some (I'm forecasting load for commercial buildings) the building was trading (or at least the aircon and lighting was on) and for others it wasn't (i.e. set Christmas 2016 to 'non-holiday' and Christmas 2017 to 'holiday' etc. This will make forecasting harder as I still have to identify going forward what's driving this. $\endgroup$ – David Waterworth Dec 18 '18 at 20:56
  • $\begingroup$ You can do two-step where you predict "has holiday cooling" (nominal logistic) and then augment the data and use as input to predict energy use as a function of log of ID vs. OD temperature (think LMTD in db, wb domains), wind speed (think about Newtonian heat transfer vs. velocity), and probability of precipitation. Where are you getting your outdoor temperature values? $\endgroup$ – EngrStudent - Reinstate Monica Dec 19 '18 at 18:04
  • $\begingroup$ @EngrStudent I'm just using the temperature, humidity, pressure and wind speed/direction from the local weather bureau, so it's not true outside temperature as it's recording at a different location. I also don't have inside temperature so I don't have all the features a physics model would use. Having said that a catboost model works quite well (probably because it models the non-temperature related mean response quite well) $\endgroup$ – David Waterworth Dec 21 '18 at 2:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.