0
$\begingroup$

I have a regression data set and I'm trying to do some feature engineering.

The data set is foot fall coming into a store measured on the hour.

I'd like to include the time of measurement as a feature and I think that mapping the time of day onto a cosine curve should do this.

The target (in_coming) peaks at around 2PM. However my cosine curve peaks at 12PM. I think that this isn't a problem for neural networks as the bias values should easily be able to introduce a phase shift to the time feature aligning it with the peak of the in_coming. I think that if I manually tried to move this peak during feature engineering it would be passing information about the target values behaviour into the training data. So i will leave the cosine peak where it is.

However, I'm not sure if a decision tree can compensate for this offset in a similar way? It's likely that I will end up investigating the use of XGBoosted regression decision trees so would like to know if the time-cosine feature makes sense with trees or whether I should use some other method for time.

$\endgroup$
1
$\begingroup$

I've been investigating this myself, yes a decision tree is quite capable of learning a shifted cosine term, i.e. using catboost and python

t = np.arange(0,24,1) w = 2*np.pi / 24 p = -8*w
X = pd.DataFrame({"S1" : np.sin(w*t),"C1" : np.cos(w*t),}) 
y = np.sin(w*t + p)

from catboost import CatBoostRegressor 
estimator = CatBoostRegressor(logging_level='Silent') 
estimator.fit(X,y) 
plt.plot(estimator.predict(X))

results in a reasonable approximation to the original y.

You can also just regress y ~ t, the fit isn't nearly as good though

X = pd.DataFrame({"t" : t})
y = np.sin(w*t + p)

from catboost import CatBoostRegressor
estimator = CatBoostRegressor(logging_level='Silent')
estimator.fit(X,y)
plt.plot(estimator.predict(X))

It's not entirely clear to me why the former results in a smooth approximation and the latter isn't. In general trees generate a piecewise constant approximation. Boosting smooths the apporixmation but it looks like catboost only grows each tree to depth 3 in the 2nd case and up to 5 in the first case. Also in case 2 after about steps the fit is as good as it gets, in case 1 it keeps improving up to ~1000 steps (I set learning rate to 1, and played with n_estimators and max_depth)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.