1
$\begingroup$

I am using XGBoost for a classification problem (3 classes) where the 6 features are (unscaled) time series. I tried applying forward chained cross validation, but I obtain very low accuracy with my model, 30-40%.

Here is the code:


from sklearn.model_selection import TimeSeriesSplit
from sklearn.metrics import accuracy_score
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import confusion_matrix
from xgboost import XGBClassifier
from sklearn.svm import SVC

tscv = TimeSeriesSplit()

def evaluate(model):
    # train model on training dataset
    for train_index, test_index in tscv.split(X2):
        X_train, X_test = X2[train_index], X2[test_index]
        y_train, y_test = Y[train_index], Y[test_index]
    
        model.fit(X_train, y_train)
        y_predict = model.predict(X_test)
        predict_values = [round(value) for value in y_predict]
        # calculate accuracy
        accuracy = accuracy_score(y_test, predict_values)
        print("Accuracy: %.2f%%" % (accuracy * 100.0))
        # confusion matrix
        print("Confusion Matrix:")
        conf_matrix = confusion_matrix(y_test, y_predict)
        print(conf_matrix)

seed=7
mxgbc = XGBClassifier(max_depth=20, learning_rate=0.1, n_estimators=100, objective='multi:softprob', booster='gbtree', n_jobs=1, nthread=None, gamma=0.1, min_child_weight=4, tree_method='hist',max_delta_step=0, subsample=1, colsample_bytree=1, colsample_bylevel=1, colsample_bynode=1, reg_alpha=0.5, reg_lambda=1, base_score=0.5, random_state=seed,missing=None)
print("## XGBClassifier:")
evaluate(mxgbc)

The accuracy I get is low, 30-40%. I compared with scaling with MinMaxScaler(), but results are the same. Before moving forward to parameter tuning, I need to know if am I doing something wrong in the model specification/ time series validation above. Help much appreciated.

$\endgroup$
1
$\begingroup$

Your code looks good to me - no obvious errors. You may find How to know that your machine learning problem is hopeless? useful.

However, don't use accuracy as a KPI. Why is accuracy not the best measure for assessing classification models?

(See here for a motivation for short answers. Longer answers are always welcome.)

$\endgroup$
2
  • $\begingroup$ Stephan, thanks. I have no real issues with imbalanced dataset. And accuracy is I believe a reasonable evaluation choice for my problem where I'm trying to predict stock market trends. However, I am using both continuous and discrete variables in the feature set. Could this be the problem? $\endgroup$
    – J.Dow
    Nov 5 '20 at 9:14
  • $\begingroup$ Accuracy is problematic even for balanced datasets, see my answer in the linked thread. I don't see how different kinds of features would be a problem for you. But now that you say you are trying to predict stock market trends, it makes complete sense for accuracy to be low, by the efficient markets hypothesis. Simply speaking: if it were simple to predict trends, people would do so and buy the stocks that will go up - which will increase their prices before the trend, and wipe out the effect. $\endgroup$ Nov 5 '20 at 9:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.