0
$\begingroup$

I am using xgboost to train a classification model. Say, there are 1 million samples totally and my model's f1 score is 0.36. But when I add more 0.5 million samples which are close to some entries of my current dataset, the f1 score is below 0.34. Actually, the additional 0.5 million samples are close to (not equal to) some samples. For example, one additional sample is [1,0.2,4,0.7,0,0,0,0,0,9] and it is close to the sample [1,0.2,4,0.4,0,0,1,0,0,8] in original dataset. So why the larger dataset of 1.5 million make the model weaker ?

Besides, I want to add some explanation of my question: original 1 million sample is from 1 million people, everybody's action feature output 1 entry to my dataset. But half of them has two actions in history and hence it can produce two entries. So from the half people, I can get more 0.5 million entries. My original intention is just make the dataset larger. the model is based only on the action features to predict the label, for example, he/she will make purchase or not.

Improve this question
$\endgroup$
4
  • $\begingroup$ Are you using a particular test set? Training-set performance metrics (what I think you report) are not helpful when trying to estimate "how good the model" is. In addition, just adding more samples doesn't necessitate that performance metrics will increase. For all we know, we initially over-fitted our dataset and know with the additional points have a more realistic assessment of the model's performance. $\endgroup$
    – usεr11852
    Commented Jul 9, 2017 at 20:46
  • $\begingroup$ I test my model on another dataset , which the test set is not used in the training course. Model has no knowledge about the test set. $\endgroup$
    – yanachen
    Commented Jul 10, 2017 at 2:50
  • $\begingroup$ Actually I split the 1.5 million data into 80% training set and 20% testset. Metric was evaluated on the test set. $\endgroup$
    – yanachen
    Commented Jul 10, 2017 at 2:52
  • $\begingroup$ @yanachen that's a fairly small change. Have you tried fitting the model multiple times on different train-test splits and then seeing what the distribution of F1 scores is. I'll bet your 0.02 difference is small compared with the standard error or the variance you'll get just from using different train-test samples. $\endgroup$
    – Dan
    Commented Jul 3, 2018 at 15:23

1 Answer 1

Reset to default
1
$\begingroup$

It sounds like there's a relationship to your data. XGBoost is not generically able to solve time-series problems (but perhaps someone has figured out a clever way to do so -- I don't know). Perhaps a better solution would be to use an actual time-series model.

Either way, the samples with 2 actions reflect dependence among your observations, whether or not that dependence is time-series in nature. XGBoost is not configured to natively fit dependent data structures.

Another possibility is that you need to tune your hyper-paraemters. The additional data might benefit from a different configuration, and this might influence your model fit.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.