2
$\begingroup$

I'm running a ML algorithm on some data, and I noticed that if I change the random state inside the train_test_split function, accuracy score change in a quite wide range.

For example, with random state = 4, I reach an accuarcy score range that may vary from 0.78 to 0.8 (it depends by the seed in the algorithm). By using another value, like 42, it goes down to 0.65 - 0.69.

I don't have duplicates in the dataset, and the task is a multi-class text classification.

I really don't understand this beahviour, is there an explanation?

Thanks.

$\endgroup$
2
  • 1
    $\begingroup$ Welcome to Cross Validated! How many observations do you have? $\endgroup$
    – Dave
    Commented Oct 23, 2023 at 14:03
  • $\begingroup$ 490 used as train, 70 used as test $\endgroup$ Commented Oct 23, 2023 at 14:08

1 Answer 1

1
$\begingroup$

You have $560$ total observations. Harrell has noted that splitting the data the way you have often leads to instability in the performance metrics until the sample size reaches $20000$ (this is probably in his Regression Modeling Strategies textbook with references to the primary literature, and he has written this sort of comment on here [1, 2] and on his blog [3], too). Consequently, your results, even if disappointing, are not surprising.

You might be curious to run your code through something like this:

# Loop over 500 seeds
#
for i in range(500):

    # Set a new seed
    # 
    np.random.seed(i)

    # Run the rest of your code to split the data, train the model, 
    # and report the performance for seed i

This will try many seeds and likely show there to be a major dependence of accuracy on the seed. The interpretation would be that your performance in production is subject to major variability and that the one number you get for the one seed is not trustworthy.

$\endgroup$
6
  • $\begingroup$ many thanks for the answer. Actually the task which I was required was to label 10k titles, in order to build the df I manually labeled 560 of them. According to Harrell, since the whole dataset is 10k, I would not have enough data for handling this with ML, right? $\endgroup$ Commented Oct 23, 2023 at 14:15
  • $\begingroup$ @Federicofkt Machine learning vs simpler models is really a separate issue. Harrell's 20k comment is really about the train-test splitting. He typically advocates for bootstrap validation. $\endgroup$
    – Dave
    Commented Oct 23, 2023 at 14:19
  • $\begingroup$ @Dave isnt it just some kind of bootstrapping ? $\endgroup$ Commented Oct 23, 2023 at 14:22
  • $\begingroup$ @RenaudBied-charreton Isn't what some kind of bootstrapping? $\endgroup$
    – Dave
    Commented Oct 23, 2023 at 14:25
  • 1
    $\begingroup$ @RenaudBied-charreton I do not see the way in which that loop samples with replacement, so I wouldn't call that bootstrap. $\endgroup$
    – Dave
    Commented Oct 23, 2023 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.