4
$\begingroup$

everyone! I find that the performance of random forest classifier in python seems to be related to the order of training samples. Can anyone help me to figure out the reason? Thanks very much!

from sklearn.ensemble import RandomForestClassifier
import numpy as np
from sklearn.datasets import load_digits
def shuffle_train():
    digits = load_digits()
    # print digits.data.shape
    cls = RandomForestClassifier(random_state=0,class_weight='balanced')
    ind = np.arange(1000) # take the first 1000 samples as training dataset
    np.random.shuffle(ind)
    training_data = digits.data[ind, :]
    training_label = digits.target[ind] # shuffle the training dataset
    cls.fit(training_data, training_label)

    testing_data = digits.data[1000:, :]
    testing_label = digits.target[1000:]
    testing_pred = cls.predict(testing_data)
    print np.sum(testing_label==testing_pred)


if __name__ == '__main__':
    for i in range(20):
        shuffle_train() # try 20 times and get different results

And this is the output:

714
715
708
701
702
719
708
712
710
714
723
725
726
722
734
714
701
717
717
689
$\endgroup$
3
$\begingroup$

Given how you've written your code, this is expected behavior.

You've set the seed of the random forest explicitly. This means that the same randomization is used. Part of the randomization procedure of a random forest is to construct boot-strapped samples of the data. The way that sklearn accomplishes bootstrapping is to assign weights to indices of samples in $X$. If you change the order of the data in $X$, but fix the seed of the random forest, the same indices will be given bootstrap weights. However, in your code snippet, those indices correspond to different samples (because the data are shuffled), hence the data provided to each tree will be different between runs.

Providing different data to a decision tree results in a very different tree, because trees are high-variance estimators.

If you need to consistent results between different runs, you'll need to impose a fixed ordering on the data in addition to fixing the seed of the random forest.

If you don't need exact correspondence between different runs, but just want results to be approximately the same between runs, you can just increase the number of trees. Increasing the number of trees in a random forest will tend to reduce the variance of the random forest procedure.

$\endgroup$
8
  • $\begingroup$ Do you have any idea why the default value of the max_features parameter introduces randomness despite fixed random state when class_weight parameter is set to 'balanced'? $\endgroup$
    – kanimbla
    Apr 11 '19 at 17:07
  • $\begingroup$ In what context? In the case of OP's question, when the data are shuffled, the same reasoning applies: different splits are chosen because the data available to each tree is different. $\endgroup$
    – Sycorax
    Apr 11 '19 at 17:12
  • $\begingroup$ But this randomness also happens when you set parameter bootstrap to False. This seems strange to me, see my answer. $\endgroup$
    – kanimbla
    Apr 11 '19 at 17:17
  • $\begingroup$ I think you would be interested in this post on the sources of randomization in RandomForestClassifier stats.stackexchange.com/questions/399819/… I agree with your intuition that using all features at each split and not using bootstrapping should eliminate all sources of randomness. However, there appears to be some non-determinism when the seed is not fixed; I believe this is related to what happens in the presence of ties. $\endgroup$
    – Sycorax
    Apr 11 '19 at 17:23
  • 1
    $\begingroup$ @SuperCodeBrah Random forest bootstrapping occurs at random, not among contiguous blocs. I'm not familiar with any time series-aware versions of random forest, but it's plausible that such a thing might exist. Perhaps asking a question about it would turn up some good suggestions. $\endgroup$
    – Sycorax
    May 29 '20 at 22:29
-1
$\begingroup$

To obtain deterministic behavior during fitting you have to set bootstrap=False and max_features=None in addition to the parameters that you have specified when calling RandomForestClassifier().

While I am not familiar with the source code, I guess that the order of the training data leads to different random samples with replacement when bootstrap samples are used to build trees.

What is less clear, why you need to set max_features=None because the subset of features considered in each split should not be a source of randomness when the random state is fixed as in your example. In fact, when you set class_weight=None you can check that the behavior is as expected, i.e. you don't need to set max_features=None to induce deterministic behavior.

$\endgroup$
1
  • $\begingroup$ The reason for the non-deterministic behavior isn't related to the random forest, it's a direct consequence of shuffling the data. $\endgroup$
    – Sycorax
    May 30 '20 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.