11
$\begingroup$

An issue that happens quite often in my experiments is the model varies in performance when random state for the algorithm is changed. So the question is simple, should I take random state as a hyperparameter? Why is that? If my model outperforms others with different random state, should I consider the model as over fit a particular random state?

a log of decision tree in sklearn: (random_rate should be random state) a log of decision tree in sklearn

$\endgroup$
  • $\begingroup$ With modern computational power, it is possible to identify a seed that provides an edge-case result. Let's say you're a researcher and you have performed an experiment, but your results are not working out the way you want. It would be pretty easy to run your experiment across millions of seeds to see which ones tell the story you are looking for. Best to have a fixed seed that you always use. Keeps you honest! $\endgroup$ – Brandon Bertelsen Feb 25 '19 at 1:52
8
$\begingroup$

No, you should not.

Hyperparameters are variables which control some high-level aspect of an algorithm's behavior. As opposed to regular parameters, hyperparameters can not be automatically learned from training data by the algorithm itself. For this reason, an experienced user will select an appropriate value based on his intuition, domain knowledge and the semantic meaning of the hyperparameter (if any). Alternatively, one might use a validation set to perform hyperparameter selection. Here, we try to find an optimal hyperparameter value for the entire population of data by testing different candidate values on a sample of the population (the validation set).

Regarding the random state, it is used in many randomized algorithms in sklearn to determine the random seed passed to the pseudo-random number generator. Therefore, it does not gobern any aspect of the algorithm's behavior. As a consecuente, random state values which performed well in the validation set do not correspond to those which would perform well in a new, unseen test set. Indeed, depending on the algorithm, you might see completely different results by just changing the ordering of training samples.

I suggest you select a random state value at random and use it for all your experiments. Alternatively you could take the average accuracy of your models over a random set of random states.

In any case, do not try to optimize random states, this will most certainly produce optimistically biased performance measures.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

What does the random_state effect? training and validation set splitting, or what?

If it's the first case, I think you can try to find differences between the splitting scheme under two random states, and this might give you some intuition in your model(I mean, you can explore why it works to train model on some data, and use the trained model to predict some validation data, but doesn't work to train model on some other data, and predict some other validation data. Are they distributed differently?) Such Analysis may give you some intuition.

And by the way, I encountered this problem too:), and just don't understand it. Maybe we can work together on investigating it.

Cheers.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ I don't understand the question and I don't understand this answer. $\endgroup$ – Michael R. Chernick Mar 11 '18 at 15:28
  • $\begingroup$ The question it, what's the usage of random_state in you case? Is it used as a seed to generate random number? $\endgroup$ – Janzen LIU Mar 21 '18 at 8:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.