1
$\begingroup$

When random state is changed between 0,1,2 manually I observed accuracy is changing and at the same time when the model was checked with random state '0' and with internally split X_test data it predicts 82% and external data it wasn't predicting correct but still accuracy is high of 82%.

With random state '2' the external data is predicting good but accuracy is 71%. Should I consider random state based on accuracy of the model or should I consider random state based on external data prediction which was correct but accuracy is 71%.

I'm using random_state=2

$\endgroup$
2
  • 1
    $\begingroup$ What model are you using? Do you understand how a random state is related to a PRNG? $\endgroup$
    – Sycorax
    Apr 29, 2020 at 6:17
  • $\begingroup$ For text classification I'm using TFIDFVectorizer and Naive bayes model. $\endgroup$
    – Jainmiah
    Apr 29, 2020 at 7:02

1 Answer 1

1
$\begingroup$

It appears that your model (or at least your modeling pipeline) includes some randomization. It may be that the model itself does (as in a Random Forest), or your effects may be due to random sampling for the train-test split.

If your modeling pipeline involves randomization, then the state of the RNG will have an impact on the model, and therefore also on your accuracy.

However, your accuracy should not vary all this much. (It's good practice to re-run your pipeline with different RNG seeds to assess precisely this variation.) Since it does, I strongly suspect that your model is overfitting.

$\endgroup$
3
  • $\begingroup$ For text classification I'm using TfidfVectorizer and naive bayes model naive_bayes = Pipeline([('vect', TfidfVectorizer()), ('clf', MultinomialNB()) ]) # train the model naive_bayes.fit(X_train, y_train) $\endgroup$
    – Jainmiah
    Apr 29, 2020 at 7:03
  • $\begingroup$ If model is over fitting what is the solution to make it correct? $\endgroup$
    – Jainmiah
    Apr 29, 2020 at 7:30
  • $\begingroup$ It's hard to say without much more information about what you are doing. Best to get more data, or reduce the complexity of your model. (I am not an expert on text classification.) I recommend you read up on overfitting in the context of text classification, I assume there is some material on this out there. Good luck! $\endgroup$ Apr 29, 2020 at 7:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.