Different train/test for model selection Imagine that I have a function that is somewhat like the following:
def evaluate_model(model, X , y):
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.33, random_state=random())
    model.fit(X_train, y_train)
    print(accuracy_score(y_test, model.predict(X_test)))

And I have several models that I want to check the predictive capability, so I use that function for those models and check the accuracy for them. Am I introducing some kind of bias by using different train/test sets for each model? Suppose that I don't want to use CV for hyperparameter tuning.
 A: If you are introducing a bias then you have a bigger problem. It means that your decision depends on the exact contents of the train and test sets. It shouldn’t. Your decision should be robust to partitioning off the dataset
A: Instead of talking about terms like bias or variance, I think it's easier to look at examples when your code will create problems.
Example 1: Unbalanced classes
Let's say we have only four positive positive examples out of 100 data points with a binary feature x. For that four positive examples, the feature x = 1 for two of them and x = 0 for the other two.
The performance of your model will largely depend on how much of the positive examples are in the training dataset, and if the training dataset has good enough coverage of the feature space for positive examples.
If the training dataset has no positive example, the model will be bad. If it has only x=1 or x=0 for the positive examples, the rest will be in the testing dataset and so the model will also perform badly in the test.
If we only perform 3-fold CV, a model can perform much better than others just because it happens to get a good split out of the three trials, while others didn't. Now we're forced to set a higher k, like 10, just to make sure every model gets some good splits, and hope that the averaging effect kicking in. Still, difference is highly likely due to chance of one model getting more epochs of good splits than others.
If all models are evaluated on the same split, a good split will inflate performance for all models, rather than just one. After all epochs, all models will have the same number of good and bad splits. So at least you're comparing apples to apples.
Example 2: Error analysis
Let's say that I found all models to perform at statistically similar level. One common analysis is to investigate if one model makes certain type of errors more often than others.
If all models are evaluated on the same split each epoch, it's easier to compare on the same testing dataset to see which testing data points each model got wrong while others models got right, and analyze their covariates.
If all models are evaluated on different splits each epoch, the difference in responses from different models on the same testing data point can be caused by the difference in the training dataset between the models, rather than the models alone.
Conclusion
In short, your code increases variance of the ranking of the model performance and make comparison of results much harder to interpret. Classic examples from scikit-learn also evaluate models on the same split each epoch.
