# Subjectivity of models and their performance tests. Sampling issue

I am trying to classify cases according to binary response variable. For this purpose I chose random forest and classification tree methods. So, as it is usually done, I randomly split the data into training and test datasets (prop 0.8). Train the model with train dataset and test it with test dataset.

1) The question is... how can I be sure that test dataset is good for testing and that the model and tests results are objective?

This question arises, because when I rerun the division of data into train and test datasets several times, each time I got a tangible variation in the results: different variable importance, tree structures and sizes for classification tree and different AUC (0.92 - 0.97) for random forest.

So to get the best test result, I can create a function, which would try out different seeds for data division into test and train samples. Then I would choose the seed which would maximise my test score. However, I consider this approach unwelcome in academical society.

So my preliminary thoughts about how I could solve this issue are:

1. Compute similarity between test and train datasets and publish it with test results. The idea is simple. The better test results are and the more dissimilar datasets are, the better model it is, as it can explain truly new data.

2. Publish several most common classification tree structures produced using different subsamples of the data. Each substructure of a tree and each tree should have assigned its rate of appearance (for example, if root variable's "X" rate of appearance is 60%, when second split variable's "Y" rate of appearance is 90%, that means that "X" variable was the root variable in 60% of produced trees, when "Y" variable was used as a second split in 90% of those 60% trees. If tree structure's rate of appearance is 25%, that means that the exact same tree structure appeared 25% of the time).

However, these are just some thoughts. I do not know if am looking at the right direction and if this approach is the right one. So I would appreciate if someone would expand on the topic of 1) question or/and on my suggested or other possible solutions of the problem, preferably with the examples of R code.

how can I be sure that test dataset is good for testing

You can't. Most things in statistics are only approximate, and the use of a train–test split to estimate predictive accuracy is no different.

Since you're concerned by the fact that different train–test splits yield different results, why not use cross-validation? Cross-validation produces an average across several splits, reducing the risk of bias from an unlucky split.

how can I be sure that… the model and tests results are objective?

Objectivity is somewhat overrated (a more objectively conducted analysis need not be more accurate or useful), but it's just a matter of following a procedure that's transparent and justified a priori. For train–test splitting, that means you should decide how you're going to make the split in advance of making it and then stick with that decision instead of changing it based on your result.

So to get the best test result, I can create a function, which would try out different seeds for data division into test and train samples. Then I would choose the seed which would maximise my test score.

This would defeat the purpose of a train–test split, which is to estimate how well a model will perform on cases it hasn't been trained with. By choosing the test set to make the model perform as well as possible, you're effectively, albeit indirectly, training it on the test set.

The better test results are and the more dissimilar datasets are, the better model it is, as it can explain truly new data.

A dataset that is maximally distant from the training set need not be maximally representative of the population. In fact, the more representative your training set is, the less representative this data will be.

Publish several most common classification tree structures produced using different subsamples of the data.

You can do this as a sort of sensitivity analysis, showing how sensitive the model structure is to the training data, but it's a distinct question from predictive accuracy.