Unstable Prediction Probabilities

We have a data set of a company that needs to predict their employee resignation status.

So We developed four classification models "Bagging","Boosting", "RandomForest" and "Logistic". The task we have is to predict the resignation probability of the non resigned employees of the data set. The non resigned employees are also in the training set. So technically we have to do the prediction on the training set too.

The problem occurred is when a specific observation is in the test set it gives a prediction probability which is completely different from the prediction probability when it is in the training set.

E.g. when the employee no 00 is in the test set the prediction probability is 30% but when he is in the training set the probability is 7%.

We fitted the models with and without feature selections. For both methods the problem exists. What would be the problems in the models and what can we do to overcome this problem?

• Your sample size must be very small. A small data size is not appropriate for machine learning. How big it is? Commented Sep 7, 2018 at 7:33
• Does this happen with sufficiently regularized logistic regression ? (which should be more stable than the others) Commented Sep 7, 2018 at 8:28
• The sample size is 154 observations. We used Logistic with LASSO penalty Commented Sep 7, 2018 at 8:34
• have you checked that all data preprocessing steps are applied prior to the prediction stage? Commented Sep 7, 2018 at 8:54
• Yes, All the pre processing steps are stored as a function in R. It runs prior to the model fitting stage. Commented Sep 7, 2018 at 8:59

Providing a definitive answer wil be difficult without knowing more about your data, understanding the models and chosen hyperparameters, and performing various checks on the results.

One possible cause of such behaviour is, as you hint in the question title, model instability.

Intuitively, a stable algorithm is one for which the prediction does not change much when the training data is modified slightly.

Which is the opposite of what seems to be happening to you (see here for a detailed description of algorithmic stability) Variance of $K$-fold cross-validation estimates as $f(K)$: what is the role of "stability"?

Without going into the details of the above post, the stabilty of a model will depend on:

• The inherent algorithm being used: for example decision trees are notoriously unstable, while $L_2$ ridge regression is inherently more stable (for appropriate regularization parameter)
• The size of the data set
• The number of outliers, their distribution, and their values

This is assuming there are no pre-processing or feature selection steps beforehand as these will add many more possible sources of instability. For example Lasso on correlated features is known to be unstable.

What you could try

Assuming your objective is to reduce the instability of your algorithm, you could try the following

• Add $L_2$ regularization, or a combination of $L_1$ and $L_2$ (elastic net)
• Use different, more stable, algorithms (e.g. bagged regularizers, soft margin SVM with regularization etc..)
• Transform your data (remove outliers, obtain more data, perform PCA, compression etc..)
• Add "robustifying" steps to your algorithm. These will depend on the algorithm and may or may not exist. An example are the robustifying iterations added to the following LOESS algorithm

Note that algorithm stability does not necessarily imply better predictions or performance of your model...

A toy example

The same polynomial linear regression algorithm is unstable in the LHS, and stable in the RHS. More data points reduce the effect of outliers on the model.