Random undersampling: is there a way to chose the best majority samples? I'm modeling credit fraud, where I have a small number of samples that result in fraud (1), and most samples that are not fraud (0). I am creating a models for detecting fraud based on new data.
I'm using the following models: logistic regression, K-nn, Support Vector Classifier and decision tree.
The dataset is very similar to this: kaggle.com/mlg-ulb/creditcardfraud
I performed random undersampling on the data to get a 1:1 ratio. This made my models perform a lot better, but since the undersampling is performed randomly every time, I get a slightly different result because of the chosen samples.
Is there a way to find out which of the 8200 majority class samples are best to use in the undersampled data? 
I was hoping to figure out which these are and only use them with the 800 positive samples on my undersampled dataset. 
 A: The methods you list in the question are discriminative (i.e. you do not indicate that you used the one-class variety e.g. of SVM). 
Have you considered one-class classification?
One class classification is probably better suited for credit fraud detection than discriminative classification and as a side effect it does (by construction) not have any difficulties with class imbalance.
One-class classification is good for situatione where


*

*a* well-defined ("positive") class is to be separated from cases that do not belong to this class.  


*

*well-defined means that the data points of this class form a (or maybe few separate) clouds, but we need not expect "surprises" in future cases (this is really not different from how we think of classes in discriminative settings) 

*and there are sufficient cases available from this class

** there can be several  such classes, they are treated separately.


*cases that do not belong to the positive class (aka the negative class) are ill-defined: 


*

*We cannot (or need not be able to) describe those cases better than saying they do not belong to the positive class. 

*There may be many reasons (classes) of cases that do not belong to the positive class

*any new case that does not belong to the positive class may not belong there for an entirely new reason. 

*Side note: specific known classes within the negatives can be modeled as their own class - this does not affect the performance of the recognition of the positive class in any way.


*In one-class classification, a case can belong to more than one class (consider medical diagnosis: a patient may have several diseases that are independent of each other). 
Your task sounds to me as if the no-fraud cases are a prime example for a positive class. In addition, if you have examples of specific known types of fraud, you can also model them as their own class. If only few cases are available, recognition of that that (sub)class of fraud will of course be uncertain. But this will not disturb the performance of recognizing no-fraud cases.
A: You say (but in comments, it should be in the Q proper!) that you evaluated the models by  precision, recall and f1 score. Those measure all depends on you making a hard decision, which in the case of say logistic regression needs a probability threshold. You didn't say how you did choose that threshold, so assuming you did use the "default" of one half. There is in general no convincing reason to always use that default. 
Now, using undersampling of the majority class, you effectively changed the threshold (in terms of the complete data.) If you just used a different threshold with the complete data, you probably would have seen similar results. It would be better for you to see the problem as one of risk estimation, and then evaluate the models using some proper scoring rule. Then you do not need to make a hard decision.  Please read carefully Why isn't Logistic Regression called Logistic Classification? (including its links and references.)
A: I have found some undersampling algorithms which can be used to find out which of the majority class samples are closest to the minority ones.
Some of these algorithms are Near Miss Undersampling, Condensed Nearest Neighbor Rule for Undersampling,... 
Using near miss data in undersampling causes massive overfitting in my model so this approach isn't viable though.
Source with reproducible examples: https://machinelearningmastery.com/undersampling-algorithms-for-imbalanced-classification/
