1
$\begingroup$

I'm facing with a challenge of unsupervised classification of unlabeled data.

The case is, I have circa 1.2 million vehicle warranty claims, and must develop a classification model to tell whether one claim is fraud or not, or to predict a claim's fraud probability. Now I have got a wide analytical base table including variables about the claim's defects and corresponding material and labor fees, about the car's characteristics such as retail price, color, car age, distance and warranty history, about the dealer's warranty and service history and fees, about 300 variables in total.

However, among the 1.2 million observations, only 4000 are labeled as fraud or normal, and the labeled dataset is not a simple random sample, but a collection of somehow believed-to-be suspicious claims, mainly selected based on dealer information. So I cannot simply use supervised learning.

I'm not that into clustering because I cannot explain or evaluate the outcome. I've tried PCA: one principle component can cover 95% of the information, and the component is mainly highly correlated with dealer history, so it can be the difference among dealers instead of claims that counts.

I've also tried some kind of label-propagation, and labeled more data based on my small labeled dataset. Then I developed a decision tree on these labeled data, and found that most important features are also the ones about dealers. So I think the result is not convincible, since my initially-labeled dataset was selected based on dealer history.

So my questions are: 1) Is label-propagation or PU reasonable in this case? 2) Any experience on which unsupervised learning method should I use?

|cite|improve this question
$\endgroup$
  • 2
    $\begingroup$ If your data is partially labeled you can use semi-supervised learning. $\endgroup$ – Tim♦ Nov 24 '17 at 9:40
  • $\begingroup$ @Tim Thanks for your comment. label propagation is also a kind of semi-supervised learning, I believe? $\endgroup$ – Deanna Nov 24 '17 at 10:48
  • $\begingroup$ It's not unsupervised learning: Fraud vs. not fraud is a supervised objective. You only have little labeled data available, so you need a semi-supervised approach, and should maybe consider an active learning approach. An unsupervised technique will find anything, most likely unrelated to fraud. For example, it may cluster the customers into male and female. That will not help. $\endgroup$ – Anony-Mousse Nov 25 '17 at 23:24
  • $\begingroup$ @Anony-Mousse thanks for your comment. I agree that unsupervised learning can lead to any result. will look into active learning thanks. $\endgroup$ – Deanna Nov 27 '17 at 2:26
  • $\begingroup$ here is a list of answered questions about positive unlabeled learning. $\endgroup$ – kjetil b halvorsen Jan 5 at 11:22
0
$\begingroup$

I suggest you to take a look at PU (positive unlabeled) learning. This may sound like two class classification but there are significant differences in what objective is optimized.

In your case a positive example is a case labeled as “possibly fraudulent” and everything else as unlabeled.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thank you for answer. I've also considered PU learning and I think PU is, in reality, also a method of trying to label more data? Do you think PU learning with 1000 positive examples v.s. 1.2 million unlabeled examples is practical? $\endgroup$ – Deanna Nov 24 '17 at 10:47
  • $\begingroup$ @Deanna In my experience, one must be careful when selecting the performance metric to optimize. Similar to working with one-class SVMs. Agree, these methods are pretty fragile but are powerful when mastered. $\endgroup$ – Vladislavs Dovgalecs Nov 25 '17 at 1:43
  • $\begingroup$ I see. Could I ask another question that, is it possible to use a completely unsupervised learning model to deal with fraud detection? $\endgroup$ – Deanna Nov 27 '17 at 3:00
  • 1
    $\begingroup$ @Deanna Unsupervised learning despite lots of research, remains a hot research topic. I cannot be authoritative here. I think the way to go is to use those unsupervised models where you actually understand the assumptions. Bayesian approach using so called probabilistic programming languages can be a way to go. You explicitly encode the assumptions on how the data is generated. You start with a simpler model and improve as you get better understanding. Again, this is only one path. $\endgroup$ – Vladislavs Dovgalecs Nov 27 '17 at 5:28
  • 1
    $\begingroup$ 4000 labeled cases is probably too small a sample size to get reliable risk predictions or reliable classifications. And since the 4000 involved unknown selection bias, I would find another project to work on. $\endgroup$ – Frank Harrell May 25 at 1:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.