0
$\begingroup$

I am very new to Machine Learning, and this is my first project on which I am working. I am trying a binary classification problem, trying to predict claim(yes/no) from an insurance dataset. In my dataset, I have about 7% positive class (1534 Positive datapoints, 21076 Negative datapoints). I have 7 features (2 numerical, 5 categorical).

My model is giving the following performance metrics:

             precision    recall  f1-score   support

          0       0.95      0.70      0.81      6311
          1       0.12      0.52      0.19       472
avg / total       0.89      0.69      0.76      6783

Being a beginner, I am having difficulty understanding how good/bad my model is based on these metrics. How much can I improve my model, given the small and highly skewed dataset which I have? I have taken the following approach: Randomly sample 50-50 splits of the data 10 times, train a Logistic Regression on each, and then average the predictions of all the 10 models.

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
1
$\begingroup$

1. How good / bad is the model?

The results aren’t very good.

Precision of 0.12 means that for each 100 cases your model classified as ‘positive’, it was right in only 12 (correct classifications out of classifications by the model).

Recall of 0.52 means that for every 100 cases in which the real result was ‘1’, the model classified 52 correctly (correct classifications of of real cases).

f1-score is a harmonic mean between the two.

Support states how many observations belonged to the class (for example, in your case there were 472 observations with a real value of ‘1’).

An answer to the question "which metric is more important" depends on the context and the price of each mistake.

There's an elaborate explanation on precision and recall here.

How can the model be improved?

As stated here, an imbalanced dataset is a common issue in Machine Learning. I’d start with undersampling the dominant class since it’s the simplest approach.

In your case, you might decide on taking around 1000 positive cases and 1000 negative cases as your train set, and keep a similar proportion in your test set.

I’d try random forest or SVM instead of logistic regression. There are a lot of considerations to take into account when choosing a model, but since you just started exploring the field, these are two very popular choices you can consider.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for your inputs. My process involved making balanced classes (1000 each) by random sampling and running the classifier, repeating the process 10 times and then make the decision based on average probabilities of each class. I tried SVM (with various kernels), with only a marginal improvement in performance. I feel I am limited by the relatively small dataset (1500 positive class rows, and only 7 features). Whatever method I am trying, based on my readings about the common techniques while dealing with unbalanced classes, I am unable to improve my precision beyond 0.15 $\endgroup$
    – RRC
    Oct 26, 2016 at 4:05
  • $\begingroup$ Your sample isn't very small, 1500 observations in the positive class is very possible to work with. $\endgroup$
    – Pretty Speeches
    Oct 26, 2016 at 6:24
  • $\begingroup$ You can try and visualize the data (I guess you're using python, and I recommend the seaborn visualization package, in order to get an intuition regarding the ability to separate the data. You can also try binning your numerical data - random forest often handle categorical data better. $\endgroup$
    – Pretty Speeches
    Oct 26, 2016 at 6:28
  • $\begingroup$ Yes, I did that. The only significant correlation I can see is with one predictor variable (Driver age for predicting number of claims). I need to study up about feature engineering in detail $\endgroup$
    – RRC
    Oct 28, 2016 at 17:39

Not the answer you're looking for? Browse other questions tagged or ask your own question.