4
$\begingroup$

I have a input data and an output binary variable . The y value is 1 if the patient get ill.

> summary(x_or)
   Symscore1        Symscore2        exercise3     exerciseduration3                       groupchange        age3      
 Min.   :0.0000   Min.   :0.0000   Min.   :1.000   Min.   :0.000     Regular to Regular          : 340   Min.   :45.00  
 1st Qu.:0.0000   1st Qu.:0.0000   1st Qu.:3.000   1st Qu.:2.000     Regular to Menopausal       : 360   1st Qu.:49.00  
 Median :0.0000   Median :0.0000   Median :4.000   Median :3.000     Transitional to Transitional: 171   Median :54.00  
 Mean   :0.5504   Mean   :0.5941   Mean   :3.651   Mean   :2.545     Transitional to Menopausal  :1492   Mean   :54.07  
 3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:5.000   3rd Qu.:3.000     Menopausal to Menopausal    : 246   3rd Qu.:59.00  
 Max.   :5.0000   Max.   :5.0000   Max.   :5.000   Max.   :4.000                                         Max.   :66.00  
   packyears           bmi3            education3  
 Min.   : 0.000   Min.   :16.77   Basic     : 348  
 1st Qu.: 0.000   1st Qu.:22.32   Highschool:1013  
 Median : 0.000   Median :24.84   University:1248  
 Mean   : 4.397   Mean   :25.60                    
 3rd Qu.: 5.714   3rd Qu.:27.72                    
 Max.   :97.143   Max.   :57.09                    
> summary(y_or)
   0    1 
2129  480 

I have fitted a random forest model and computed the outlier measure

    rf = randomForest(x = x_or, y = y_or, proximity = T, ntree = 1000)
out = outlier(x = rf$proximity, cls = y_or)

plot(out, col=y_or)

From the outlier plot I can see that some of the samples with y=1 have a very low outlier measure.

The general prediction performance of my model are very low. Can I deduce from this plot that there is a subgroup of patient that will very likely get ill?

Basically I would like to say at the medical team that it is difficult to obtain a general model but that we are able to classify people with this characteristics...

enter image description here

$\endgroup$
  • 1
    $\begingroup$ I would think about how to effectively transpose the data, and try to predict the patient by the fact they got ill. I would also consider adding an unsupervised clustering (kmeans + cubic cluster criteria, gmm + AICc) to the input data as an augment for training and see if it substantially improved results. If it did, then I could summarize ill by cluster, and look at the "higher ill-rate" clusters. Look at the Boruta package, you might be able to find the key characteristics that drive the "ill" disposition. $\endgroup$ – EngrStudent Oct 25 '17 at 19:55
1
$\begingroup$

(1) You have an unbalanced dataset (~18%). Random forests may not perform well in this setting. You might want to compensate for this (internal undersampling)
(2) I would argue that you should have a strong justification for not using a logistic model. For many reasons, including that probabilities obtained are believable.
(3) There very likely is a high risk group - there always is. But seems like the "outlier" function is build to detect outliers not high risk subgroups. I wouldn't force it to do this.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I would like to understand your reasons for ruling out a logistic model (2). Thx $\endgroup$ – Soren Havelund Welling Oct 15 '15 at 10:07
  • 3
    $\begingroup$ I wrote the answer poorly. I think the default model for binary outcomes should be a logistic regression model (but this may well be bias of mine). If you chose another model (random forest) - then you should justify this. I haven't seen anything in the question which suggested another method would be preferable. $\endgroup$ – charles Oct 15 '15 at 17:39
  • $\begingroup$ + i've been meaning to look at randomFloor but haven't had time yet $\endgroup$ – charles Oct 15 '15 at 17:41
  • $\begingroup$ @charles could you please specify why RF might not perform well on unbalanced dataset? We could just use F1 score instead of accuracy to investigate the fitness. With an undersampling one would also need to adjust the weights because the class distribution in test set would be different from one in train set. – $\endgroup$ – Alina Jul 16 '17 at 22:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.