0
$\begingroup$

I am testing if the sound pressure levels(rms) of shipping noise affect the presence or absence of the fin whales by acoustic monitoring for 4 months of data from one location. Here my response variable is binary with 0 for the absence of acoustic detection and 1 for the presence and the main predictor is the variable "rms" is continuous(dB re 1microPa), along with some environmental variables. My null hypothesis that the shipping noise does not affect the presence of Fin whales detections and my final logistic regression model after completely a forward selection is : glm(formula = FWbi ~ rms + shptnl + day + hour + partday + month + ssh + chlora, family = binomial, data = noise).

I conducted k-fold cross-validation and the confusion matrix is as follows.

crossValSettings_fw <- trainControl(method = "repeatedcv", number = 10, savePredictions = TRUE) crossVal_fw <- train(FWbi ~ rms + shptnl + day + hour + partday+ month + ssh + chlora ,family = binomial,data = noise, method = "glm", trControl= crossValSettings_fw) pred <- predict(crossVal_fw, newdata = noise) confusionMatrix(data = pred, noise$FWbi)

Confusion Matrix and Statistics enter image description here

The dataset/Reference of the actual absence of fin whale and Prediction correctly predicts the absence of fin whale(true positive)- 102078. The dataset/Reference the actual presence but Prediction falsely predict it as the absence of fin whale(false positive)- 4447. The reference of the actual absence of fin whales but the prediction falsely predicts the presence of fin whale(false negatives) -19. The reference of the actual presence of fin whales and the prediction correctly predicts the presence of fin whales(true negative)- 15.

           Accuracy : 0.9581          
             95% CI : (0.9569, 0.9593)
No Information Rate : 0.9581          
P-Value [Acc > NIR] : 0.5284          

              Kappa : 0.006

Mcnemar's Test P-Value : <2e-16 

        Sensitivity : 0.999814        
        Specificity : 0.003362        
     Pos Pred Value : 0.958254        
     Neg Pred Value : 0.441176        
         Prevalence : 0.958126        
     Detection Rate : 0.957948

Detection Prevalence : 0.999681
Balanced Accuracy : 0.501588 

   'Positive' Class : 0

[email protected] 2 0.8425831

[email protected] 2 0.6886156

enter image description here According to my interpretation of this, it can be seen that my false positive is higher than my true negative and seems to cause a type 1 error according to what I know. I would really appreciate if someone could tell me if I interpreted it right or not. If yes, can I fix type 1 errors? And how do I fix it? Also, my data is zero heavy so idk if that is causing the problem? I looked up several sites but couldn't conclude on how to end my dilemma. Please help.

Improve this question
$\endgroup$
8
  • $\begingroup$ Can you include the confusion matrix instead of describing your results in words. Seems like you have a imbalanced dataset. Is there any variable that is associated / correlated with the presence in the first place? $\endgroup$
    – StupidWolf
    Apr 27, 2020 at 17:07
  • $\begingroup$ One way to manipulate Type 1 error rates in logistic regression prediction is to adjust the scoring threshold for classifying predictions into yes-no buckets. For instance, the usual rule-of-thumb is to use a threshold of the median score value, greater than that are yes, otherwise no. $\endgroup$
    – user234562
    Apr 27, 2020 at 17:17
  • $\begingroup$ @StupidWolf Hey I have attached the picture for the confusion matrix. No, I ran correlation tests and vifs to remove correlated variables. My final model does not have any variables that are associated. $\endgroup$
    – Kavya Ramesh
    Apr 27, 2020 at 20:12
  • $\begingroup$ @user332577 Hey thanks for the suggestion. I am very new to R and stats. Can you send me a link or the concept in R that I would need to use to do the scoring threshold for classifying? $\endgroup$
    – Kavya Ramesh
    Apr 27, 2020 at 20:17
  • $\begingroup$ ok this is a bit tricky. You have mostly negative labels, so when you fit a logistic the intercept is going to be really low.. May not be easier for you to get a prediction > 0.5 . Can you try plotting your prediction probabilities against the actual label $\endgroup$
    – StupidWolf
    Apr 27, 2020 at 21:03

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.