1
$\begingroup$

Say we have a balanced image dataset of two classes of objects: green light, red light. After running deep NN classification, the model gives about 95% accuracy for both classes.

However, I need to increase the accuracy of green-light predictions only, possibly at the cost of the accuracy of red-light predictions.

That is, I want most predicted green lights to actually be green light. It is OK to have some predicted red lights to be actually green light, because no move in front a green light is not dangerous.

It is NOT OK to have predicted green light to actually be red lights, because moving in front of a red light is dangerous.

Are there any layers/methods I can use?

One method I am trying is outputting a confidence level/confidence score. Predict green-lights only if the confidence level is high. But NN models are usually bad at confidence level.

$\endgroup$
8
  • 1
    $\begingroup$ Do you want to to improve the probability that a light is predicted green given that is is green, or the probability that the light is green given that it is predicted to be green? They are not the same, and the former (roughly) corresponds to sensitivity. $\endgroup$
    – Dave
    Aug 8, 2022 at 1:10
  • $\begingroup$ The probability that the light is green given that it is predicted to be green. I think this is called "accuracy"? Correct me if I am wrong $\endgroup$
    – High GPA
    Aug 8, 2022 at 1:12
  • $\begingroup$ Accuracy is related to the probability of predicting class K given that a observation belongs to class K, and it applies across all categories. $\endgroup$
    – Dave
    Aug 8, 2022 at 1:44
  • $\begingroup$ @Dave OK to be clear, I am OK with some false negative of green, but false positive of green is intolerable $\endgroup$
    – High GPA
    Aug 8, 2022 at 3:15
  • 1
    $\begingroup$ If you call something green, there’s always some chance that it is not green. If calling something green when it is not green is as intolerable as you say, then you can’t call anything green. If this is an unacceptable solution, then you might be interested in the usual links I post about probability outputs of “classification” models that are collected in a Meta post This topic tends to come up in the context of imbalance, but it applies to balanced classes, too. $\endgroup$
    – Dave
    Aug 8, 2022 at 3:37

1 Answer 1

3
$\begingroup$

Here is a "toy" where you can adjust the threshold to modify the classification.

code:

### LIBRARIES

library(pacman)

p_load(dplyr,       #munging
       ggplot2,     #graphing
       here,        #local reproducibility   
       stringr,     #string printing
       ModelMetrics #confusion matrix
)

### PARAMETERS

num_samp   <- 10000  #how many samples
class_prob <- 0.95   #what percentile overlaps
threshold <- 1       #split line

### MAIN CODE


#distribution parameters (derived)
mid <- qnorm(class_prob)
mu0  <- -mid; sig0 <- 1
mu1  <- mid; sig1 <- 1

#sampling
y0 <- rnorm(num_samp, mean=mu0, sd=sig0)
y1 <- rnorm(num_samp, mean=mu1, sd=sig1)

#assembly
df <- rbind(cbind(0,y0),cbind(1,y1)) %>% as.data.frame()
names(df) <- c("label","value")
df$label <- as.factor(df$label)

#shuffle the data
df <- df[sample(1:nrow(df),nrow(df),F),]

#plot
ggplot(data=df, aes(x=value)) + 
  geom_density(aes(color=label)) +
  geom_histogram(aes(y=..density..,fill=label), 
                 position = "identity",
                 alpha=0.2, bins=35, col=I("gray")) +
  geom_vline(xintercept = threshold, colour="Black")

#basic predictor

pred <- ifelse(df$value <= threshold,0,1) %>% as.factor()

#confusion matrix
confusion_matrix <- table(pred, df$label)
print(confusion_matrix)

#true/false pos/neg rates
tpr_0 <- confusion_matrix[1,1]/sum(confusion_matrix[,1])
fnr_0 <- 1-tpr_0

tnr_0 <- confusion_matrix[2,2]/sum(confusion_matrix[,2])
fpr_0   <- 1-tnr_0

#printing
print( str_c( "tpr:", tpr_0, "(what percent of predicted zeros are actual zero)", sep=" "  ) ) 
print( str_c( "fpr: ", fpr_0, "(what percent of predicted zero are NOT actual zero)", sep=" "  ) ) 

print( str_c( "fnr: ", fnr_0, "(what percent of predicted ones are actual zero)", sep=" "  ) ) 
print( str_c( "tnr: ", tnr_0, "(what percent of predicted ones are actual ones)", sep=" "  ) )  

A sample output plot looks like this: enter image description here

The dividing line is shown as the vertical black line. Anything to the left of it is classified as class "zero", and anything to the right is "one". You can see how the true/false rates change as you modify the threshold.

If you want to get more into the code, you can change how it works by modifying sample size and rerunning it several (~30-ish) times to get a sense of how sample-size impacts disposition, or by modifying classification probability (overlap of distributions) to see how that changes things.

The "confusion matrix" that shows how predictions line up with actuals, looks like this:

> print(confusion_matrix)
    
pred    0    1
   0 9961 2605
   1   39 7395

In it you see that with the current threshold, the model predictions contain 9961 of the actual zeros but 2605 1s. It is greedy for 0's, but that means it has to accept more 1's. In that process it missed only 39 of the 0's, but when it said something was a "1", it was only correct for 7395 of the samples.

Because this is from actual, not-horrible, random number generation every run is different from the others. To compute actual statistics we get the last 4 lines.

Here is a sample output:

> print( str_c( "tpr:", tpr_0, "(what percent of predicted zeros are actual zero)", sep=" "  ) ) 
[1] "tpr: 0.9961 (what percent of predicted zeros are actual zero)"

> print( str_c( "fpr: ", fpr_0, "(what percent of predicted zero are NOT actual zero)", sep=" "  ) ) 
[1] "fpr:  0.2605 (what percent of predicted zero are NOT actual zero)"

> print( str_c( "fnr: ", fnr_0, "(what percent of predicted ones are actual zero)", sep=" "  ) ) 
[1] "fnr:  0.00390000000000001 (what percent of predicted ones are actual zero)"

> print( str_c( "tnr: ", tnr_0, "(what percent of predicted ones are actual ones)", sep=" "  ) )  
[1] "tnr:  0.7395 (what percent of predicted ones are actual ones)"

Again, if you have real world sample sizes, like under 100 tests, then you need to change the sample size, and then rerun this many times to get a sense of what the decisions mean. This also grossly assumes that your distributions are normally distributed, and that they have the same variance parameter (which can be derived from the standard-deviation parameter). Some things act like a normal distribution, in a gross sense, but many times when you get many samples and drill deep, the physical distribution can be things like Weibull, Poisson, or Gamma. If you are applying this to things with money or business behind them, get a paid human statistician (consultant) to make sure the analysis isn't junk.

I like to think of these as trying to catch defects in a production process. If they get to your customer, it costs them money so it costs you money. I like to think of those as "escapees". If you throw away good products, it costs you the cost of producing what you threw away. I think of those as throwing away good, as false positive disposition as defect. Maybe the metaphor can help you make sense of the problem in a useful way.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.