2
$\begingroup$

I was trying to train a Neural network ReLus to learn to learn to predict boolean functions (with +1, -1 instead of 0,1). In particular I tried the parity function which is just the product function of its coordinates. i.e. $f(x) = \prod^n_{i=1} x_i$

Just for fun, I tried training the network with the l2 loss (instead of the softmax). I get that the training error is about $10^{-6}$ but I get that the accuracy (0-1 loss) is slightly bellow chance $0.49$. How is this possible? I know the two are not equivalent but I thought that this was super odd. Does someone know why this is happening? Is it possible to have an extremely low l2 loss but still have a bad accuracy?


I know I could just train things with a softmax (logistic func) and the cross-entropy loss to make the modeling more accurate, but I thought that the above result was super odd. Does someone know what is going on? Is it possible to have an extremely low l2 loss but still have a bad accuracy?


Like I a aware that the loss function essentially decides what is our target function. For example, 0-1 loss leads to bayes decision rule and l2 loss I think gives something like:

$$ f(x) = E[Y \mid X=x]$$

but even if its not the perfect modeling, I expect l2 loss to at least be better than chance...at least by a little bit. Specially in this case since being closer -1 or +1 definitively means at least when we get the sign we are closer to the right target value.

$\endgroup$
1
  • $\begingroup$ Are your categories balanced? $\endgroup$
    – Dave
    May 24, 2023 at 17:16

1 Answer 1

1
$\begingroup$

In the example below, I am able to score chance-level accuracy at a particular threshold (needed to convert your continuous predictions to discrete categories) despite what looks like a low square loss.

# Taken from: https://stats.stackexchange.com/a/46525/247274

library(pROC)
library(MLmetrics)
set.seed(2023)
N <- 1000
x1 <- rnorm(N)           # some continuous variables 
x2 <- rnorm(N)
z <- 1 + 2*x1 + 3*x2     # linear combination with a bias
pr <- 1/(1 + exp(-z))    # pass through an inv-logit function
y <- rbinom(N, 1, pr)    # Bernoulli outcome variable
L <- glm(y ~ x1 + x2, family = "binomial")
preds <- 1/(1 + exp(-predict(L))) 
r <- pROC::roc(y, preds)
thresholds <- r$thresholds
accuracies <- rep(NA, length(thresholds))
for (i in 1:length(thresholds)){
  
  idx1 <- which(preds > r$thresholds[i])
  
  yhat <- rep(0, N)
  yhat[idx1] <- 1
  accuracies[i] <- MLmetrics::Accuracy(yhat, y)
  
}
plot(thresholds, accuracies)
abline(h = mean(y))
mean(y)             # 0.593, so the chance-level accuracy is 59.3%
accuracies[811]     # 0.593, which is chance-level accuracy
thresholds[811]     # 0.9868599 is the threshold giving chance-level accuracy
mean((y - preds)^2) # 0.08799204 looks pretty low to me
r$auc               # Area under the curve: 0.9468

Overall, this logistic regression model, along with a decision rule of using $0.9868599$ as the threshold for categorizing outcomes, scores chance-level accuracy. However, the square loss of the logistic regression model seems rather low at $0.08799204$, and the ROCAUC is quite high at $0.9468$.

Consequently, I am inclined to believe your results to be possible.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.