2

Using the binary log-loss classification as an objective is a good move in this situation (and in most situations). We might want to point Optuna (or our general hyper-parameter search framework) to minimise the Brier score of the predictions if we care about how much the probabilities might be off; the AUC-ROC is a ranking score, it is better than F1-score ...


1

Average (weighted) ROC-AUC based on all possible class comparisons is appropriate to report.


1

This isn't surprising. A model with more features has a richer space of functions to approximate as compared to a function with fewer features. Feature selection, in my own opinion, is not about increasing performance. On the contrary, it is about finding a set of features which does good enough as compared to the model with the full set. I wrote a small ...


1

Flat regions in a PR curve are generally speaking "good". They effectively imply that we are able to increase Recall (i.e. recognise additional True Positives instances) without inflating the number of wrongly classified true negative results (i.e. avoiding more False Positive instances). When seeing a straight line in a PR curve, it is at times ...


1

ROC-AUC always tracks with precision, recall, and f1-score, and since you are obtaining AUC values of 0.8 with everything else near 0.48-0.5, you are obviously overfitting. You didn't say anything about number of objects, features, folds, or anything about how you are testing objects left out of training. It seems that you are also employing one data set, ...


1

Many thanks to R Carnell for providing the derivation. To increase my understanding, I ran some simulations to validate the theoretical results. First some helper functions. import numpy as np import matplotlib.pyplot as plt from scipy.stats import norm from scipy.stats import bernoulli # expectation of Y given X -- logistic model def EYX(x,beta0,beta1): ...


1

This notation is not perfect, but it helps keep the conditioning straight: $$E(Y|X=x) = \frac{1}{1+e^{-(\beta_0 + \beta_1 x)}}$$ Since $X \sim N(0,1)$ and $N(\mu, 1)$, $$f(X|Y=1) = \frac{1}{\sqrt{2 \pi} \sigma} e^{\frac{-(x-\mu)^2}{2 \sigma}} = \frac{1}{\sqrt{2 \pi}} e^{\frac{-(x-\mu)^2}{2}}$$ $$f(X|Y=0) = \frac{1}{\sqrt{2 \pi}} e^{\frac{-x^2}{2}}$$ ...


Only top voted, non community-wiki answers of a minimum length are eligible