New answers tagged

0 votes

Gaussian RBF vs KNN explanation

One thing has to be said at the beginning RBF is not classification algorithm. It is used in SVM as a kernel, because it has properties of the kernel. SVM doeas all the work with finding hyperplane. I ...
3 votes
Accepted

Selecting best classification probability threshold with ROC/AUC doesn't necessarily improve F1 score

You have two metrics, the F1-score on the one hand and Youden's J on the other hand. Those two metrics measure different aspects of the model which are not functions of each other, so you cannot ...
  • 8,694
0 votes

How is the direction of intercept determined?

For linear regression, the intercept can be positive or negative (or 0), and simply represents the value of the mean of Y when the hyperplane (your linear relationship) passes through the point where ...
0 votes

Take into account uncertainty in ground-truth in a classification

It sounds like the uncertainty in your ground truth stems primarily from uncertainty about the true duration and intensity of the phenomena's presence: when the sensors show positive, it's not clear ...
1 vote

Was Amazon's AI tool, more than human recruiters, biased against women?

One additional thought, not fully explored: Maybe we should think whether overfitting can amplify existing bias. In a linear model (where we understand better what happens), we observe overfitting ...
0 votes

Could we explain the disadvantage of imbalanced data mathematically?

Not an explanation but an example illustrating that strong imbalance forces dismissal of the underrepresented class. Let $y = y_1, ..., y_N$ be a sample where $y_k = 0$ except for $y_1 = 1$. Also, ...
  • 1,369
1 vote

F1 score for change point detection

The issue in changepoint detection is that there is no single measure that assesses all the important properties for changepoint detection. At a minimum you want a measure that assesses closeness to: ...
  • 1,167
5 votes

Could we explain the disadvantage of imbalanced data mathematically?

There is indeed a bias with logistic regression and maximum likelihood estimation when the classes are not equal. Below is a demonstration by coding a simulation (example in literature here) ...
0 votes

Best way to impute missing values in a binary variable

You would be better off using something like multiple imputation. There is a package in R called mice for this purpose. The imputation will essentially fill in the ...
7 votes

Could we explain the disadvantage of imbalanced data mathematically?

In general statisticians are not worried about bias in imbalanced data (not a problem per se), since they use probabilistic classifiers like logistic regression the bias (in small samples) of ...
  • 4,548
10 votes

Could we explain the disadvantage of imbalanced data mathematically?

Not a formal proof, but an intuition: Unbalanced data is not a problem per se, the problem is that you don't have many samples to represent the minority class. Imagine a trivial model, where you would ...
  • 121k
3 votes

Could we explain the disadvantage of imbalanced data mathematically?

Not all strongly unbalanced data would result in logistic regression underestimating the probability of the minority in a relevant way. If the data is perfectly separated (and logistic regression is ...
  • 8,694
0 votes

Generalized classification accuracy metric that coincides with accuracy metric when classes are balanced

Yes, there is such a metric and it's called balanced accuracy. It's the arithmetic mean of true positive and negative rates (TPR and TNR(. Here is some intuition on it. The balancing logic explained ...
  • 53.2k
1 vote
Accepted

What happens if we change the threshold probability value for classifying into different class?

In addition to @Dave's pointer in his comment, I'd like to add simply that a threshold is a hyperparameter. You pick the one that gives you the best compromise between false positives and false ...
0 votes

How to compute precision/recall for multiclass-multilabel classification?

In case if you want to see the results directly: from sklearn.metrics import classification_report, confusion_matrix classification_report(y_test, y_pred) This ...
5 votes
Accepted

Binary classification cross validation ROC score - only consider higher confidence class probabilities

When I do this, I lose about 85% of the test samples, but the resulting ROC score of the high confidence test set is boosted to 0.87, which makes it useful for downstream analysis. It sounds to me ...
5 votes

Binary classification cross validation ROC score - only consider higher confidence class probabilities

The ROC curve and corresponding AUC relate to assessing performance across a spectrum of thresholds, not performance at one particular threshold. Consequently, your plan seems to be equivalent to ...
  • 35.8k
2 votes
Accepted

Predicted Probability with XGBClassifier ranging only from 0.48 to 0.51 for either class

As @Adrian already pointed out, the reason the predicted probabilities are close to 0.5 is because the model has very small number of trees 'n_estimators': 15 and ...
  • 36
0 votes

Accuracy Classification Result of Multiclass SVM

It is hard to give a qualified answer without the data, so I'll have to speculate: $100 \times 100$ is a $10,000$-dimensional feature space (and that's if we're taking greyscale only). You have no ...
  • 6,860
0 votes
Accepted

Why is $AUC=0.5$ and a 45-degree line for a ROC curve considered baseline performance?

When it comes to AUC, baseline performance means the performance of a model that has the same distribution of predictions for both classes. From this, the diagonal line and $AUC=0.5$ follow naturally ...
  • 35.8k
1 vote

Why is $AUC=0.5$ and a 45-degree line for a ROC curve considered baseline performance?

The AUC is the probability that a randomly selected positive is ranked higher than a randomly selected negative. So an AUC of 0.5 is the performance of a classifier that does not rank positives higher ...
  • 81.3k
1 vote

How to *formalize mathematically* that a binary classifier has no predictive performance?

This is complicated, since supervised learning can have so many flavors, but a few general principles can lead you to solve special cases as they arise. The first important topic to consider is what a ...
  • 35.8k
2 votes

How do I combine arbitrarily-ranked lists?

From your example it seems that you want to weight the top ranks heavily. 500-1-1 is ranked higher in your view than 3-3-3, and even 1-500-500 should still be rated high. What about this: Rank ...
2 votes
Accepted

How to design cross-validation and testing scheme when N is small?

In addition to the excellent points in Camille's and EdM's answers: Purpose of the model/study Whether something sensible can be done or not depends crucially on the purpose of the modeling. For any ...
2 votes

How to design cross-validation and testing scheme when N is small?

I understand that N is too small. Is it foolhardy to even attempt classification in this scenario? Probably, at least as you propose to proceed. First, for train/test splits to work well, you need on ...
  • 69.1k
2 votes

How to design cross-validation and testing scheme when N is small?

Cross validation is not the only way to compare different models. Several model selection criteria, such as the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC) can ...
0 votes

Importance for Color in X ray imaging for Detection of Pneumonia

transforming RGB to grayscale lead to loss of important information of colours It would be easier to ask this to a radiologist instead of a statistician. From the statistical point of view you try ...
1 vote

Importance for Color in X ray imaging for Detection of Pneumonia

Are we talking classic X ray radiographs or some fancy modern technique ( MRI, PET-CT, Spectral-CT, ...). Classic X ray radiography has no color information. They measure how much of an electric ...
  • 7,997
0 votes

How to find the confidence level of a classification?

Understand what you are modeling, talk to domain experts, collect relevant data. Accept that some things simply cannot be predicted with certainty. How to know that your machine learning problem is ...
0 votes

What impact does increasing the training data have on the overall system accuracy?

I agree with @Serendipity: The performance of neural networks can continually improve as more and more data is provided to the model, BUT the capacity of the model must be adjusted to support the ...
  • 101
7 votes
Accepted

Does statistically simple algos qualify as AI algos?

There's no universally accepted definition of artificial intelligence. Since this directive came from your superiors, it does not matter whether anyone here considers the projects you've proposed to ...
  • 330
1 vote

What is the statistical model for a multi-label problem?

You can think of the multinomial (aka multinoulli) distribution used in multinomial classification as a multivariate discrete distribution obtained in two steps: assign a Bernoulli marginal ...
  • 1,133
0 votes

Resizing input vectors of different lengths

If I understand your use case correctly, it is important to keep the time between the sensor measurements and the failure evaluation approximately comparable. So one approach could be to bin the time ...
  • 8,694
3 votes

What is the statistical model for a multi-label problem?

Let's generalise your example by supposing that we have $K$ types of animal/object. As you correctly point out, if we are dealing with observations where these types are mutually exclusive then we ...
  • 102k
0 votes
Accepted

Measuring response bias from a confusion matrix

The Python package sdt_metrics by Roger Lew implements several non-parametric response bias measures. Unfortunately, the package is not maintained, but the references are still useful. One of these ...
3 votes

What is the procedure to find the optimal decision threshold in an imbalanced classification problem to maximize F1 score?

Find a good probabilistic model (by optimizing a proper scoring rule). Then vary your threshold to optimize F1 - this is a straightforward optimization problem, and bisection search will work well. ...
2 votes
Accepted

How can area under ROC (AUC) be bad when precision, recall, and accuracy are all good?

Wrangling the numbers, I get the following as percentages of the dataset: TP≈0.685037 FP≈0.288597 FN≈0.0164671 TN≈0.00989861 (So it seems you've swapped the positive class. That's not important for ...
  • 3,251
0 votes

Why do we need natural log of Odds in Logistic Regression?

Why do we need natural log of Odds in Logistic Regression? The log of odds is logistic regression by definition. So the question is more like, why do we need logistic regression? For this there are ...
0 votes

Why do we need natural log of Odds in Logistic Regression?

In the equation: $$ ln (p/1-p)= \beta_0 + \beta_1X $$ The range of the right hand term is $(-\infty,+\infty)$ while without log the range of the left hand term $(p/1-p)$ is $(0,\infty)$. Without log, ...
0 votes

Categorical data for binary classification

Yes, you can do it, but be careful with the meaning of your content. All data can change from fine-scale (numeric) to coarse-scale (binary) but it will lose some detail.
12 votes
Accepted

Is higher AUC always better?

AUC is a simplified performance measure AUC collapses the ROC curve into a single number. Because of that a comparison of two ROC curves based on AUC might miss out on particular details that are left ...
6 votes

Is higher AUC always better?

(observed) AUC can be influenced by statistical fluctuations The ROC Curve is usually based on a sample of real world data and taking a sample is a random process. So there is some randomness in the ...
  • 7,997

Top 50 recent answers are included