Accuracy of an estimator is the degree of closeness of the estimates to the true value. For a classifier, accuracy is the proportion of correct classifications. (This second usage is not good practice. See the tag wiki for a link to further information.)
0
votes
3answers
30 views
accuracy of a binary classifier vs probability of binomial distribution
I got confused over a simple concept.
Imagine that I have a binary classifier with 50% accuracy. So, if there are 10 samples to be classified as "y", "n", it has predicted 5 of them correctly.
Now, ...
0
votes
0answers
9 views
Evaluating accuracy of the classifier based on sample?
I have a rather odd question which unfortunately goes beyond my knowledge of stats so any advice will be much appreciated. We built a clustering model on the text data (LDA) and then assigned classes ...
0
votes
0answers
9 views
Difference between Mean/average accuracy and Overall accuracy
I just got confusion while reading the paper "Local Binary Pattern-Based Hyperspectral Image Classification With Superpixel Guidance".
They mentioned that they repeated each experiment 10 times and ...
1
vote
1answer
23 views
Does high training accuracy for an NN mean that it has a potential to reach high validation accuracy?
I saw quite a few discussions related to the problem of high training accuracy with low validation accuracy and what steps to take to address it. I have the same problem with a binary classification ...
3
votes
1answer
36 views
Quantitative evaluations for image classification
Hello I am working on the classification of different weed categories. I want to know what quantitative evaluation I can do other than find the accuracy?
0
votes
1answer
30 views
why does data external from training and testing my neural network perform much worse than statistical accuracy?
I've got a problem with my neural network (used to recognize audio signals, an expansion of the UrbanSound dataset problem): when I fit the model the accuracy of both train and validation is near 90%. ...
2
votes
0answers
21 views
Can F1-Score be higher than accuracy?
I'm using sklearn's confusion_matrix and classification_report methods to compute the ...
1
vote
0answers
43 views
Accuracy metric for comparing Time Series models?
I'm writing a blog post on forecasting time series with autoregression. In it, I compare the performance of SLR, ARIMA, and SARIMAX on forecasting the number of Home Sales in Seattle (see below).
All ...
0
votes
0answers
7 views
Can the size of the test set affect the results?
I have a language model, a bi-LSTM, which predicts the next word.
I divide the corpus in 80% training, 10% development and 10% test and I obtain 51% accuracy. Afterwards I wanted to test with a ...
0
votes
1answer
41 views
Incomprehensible lowering of VGG19's accuracy if data is preprocessed
I'm using the VGG19 convolutional network for image recognition (to be specific, the keras implementation). I've downloaded the ILSVRC2012 validation dataset and the ground truth from here, to check ...
0
votes
0answers
6 views
What features or architecture for a text answering system?
From this dataset with paragraphs, questions about these paragraphs and answers from these paragraphs, when there exists, I'm trying to predict the sentence where there is an answer. After ...
1
vote
1answer
32 views
What influences fluctuations in validation accuracy?
This is the Tensorboard output of some machine learning experiments I'm doing.
What are the factors that influence the entity of the fluctuations in the validation accuracy? For example: the batch ...
2
votes
1answer
121 views
Improving spam classification with tensorflow logistic regression
I would like to classify a mail (spam = 1/ham = 0), using logistic regression. My implementation is similar to this implementation and using tensorflow.
A mail is represented as a bag-of-words ...
0
votes
1answer
25 views
Accuracy percent seems too high in Neural Network
I built an artificial neural network that has a dependent variable called "Suspicious". This column is binary so only two outcomes. I have 297,771 "0" not suspicious or known good. Then I have only ...
0
votes
0answers
22 views
reconstruction loss for disentangled variational autoencoders
I am using disentangled variational autoencoders which is a variant of VAE. You will find the github code in this link.
I want to quantify the difference or the loss between the ground truth (...
-1
votes
1answer
47 views
0
votes
1answer
38 views
Is there an approach to predict the accuracy of a model without the test_labels? [closed]
I am working with the Loan Prediction Problem. My testset however does not contain the test labels. Is there any way in which accuracy score could be obtained without the test labels?
0
votes
0answers
9 views
When do we have equal accuracy and f1-score in binary classification?
I am working on a link prediction method and when I run the code, the output of accuracy and f1-score are equal in all iterations. I can't interpret the results. The number of positive and negative ...
0
votes
0answers
8 views
Feature selection using PArticle swarm optimization
I want to perform classification using weka tool but I am keen to do feature selection using PSO. I am using cocomo dataset for ...
0
votes
0answers
27 views
AUC and Accuracy baseline
I implemented the different classification algorithms like Bayesian network, Decision tree or Naive Bayes, on my data to predict the right class (binary classes). By considering confusion matrix, I ...
4
votes
1answer
94 views
Is accuracy an improper scoring rule in a binary classification setting?
I have recently been learning about proper scoring rules for probabilistic classifiers. Several threads on this website have made a point of emphasizing that accuracy is an improper scoring rule and ...
0
votes
0answers
36 views
Keras accuracy not affected by any change in model [duplicate]
I am trying to construct a model for single-label multiclass classification using Keras in a Jupyter notebook. Here's my model (or see full jupyter notebook):
...
2
votes
1answer
104 views
Calculating the accuracy of matching between two sets of strings lists
To give some background - the question is about measuring the accuracy of name disambiguation algo results (and not about the algo itself)
Let's say we have three groups of entities, each ...
2
votes
2answers
16 views
Distribution of weighted accuracies for classification
A long time ago I seem to remember learning that rather than just assigning a 0 or 1 accuracy values for samples, one could calculate the weighted accuracy of a sample based on the confidence level of ...
0
votes
1answer
28 views
Educational purpose :Calculation of sensitivity and specificity from confusion matrix for this example
Consider a case where the number of labelled data as 0 = 1400 and labelled as 1 =100. The data labelled as 0 denote normal operating conditions and data labelled as ...
-1
votes
1answer
34 views
Clustering evaluation metric when overclustering is of [closed]
What evaluation metric should be used to measure clustering performance when over-clustering is OK as long as it happens only within ground truth clusters, with no confusion of ground truth clusters.
...
0
votes
0answers
12 views
Accuracy in multi-label classification [duplicate]
In a multi-label classification, the accuracy is commonly defined as [1]
$$
\text{Accuracy}(\boldsymbol{Y},\, \boldsymbol{Z}) := \frac{1}{n} \sum_{i=1}^n \frac{\lvert Y_i \cap Z_i \rvert}{\lvert Y_i \...
0
votes
0answers
19 views
Is there a way to find the overall probability of the model fitting if it fits all models?
I have developed 4 models, using machine learning.
I have been working on using machine learning on molecular fingerprints, to determine if new compounds are active or inactive.
If i have 4 models, ...
0
votes
0answers
15 views
Sample size for F1 Score
I'm planning on running study to evaluate the effectiveness of a neural network classifier. The output will be binary and I'l have ground truth data to compare it against.
In terms of reporting the ...
1
vote
0answers
34 views
Alternative measurement for Cross Entropy Loss than accuracy when using soft labeling?
Suppose one has multi-class ($n>2$) data, where class labels are soft, e.g. two samples of for 3-classed data might look like:
...
1
vote
1answer
96 views
sklearn.metrics.accuracy_score vs. LogisticRegression().score?
I'm currently testing some models on a simple binary classification task, however, I've found a strange discrepancy between two accuracy score metrics from SK Learn: sk_learn.metrics.accuracy_score ...
1
vote
1answer
43 views
Why would my cross validation accuracy be noticeably lower than my actual testing accuracy?
I coded up a nested cross validation scheme for model selection, feature selection, and hyper parameter optimization. Here are some results I got:
Model selection accuracies (for the two models I ...
0
votes
0answers
22 views
Not smooth validation accuracy vs test accuracy. When to stop
I have a convolutional model with this training history.
Validation accuracy is growing slowly but it is almost flat.
The net is a two conv+maxpool layers and 3 dense layers. The training dataset ...
0
votes
0answers
20 views
Improving cold-start recommend systems
I am using tensorrec (python framework for recommendation, based on tensorflow) to predict a users choice of content, based on the users meta-data. My current accuracy is at about 2,5% *. Since this ...
4
votes
2answers
116 views
Performance measure for correct/incorrect guesses
I have an algorithm that makes 'informed guesses' about a set of inputs; the guess is either correct, incorrect, or missing (no guess made). From the correct/incorrect/missing guesses I can calculate ...
4
votes
3answers
101 views
How do I call a forecast that is both accurate and precise?
How do I call a forecast (more precisely, a forecasting rule) that is both accurate and precise?
Is there a word that expresses both properties combined?
I do not mean the forecasting rule is perfect,...
3
votes
1answer
40 views
Accuracy very different from CV
In my random forest classifier in Python (2 classes), I am getting an accuracy score of 46.9 % (using accuracy_score from sklearn.metrics).
However, when I print clf.cv_results_['mean_test_score'], ...
1
vote
1answer
231 views
LSTM time series forecasting accuracy [closed]
EDIT3: [Solved] I experimented with the LSTM hyperparameters and tried to reshape or simplify my data, but that barely changed the outcome. So I stepped back from LSTM and tried a simpler approach, as ...
2
votes
2answers
91 views
Time Series Forecast - Complex seasonality
I have a daily time series that I am having issues forecasting accurately.The time series is stationary and it looks I have tried ARIMA(3,1,1),(0,1,1)- 7 Period, auto.arima(D=1), Holt-Winters, nnetar,...
0
votes
0answers
24 views
In which cases can SVM totally fail but Decision trees/Multilayer Perceptrons succeed in classification?
I have been working with a binary classification problem recently. The dataset contained 20 features.
I used SVM with different parameter settings and got 50% accuracy (i.e. all predictions are ...
0
votes
0answers
92 views
Recall equals to accuracy but different to precision
I've read this question, and basically I'm having the same issue.
I'm dealing with a binary classification problem.
I'm calculating the precision, recall and f1 using ...
2
votes
0answers
49 views
Difference between cumulative gains chart and cumulative accuracy profile for binary classifier
I am confused about the following:
Here I find the definition of cumulative gains chart as the plot of
x-axis: the rate of predicted positive
y-axis: true positive rate
It turns out that we can e....
0
votes
0answers
15 views
What is the difference between precision and accuracy in Statistics? [duplicate]
Are they the same, do they depend on each other or do they vary?
0
votes
1answer
28 views
Plotting the accuracy of cutpoints in a logistic regression model
I would like to perform a simple logistic regression and draw a plot showing how the accuracy (that is "(Σ True positive + Σ True negative) / Σ Total population") change through different cutpoints, ...
0
votes
0answers
17 views
Is it more accurate to measure an object with 4 scales, or 1 scale?
If I know the relative uncertainty of an instrument, and I take multiple measurements of the same thing with that, how do I know my new relative uncertainty?
For example:
Does my relative uncertain ...
0
votes
0answers
88 views
What is the meaning of AUC being high when accuracy is not? [duplicate]
I'm testing several classifiers in Weka Experimenter.
Some of them have — at the same time — low accuracy (Percent_correct statistic) and high AUC.
How should the quality of such ...
1
vote
0answers
17 views
Can i use Diebold Mariano test for comparison of 2 models across multiple time series?
I have 2 models (for simplicity, let's call them AR(1) and MA(1)) each making 1 day ahead forecasts of time series.
If I had only 1 time series I would just use ...
0
votes
1answer
34 views
R train function input multiple metrics [closed]
I want to get multiple metrics results like the code below but it didnt work because of metric = list("ROC","F1","Accuracy","Kappa"). It works perfect for ...
0
votes
1answer
205 views
How to report confusion matrix for repeated K-fold cross-validation?
I am trying to construct confusion matrices in R with CARET package for repeated K-fold cross-validation, specifically, 10-fold cross-validation with 10 repeats.
I realized there was already a ...
0
votes
1answer
56 views
Sampling design to measure the performance of a classifier on large set of new data
I have an certain labeled data set suitable for a classification task. A classifier was optimized on this labeled data.
Time after this I received a very large set of the same type of data, albeit ...
