35
votes
Accepted
Are there parameters where a biased estimator is considered "better" than the unbiased estimator?
One example is estimates from ordinary least squares regression when there is collinearity. They are unbiased but have huge variance. Ridge regression on the same problem yields estimates that are ...
- 128k
22
votes
Are there parameters where a biased estimator is considered "better" than the unbiased estimator?
Yes there are plenty of cases; you're beating around the bush that is the topic of Bias-Variance tradeoff (in particular, the graphic to the right is a good visualization).
As for a mathematical ...
- 1,046
15
votes
Is it okay to say that 95% confidence interval is more significant than 80%?
The language of statistical "significance" is not directly applicable to a confidence interval, though intervals are often related to similar hypothesis tests. If you were to compare a ...
- 133k
10
votes
Accepted
What is the difference between accuracy and precision?
(Just for reference, I am posting my comments as an answer. Note that the first version of the question did not include the formula.)
"Accuracy" and "precision" are general terms throughout science. ...
- 13.1k
9
votes
How to compute accuracy for multi class classification problem and how is accuracy equal to weighted precision?
I've got a wonderful solution and a perfect understandable solution for this problem as I was looking for same from this Question
You can calculate and store accuracy with:
...
- 201
9
votes
Are there parameters where a biased estimator is considered "better" than the unbiased estimator?
There are numerous examples where the MLE has smaller mean square error (MSE) than the best available unbiased estimator (though often there are estimators with smaller MSE still). The "standard" ...
- 291k
9
votes
How do you interpret the area under the precision-recall curve?
Yes, it is average precision, where the average is taken across different thresholds for saying "yes".
The precision-recall curve typically starts out relatively high, and descends though ...
- 416
9
votes
Is it okay to say that 95% confidence interval is more significant than 80%?
@Ben gave an excellent answer (+1) I'm just adding to it.
You have to decide which error is worse and by how much. In psychology, where I did most of my work, there's a strong convention for power of ...
- 128k
7
votes
ROC/AUC Curve for False Negatives (Type 2 Errors)?
The ROC curve is the curve
$$f(t) = (\mathrm{FPR}(t), \mathrm{TPR}(t)),$$
for a threshold $t \in \mathbb R$.
You are proposing a new curve $g(t) = (\mathrm{FNR}(t), \mathrm{TNR}(t))$, but remember ...
- 19.5k
7
votes
How does population size impact the precision of the results?
"For simplicity let's assume that in both cases the sampling fraction is small enough for no finite sampling/population correction (FPC) to be required." But the finite population correction ...
- 28.3k
7
votes
Judging a model through the TP, TN, FP, and FN values
Do not use any of accuracy, precision, recall, or the F1 score. They all suffer from the same issues, especially - but not only - for "unbalanced" data: Why is accuracy not the best measure ...
- 131k
6
votes
How does population size impact the precision of the results?
Intuitively, the same sample from a smaller population should have less variance...
That is correct. Suppose you have a population of size $N$ and a sample of size $n$.
If the underlying (infinite) ...
- 133k
5
votes
Accepted
Precision of Success
This is the oft-encountered problem of estimating a "confidence interval for a Binomial proportion". There are several different techniques, which vary in their assumptions and are well described on ...
- 13.1k
5
votes
Accepted
accuracy and precision in regression vs classification
As you've pointed out, they are not the same, and sometimes refer to wildly different things (i.e., precision is a property of the model in classification, and refers to a measure of variance in ...
5
votes
How many significant figures should I report for a regression equation?
My eyebrows go up when numbers are reported with far too much precision, too, but there's more going on in a regression setting than we might expect. So much, so, that I won't venture a thorough ...
- 334k
5
votes
Combine accuracy, precision, and recall
This approach is questionable. Instead use establish statistical principles as detailed here. Here are a few.
Turn the problem into a prediction problem instead of a classification problem, by ...
- 98.5k
5
votes
Accepted
ROC/AUC Curve for False Negatives (Type 2 Errors)?
Since there are just two categories, and the TNR and FNR are so related to the TPR and FPR, this sounds like a graph that would not contain new information. Let's see what happens in a simulation.
<...
- 67.2k
5
votes
Is F-score the same as accuracy when there are only two classes of equal size?
No.
The F-score is a weighted average of precision (accuracy among predicted positives) and recall (accuracy among actual positives). Neither measure says anything at all about the group of true ...
- 10.4k
4
votes
Accepted
Exactness of p-value in testing by simulation
This effect is not limited to p values obtained by simulation. P values are a statistic, i.e., a function of the data you have observed (and your distributional assumptions and model). As such, they ...
- 131k
4
votes
bias-variance tradeoff vs precision and recall
In general precision-recall tradeoff is seen as a discrimination threshold for what we consider positive. A more strict/picky/pessimistic threshold will lead to higher precision (at the cost of ...
- 151
4
votes
Accepted
How to decrease data precision correctly?
This is a fascinating question because (a) this issue does come up in practical work and (b) its solution can reveal hidden information in the data. Such is the case for the 80 values given in the ...
- 334k
4
votes
What are best practices for choosing the beta for an F-measure score?
Don't use F scores at all. Every criticism of accuracy collected at Why is accuracy not the best measure for assessing classification models? applies completely equally to precision, recall and all F ...
- 131k
4
votes
Accepted
False positive rate at K recall
I've found the answer in some paper.
FPR at 95% TPR can be interpreted as the probability that a negative (out-of-distribution) example is misclassified as positive (in-distribution) when the true ...
- 241
4
votes
Accepted
Disadvantage of precision at k
You need to understand
but fails to take into account the positions of the relevant documents among the top k
in the context of a fixed $k$. In your example, with $k=10$, precision at $k=10$ does ...
- 131k
4
votes
Is it okay to say that 95% confidence interval is more significant than 80%?
The term "significance" should optimally be used referring to test results together with a level indication, such as "the data provide a significant indication against the null ...
- 28.3k
4
votes
Calculate area under precision-recall curve from area under ROC curve and the prevalence
You could follow these steps:
read all the pairs of sensitivity and specificity values of the ROC figure (perhaps at steps of 0.01 in sensitivity, there's software that supports you in doing this ...
- 35.3k
4
votes
Is F-score the same as accuracy when there are only two classes of equal size?
NO, see the counterexample below
Classify every instance the same way. You have a recall of $1$, a precision of $0.5$, and an accuracy of $0.5$. However, your $F_1$ score is:
$$
F_1 = 2\dfrac{\text{...
- 67.2k
4
votes
What is the proper formatting of very low p values?
The relationship between p-values and the the "strength" of evidence in data against the null hypothesis is multi-dimensional and wildly non-linear (see https://arxiv.org/pdf/1311.0081), but ...
- 17.3k
4
votes
Judging a model through the TP, TN, FP, and FN values
What scoring rule you should use depends on the particular situation and how good the TP and TN are and how bad FP and FN are.
These vary hugely by situation. For instance, if the event is "the ...
- 128k
3
votes
How good is my recommender?
An easy and widely applicable way to help interpret the performance of a model is to compare it to an appropriate baseline. For example, imagine a trivial model that predicts that everybody will buy ...
- 20.6k
Only top scored, non community-wiki answers of a minimum length are eligible