Hot answers tagged performance
4
Bayes error rate quantifies a lower bound on classification error given the inherent overlap/noise between classes. For the two-class case let $\Omega_1, \Omega_2$ be how the model partitions the sample space and $C_i, i=1,2$ be the true class labels. Then
$\begin{align} berr &= \mathbf{P}(x\in \Omega_2,C_1) + \mathbf{P}(x\in \Omega_1,C_2) \\
...
4
Answer seems to be: no, because MAE doesn't lead to a proper scoring rule.
See Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications where the MAE is discussed under "Counterexamples of proper scoring rules".
2
Evaluating an unsupervised method using supervised measures is somewhat unfair anyway. Obviously, any supervised method can be better.
It actually takes the whole concept of unsupervised methods ad absurdum:
You use unsupervised methods to discover something new, not something you already knew (but didn't tell your program).
So in general, you should ...
2
You can't expect anything similar to a supervised error rate without known class labels, however you should take a look at Prediction Strength which determines the quality of a clustering by how stable it is.
My answer to Clustering On Informative Features describes how to compute this value. The idea is to split the data into two halves (train/test sets), ...
2
First, I would advise you to not use squared error but the cross entropy error. Squared error results from the assumption that your labels are subject to Gaussian noise, which will probably not be the case.
First, the output of your network should be a softmax:
$z_k = \frac{\exp{y_k}}{\sum_i\exp{y_i}}$
This is basically a logistic regression layer on top ...
1
Have a look at the Matthews Correlation Coefficient
$$MCC = \frac{TP \cdot TN - FP \cdot FN}{\sqrt{ (TP + FP)(TP + FN)(TN + FP)(TN + FN) }}$$
I have seen it pretty often as performance metrix in classification of SNPs dataset. Have a look at this link as well, they discuss the difference between AUC and MCC
Otherwise you can just compute an average ...
1
1. To answer the question, you have to define what you mean - I can't tell you what 'consistency' means for your application
2. This depends on (i) what the distribution is (mostly, if N is large, on the variance), (ii) what the comparison measure is, and (iii) what you mean by 'good confidence'. If you specify enough things, specific answers can be ...
1
Don't use the Silhouette measure for arbitrary clustering algorithms. In particular, don't use it for clustering algorithms which can produce non-convex clusters.
On data sets such as this:
https://en.wikipedia.org/wiki/File:SLINK-density-data.svg
the silhouette may be misleading.
The silhouette coefficient and silhouette plots are meant to compare ...
1
This is a different question depending on whether you are using a loess or an ARIMA model. I will answer just the loess question for now, as I suspect there are little efficiencies possible in the ARIMA case other than perhaps having a good set of starting values.
A loess model works by fitting a weighted regression to different subsets of the data. Only ...
1
I'm not enough of an expert on neural networks (or other fancier machine learning methods like that, i.e. SVMs, etc.) to know what the state of the art is in terms of handling missing data in those areas.
However, I do know that decision trees (I'm most familiar with the CART variant) are able to "handle" missing values relatively easily. I suspect that you ...
1
False discovery is a concept due to Benjamini and Hochberg. It came about explicitly to as a criteria for adjusting p-values in multiple testing when a large number of tests are involved. This came up in the context of gene expression analysis using microarrays. The false discovery rate that you are using to evaluate your classifier is the same. There is ...
1
Since no one else has replied, let me give it a shot. As to your first question, the t-test you describe would tell you if the means for E and C were different overall, but you would be getting your error estimate from among the 15 questions rather than from among your many respondents. That is not ideal. You could consider doing a t-test between C and E ...
Only top voted, non community-wiki answers of a minimum length are eligible
