# Tag Info

6

Answer seems to be: no, because MAE doesn't lead to a proper scoring rule. See Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications where the MAE is discussed under "Counterexamples of proper scoring rules".

4

I think random forests are still mostly used in the form they were introduced by Breiman in his 2001 paper. There have been some attempts to improve them by e.g. moving beyond majority voting (http://link.springer.com/chapter/10.1007/978-3-540-30115-8_34), but my impression that this stuff isn't main-stream practice. You can find a good recent review of ...

4

Bayes error rate quantifies a lower bound on classification error given the inherent overlap/noise between classes. For the two-class case let $\Omega_1, \Omega_2$ be how the model partitions the sample space and $C_i, i=1,2$ be the true class labels. Then \begin{align} berr &= \mathbf{P}(x\in \Omega_2,C_1) + \mathbf{P}(x\in \Omega_1,C_2) \\ ... 2 Evaluating an unsupervised method using supervised measures is somewhat unfair anyway. Obviously, any supervised method can be better. It actually takes the whole concept of unsupervised methods ad absurdum: You use unsupervised methods to discover something new, not something you already knew (but didn't tell your program). So in general, you should ... 2 You can't expect anything similar to a supervised error rate without known class labels, however you should take a look at Prediction Strength which determines the quality of a clustering by how stable it is. My answer to Clustering On Informative Features describes how to compute this value. The idea is to split the data into two halves (train/test sets), ... 2 First, a population that large, 10, is not going to give you anything at all. Even with good predictors from the demographics there will be a great deal of noise that is difficult to associate meaningfully even with more descriptive and elaborate algorithm designs. Second, the particular normalization you use is dependent on the model you decide to ... 2 I'm glad that @cbeleites opened the door ... The concordance probability orc$-index, which happens to equal the ROC area in the special case of binary$Y$, is a nice summary of predictive discrimination. The ROC curve itself has a high ink:information ratio, but the area under the curve, because it equals the concordance probability, has many nice ... 2 A few possibilities come to my mind. Looking at the overall hit rate is usually not a very good idea as it will depend on the composition of the test set if the performance for the different classes differs. So at the very least, you should be specify (and justify) the relative frequency of the classes in your test data in order to derive a meaningful ... 2 First of all: are all hits equally important and all misses equally important? If so, then there is nothing wrong with your null-model scoring that good: it simply is an excellent solution. If you find it important to have a good performance on predicting the 1's, you could use the F-measure instead. It is basically the harmonic mean of recall (what portion ... 2 The use of ROC curves may be misleading when your data is strongly imbalanced. In such cases, precision-recall curves and their AUC is often a better choice. Particularly, the area under the PR curve may differ quite a lot between classifiers with comparable AUROC, even when the AUROC is very high. This paper by Davis et al. is an excellent reference on ... 2 First, I would advise you to not use squared error but the cross entropy error. Squared error results from the assumption that your labels are subject to Gaussian noise, which will probably not be the case. First, the output of your network should be a softmax:$z_k = \frac{\exp{y_k}}{\sum_i\exp{y_i}}$This is basically a logistic regression layer on top ... 1 (This will be rather long, but you are asking for a whole model here). We have a "dependent" variable (Newspaper sales =$Y\$), and we believe that this variable is affected by various environmental factors (like population density in each area), but also it is affected by the sales-employees' "intrinsic" performance. We want to separate these so as to ...

1

I recently did this kind of analysis with CRM data and logistic regression, ANN and recursive partioning regression trees. Software was R. I used R package ROCR to discriminate between results obtained via these different classifiers. But besides AUC I also used various other measures as error rate, accuracy and phi correlation between observed and ...

1

Here is a link to some NIST pages: http://www.itl.nist.gov/div898/handbook/ppc/ppc.htm http://www.itl.nist.gov/div898/handbook/ppc/section5/ppc52.htm http://www.itl.nist.gov/div898/handbook/prc/prc.htm Here is the one that I think is most interesting to you: http://www.itl.nist.gov/div898/handbook/prc/section2/prc2.htm Bottom lines: Mileage is ...

1

Have a look at the Matthews Correlation Coefficient $$MCC = \frac{TP \cdot TN - FP \cdot FN}{\sqrt{ (TP + FP)(TP + FN)(TN + FP)(TN + FN) }}$$ I have seen it pretty often as performance metrix in classification of SNPs dataset. Have a look at this link as well, they discuss the difference between AUC and MCC Otherwise you can just compute an average ...

1

1. To answer the question, you have to define what you mean - I can't tell you what 'consistency' means for your application 2. This depends on (i) what the distribution is (mostly, if N is large, on the variance), (ii) what the comparison measure is, and (iii) what you mean by 'good confidence'. If you specify enough things, specific answers can be ...

1

Don't use the Silhouette measure for arbitrary clustering algorithms. In particular, don't use it for clustering algorithms which can produce non-convex clusters. On data sets such as this: https://en.wikipedia.org/wiki/File:SLINK-density-data.svg the silhouette may be misleading. The silhouette coefficient and silhouette plots are meant to compare ...

1

This is a different question depending on whether you are using a loess or an ARIMA model. I will answer just the loess question for now, as I suspect there are little efficiencies possible in the ARIMA case other than perhaps having a good set of starting values. A loess model works by fitting a weighted regression to different subsets of the data. Only ...

1

I'm not enough of an expert on neural networks (or other fancier machine learning methods like that, i.e. SVMs, etc.) to know what the state of the art is in terms of handling missing data in those areas. However, I do know that decision trees (I'm most familiar with the CART variant) are able to "handle" missing values relatively easily. I suspect that you ...

1

False discovery is a concept due to Benjamini and Hochberg. It came about explicitly to as a criteria for adjusting p-values in multiple testing when a large number of tests are involved. This came up in the context of gene expression analysis using microarrays. The false discovery rate that you are using to evaluate your classifier is the same. There is ...

1

Since no one else has replied, let me give it a shot. As to your first question, the t-test you describe would tell you if the means for E and C were different overall, but you would be getting your error estimate from among the 15 questions rather than from among your many respondents. That is not ideal. You could consider doing a t-test between C and E ...

Only top voted, non community-wiki answers of a minimum length are eligible