The tag has no usage guidance.

learn more… | top users | synonyms

0
votes
0answers
6 views

Performance metrics, prioritizing based on impact

I apologize in advance for the poorly worded question. If anyone has any suggestions, please advise. Also, help with tagging appropriately would be appreciated. I'm collecting performance metrics ...
1
vote
2answers
41 views
+50

Visually summarising multiple diverse metric results

I have a multivariate time series (climate fluxes), and I have a number of model outputs that I want to asses. I also have number of metrics that I can use to assess the performance of each variable ...
0
votes
0answers
18 views

Performance measure

I am currently trying to show the trend of different team performances based on what percent of customer complaints they receive that they are able to handle within a given month. So if one team ...
2
votes
1answer
62 views

CART with Ordinal Response Variable using rpartScore Stuck

I'm trying to fit a decision tree over some data which has ~40K rows and ~200 features. The response variable, y, is ordinal and takes values ...
2
votes
1answer
26 views

What should the AUROC be on the test set when no positive example is present?

Assume we have a probabilistic, binary classifier. We compute the AUROC on a test set in which no positive example is present (i.e. the ground truth is always 0). What should the AUROC be?
0
votes
0answers
9 views

suitable flexibility of statistical learning method and number of predictors and number of samples

Out of a flexible and an inflexible statistical learning method, which one is expected to perform better when the number of samples is extremely large and the number of predictors is small? the ...
4
votes
1answer
84 views

What's the reasoning behind presenting unvalidated AUC as a measure of model fit or performance?

Often one sees, particularly in the biomedical literature, papers that analyze the performance of a risk prediction model in terms of the AUC or the area under the ROC curve. If the AUC is suitably ...
1
vote
0answers
16 views

Taking longer sample should not reduce the variance of the mean

I have not realized completely why n times longer samples should give you n times shorter variance. However, I measure performance of my programs and there are some fluctuation due to PC state ...
0
votes
1answer
21 views

Relative speed of gaussian process classification

I used gaussian process classification implemented in Matlab gpml toolbox and also in R kernlab. For my problem - 600*14 matrix with two classes - it trains order of magnitude slower than other ...
4
votes
2answers
111 views

How to compare (probability) predictive ability of models developed from logistic regression?

I know some well-known measures are $c$ statistic, Kolmogorov-Smirnov $D$ statistic. However, as far as I know, those statistics take into account only of the rank order of the observations, and is ...
1
vote
0answers
18 views

Which validation method to choose if the number of instances is limited?

I have a question regarding how to measure the performance of a model when the distribution of instances per class is limited. In my scenario I have five data sets $\mathcal{D}_{1}, \mathcal{D}_{2}, ...
0
votes
2answers
39 views

Leave one out cross validation for neural network perfomance

When using leave one out cross validation in neural network, do I have to fix the epoch number for each training model? The test results of these models are averaged to show performance. So can I ...
3
votes
1answer
57 views

Statistical test to compare method performance over multiple problems

I am comparing several methods of ordering training patterns for on-line neural network training. Let's call those methods I-VII. Then, I have a sample of 20 data sets within a given "data set type" ...
0
votes
0answers
25 views

How to build kpi on employee<>customer<>sales

For further analysis i want to build a KPI which describes the "critically" of a employee<>customer relationship. To make it clear, some example data: Employee - Customer - ...
2
votes
0answers
37 views

Is it possible to explain the performance of a classifier by the specific properties of a data set?

I receive the following comment from a reviewer : "I think that the authors could explain in more details the results. For instance, there is no discussion linking the specific properties of the ...
0
votes
1answer
110 views

Bad Performance of PMML Evaluator

I have been working on a proof-of-concept setup of evaluating various PMML-based models. Specifically, I've been using the JPMML evaluator (https://github.com/jpmml/jpmml-evaluator), which is in Java, ...
0
votes
0answers
28 views

Why can I use the posterior probability of a classifier as a new classifier?

I have read that, when doing discriminant analysis, you can use the posterior probability you obtain using your classifier as a new fine-tuned classifier. Can anyone talk me through the rationale of ...
2
votes
1answer
39 views

Representative transaction KPIs - what method to chose (average, median…) / how?

In my dataset i have about 30k of transactional data. For analysis, I want to show the end user some important KPIs: Average of transactions with loyalty cards Average turnover of these transactions ...
0
votes
0answers
63 views

Difference between regression and performance plot of Artificial neural network in MATLAB

I am having problem understanding regression and performance plots of ANN. My data consists of 13 inputs and 3 outputs. Parameters used for simulation can be found here. The problem I am facing is ...
2
votes
1answer
49 views

So many significant explanatory variables and so small auc

Have you ever seen a model with almost every significant variable and such small auc (area under the ROC curve) ? What might be the cause of it? When I saw summary of a model I thought this model will ...
2
votes
1answer
52 views

How to measure loss of performance of clustering by applying dimensionality reduction

Let's suppose I have a given dataset with $n$ features. Having a data-centric approach, I would like to measure the loss of performance of applying a given dimensionnality reduction technique, for a ...
0
votes
0answers
16 views

Usage of precision and recall vs type I and II errors

What is the advantage of using precision and recall, instead of type 1 and 2 errors? How about disadvantage? The two groups can be derived from each other. But it seems that they are used often in ...
0
votes
0answers
38 views

How use F1 score in an unbalanced binary classification problem?

I have two trained models (MLP and SVM) that want check on unbalanced binary samples (out of sampl - True samples =3000, False samples = 200). I found that i can use ...
1
vote
0answers
14 views

paired subject design test of proportions

I am analyzing a performance data set and would like to compare across two different conditions. I have three subjects, which were tested over a number of trials. Each trial had a chance of being in ...
0
votes
0answers
33 views

Why added predictors improved performance in decision trees if they don't appear in the model?

although I've been working some months now with decision trees, I still have issues understanding some things and also finding a right source to answer my questions. Maybe I'm not using the right ...
1
vote
0answers
18 views

Measure performance of a multiples classifier that is a combination of binary support vector machines

I am comparing the performance of multiple schemes to combine binary SVMs into one multiples classifier. I was hoping to use the cross entropy, however I cannot figure out how to apply it. For ...
10
votes
3answers
1k views

Why isn't the holdout method (splitting data into training and testing) used in classical statistics?

In my classroom exposure to data mining, the holdout method was introduced as a way of assessing model performance. However, when I took my first class on linear models, this was not introduced as a ...
2
votes
0answers
29 views

does the area under the curve (AUC) has some interpretation? [duplicate]

I know that the ROC curve plots true positives vs false positives, but does it have any other interpretation, or is it just an arbitrary performance measure? Also, in the case of very unbalanced ...
0
votes
0answers
44 views

Graph generation model effect on performance of Spectral graph algorithms

I use spectral graph algorithms for finding community structures, specifically the Leading Eigenvector Method (http://arxiv.org/abs/physics/0605087). I try analyzing the performance of these ...
0
votes
1answer
23 views

Ranking performance measure

I have a system able to produce a ranking of some operations according to their anomalous score. I have N anomalous operations inside the dataset, which I expect to be in the first N positions (best ...
0
votes
2answers
388 views

SVM and SMO main differences

I am unable to clearly see the main differences between SVM & SMO. While I get the fact that SMO provides better algorithm for QP solvers but I see that when I use this in Weka on my MacBook it ...
2
votes
2answers
242 views

What are good criteria for performance evaluation of algorithms in a regression problem?

I'm classifying different algorithms on the wine quality dataset. The quality ranges from between 0 - 10 based on 11 other attributes. Here is the data. I'm treating this as a regression problem. ...
0
votes
0answers
124 views

Combining different metrics to calculate composite score

Let's assume I have 3 metrics to evaluate a student. 1)Test Marks (0-50) 2)Weight of the student (KGs) 3)% Attendance (0%-100%) I want to calculate a total composite score which is a function of ...
0
votes
1answer
94 views

Interpreting high P-Value in ordinal rank correlation measures

I'm currently evaluating a machine-learned ranking task. Two (among other) performance measures are Spearman's Rho and Kendall's Tau (rank or concordance correlation). I'm basically comparing a ...
1
vote
0answers
57 views

Slow Lasso Performance Using sklearn

I am trying to use scikit-learn's LassoCV and/or ElasticNetCV functions to model a dataset with a large (>800) number of ...
2
votes
2answers
633 views

Levinsohn-Petrin TFP estimation in Stata without proxy

I'm trying to compute TFP by using levpet Stata command. I use revenue estimation, yet it requires proxy option which is intermediate input (usually material ...
0
votes
0answers
17 views

Benchmarking Software Exercises

I am building a website for practicing programming in various languages. The content is user driven; some users write exercises with test frameworks for validating solutions, and other users write ...
1
vote
1answer
34 views

Evaluating a fixed classifier

I have a classifier that is fixed and wish to evaluate its predictive performance using a test dataset. I'm familiar with the situation (e.g. in k-fold CV) where the data is split and the classifier ...
1
vote
0answers
32 views

Does within-group heterogeneity negatively impact random forest classification?

I have two rather conceptual questions about random forest classifiers. Before we get there, I quickly want to lay out the problem I am working on: I have large a large data set consisting of 300 ...
2
votes
0answers
253 views

Unbalanced dataset - ROC curve to compare classifiers?

I use the machine learning software WEKA for data mining on biological data. I would describe my dataset as unbalanced: It comprises around 2000 instances, ...
1
vote
1answer
62 views

Possible values for sensitivity, specificity, precision and accuracy

I have some results where the tester claims the following values: Sensitivity: 0.525, Specificity: 0.925, Precision: 0.516, Accuracy: 0.907 Where Sensitivity=TP/(TP+FN), Specificity=TN/(TN+FP), ...
0
votes
2answers
287 views

How is the “classification error rate” of an artificial neural network calculated?

Frequently I see artificial neural networks compared by their "classification error rates" or "error rates", particularly for multi-class problems like CIFAR-10. What does this error rate actually ...
2
votes
1answer
113 views

For a model like this what performance measures can I calculate and how?

Methods: From the machine learning literature, I understand different parameters can show performance of model in machine learning. I would briefly expand my understanding with confusion matrix: ...
2
votes
0answers
72 views

How do I measure a speedup?

I have two versions of the same program. I apply a number of performance tests to both and measure their response time, for instance https://gist.github.com/valtih1978/d2cc2fe96fbbe1987ada ...
1
vote
1answer
72 views

How to speed up Kernel density estimation

Background A couple of days ago I asked here about: How to reduce number of points for clustering . Instead of reducing number of points, a method "Kernel Density Estimation" (KDE) was suggested to ...
0
votes
0answers
46 views

Performance metric

Let's say I have two people, A and B, who are each performing a different task. They have an aptitude score which designates how well they perform any task; A has apitude score a and B has aptitude ...
0
votes
1answer
290 views

precision recall breakeven point

There are two popular measures that aggregate $Precision$ and $Recall$, there are $F1$ and $\text{Precision Recall Breakeven point}$. $F1$ can be calculated easily by formula, but how to calculate ...
1
vote
1answer
162 views

Multinomial Naive Bayes performance with and without Bigrams/Stopwords

I have a dataset of 300 text documents that I have manually classified into three classes. The first two classes are relatively equal in size; the third class is about 40% larger than each of the ...
4
votes
0answers
332 views

Reduce Random Forest model memory size

I've created a regression model on my data using random forests in R. The output is quite large, I'm wondering if there's any way to reduce this to only the necessary pieces to make a prediction? The ...
2
votes
1answer
165 views

Alternatives to box plots for comparing clustering results

In my research paper, I wish to compare the accuracy of 2 clustering algorithms. I have conducted clustering using these 2 algortihms on 300 proprietary datasets. I use adjusted rand index as the ...