The tag has no wiki summary.

learn more… | top users | synonyms

0
votes
1answer
16 views

Relative speed of gaussian process classification

I used gaussian process classification implemented in Matlab gpml toolbox and also in R kernlab. For my problem - 600*14 matrix with two classes - it trains order of magnitude slower than other ...
3
votes
2answers
62 views

How to compare (probability) predictive ability of models developed from logistic regression?

I know some well-known measures are $c$ statistic, Kolmogorov-Smirnov $D$ statistic. However, as far as I know, those statistics take into account only of the rank order of the observations, and is ...
1
vote
0answers
17 views

Which validation method to choose if the number of instances is limited?

I have a question regarding how to measure the performance of a model when the distribution of instances per class is limited. In my scenario I have five data sets $\mathcal{D}_{1}, \mathcal{D}_{2}, ...
0
votes
2answers
33 views

Leave one out cross validation for neural network perfomance

When using leave one out cross validation in neural network, do I have to fix the epoch number for each training model? The test results of these models are averaged to show performance. So can I ...
3
votes
1answer
51 views

Statistical test to compare method performance over multiple problems

I am comparing several methods of ordering training patterns for on-line neural network training. Let's call those methods I-VII. Then, I have a sample of 20 data sets within a given "data set type" ...
0
votes
0answers
21 views

How to build kpi on employee<>customer<>sales

For further analysis i want to build a KPI which describes the "critically" of a employee<>customer relationship. To make it clear, some example data: Employee - Customer - ...
2
votes
0answers
36 views

Is it possible to explain the performance of a classifier by the specific properties of a data set?

I receive the following comment from a reviewer : "I think that the authors could explain in more details the results. For instance, there is no discussion linking the specific properties of the ...
0
votes
1answer
94 views

Bad Performance of PMML Evaluator

I have been working on a proof-of-concept setup of evaluating various PMML-based models. Specifically, I've been using the JPMML evaluator (https://github.com/jpmml/jpmml-evaluator), which is in Java, ...
0
votes
0answers
28 views

Why can I use the posterior probability of a classifier as a new classifier?

I have read that, when doing discriminant analysis, you can use the posterior probability you obtain using your classifier as a new fine-tuned classifier. Can anyone talk me through the rationale of ...
2
votes
1answer
34 views

Representative transaction KPIs - what method to chose (average, median…) / how?

In my dataset i have about 30k of transactional data. For analysis, I want to show the end user some important KPIs: Average of transactions with loyalty cards Average turnover of these transactions ...
0
votes
0answers
56 views

Difference between regression and performance plot of Artificial neural network in MATLAB

I am having problem understanding regression and performance plots of ANN. My data consists of 13 inputs and 3 outputs. Parameters used for simulation can be found here. The problem I am facing is ...
2
votes
1answer
46 views

So many significant explanatory variables and so small auc

Have you ever seen a model with almost every significant variable and such small auc (area under the ROC curve) ? What might be the cause of it? When I saw summary of a model I thought this model will ...
2
votes
1answer
52 views

How to measure loss of performance of clustering by applying dimensionality reduction

Let's suppose I have a given dataset with $n$ features. Having a data-centric approach, I would like to measure the loss of performance of applying a given dimensionnality reduction technique, for a ...
0
votes
0answers
16 views

Usage of precision and recall vs type I and II errors

What is the advantage of using precision and recall, instead of type 1 and 2 errors? How about disadvantage? The two groups can be derived from each other. But it seems that they are used often in ...
0
votes
0answers
37 views

How use F1 score in an unbalanced binary classification problem?

I have two trained models (MLP and SVM) that want check on unbalanced binary samples (out of sampl - True samples =3000, False samples = 200). I found that i can use ...
1
vote
0answers
12 views

paired subject design test of proportions

I am analyzing a performance data set and would like to compare across two different conditions. I have three subjects, which were tested over a number of trials. Each trial had a chance of being in ...
0
votes
0answers
31 views

Why added predictors improved performance in decision trees if they don't appear in the model?

although I've been working some months now with decision trees, I still have issues understanding some things and also finding a right source to answer my questions. Maybe I'm not using the right ...
1
vote
0answers
16 views

Measure performance of a multiples classifier that is a combination of binary support vector machines

I am comparing the performance of multiple schemes to combine binary SVMs into one multiples classifier. I was hoping to use the cross entropy, however I cannot figure out how to apply it. For ...
10
votes
3answers
1k views

Why isn't the holdout method (splitting data into training and testing) used in classical statistics?

In my classroom exposure to data mining, the holdout method was introduced as a way of assessing model performance. However, when I took my first class on linear models, this was not introduced as a ...
2
votes
0answers
29 views

does the area under the curve (AUC) has some interpretation? [duplicate]

I know that the ROC curve plots true positives vs false positives, but does it have any other interpretation, or is it just an arbitrary performance measure? Also, in the case of very unbalanced ...
0
votes
0answers
44 views

Graph generation model effect on performance of Spectral graph algorithms

I use spectral graph algorithms for finding community structures, specifically the Leading Eigenvector Method (http://arxiv.org/abs/physics/0605087). I try analyzing the performance of these ...
0
votes
1answer
21 views

Ranking performance measure

I have a system able to produce a ranking of some operations according to their anomalous score. I have N anomalous operations inside the dataset, which I expect to be in the first N positions (best ...
0
votes
2answers
330 views

SVM and SMO main differences

I am unable to clearly see the main differences between SVM & SMO. While I get the fact that SMO provides better algorithm for QP solvers but I see that when I use this in Weka on my MacBook it ...
2
votes
2answers
218 views

What are good criteria for performance evaluation of algorithms in a regression problem?

I'm classifying different algorithms on the wine quality dataset. The quality ranges from between 0 - 10 based on 11 other attributes. Here is the data. I'm treating this as a regression problem. ...
0
votes
0answers
112 views

Combining different metrics to calculate composite score

Let's assume I have 3 metrics to evaluate a student. 1)Test Marks (0-50) 2)Weight of the student (KGs) 3)% Attendance (0%-100%) I want to calculate a total composite score which is a function of ...
0
votes
1answer
91 views

Interpreting high P-Value in ordinal rank correlation measures

I'm currently evaluating a machine-learned ranking task. Two (among other) performance measures are Spearman's Rho and Kendall's Tau (rank or concordance correlation). I'm basically comparing a ...
1
vote
0answers
53 views

Slow Lasso Performance Using sklearn

I am trying to use scikit-learn's LassoCV and/or ElasticNetCV functions to model a dataset with a large (>800) number of ...
2
votes
2answers
577 views

Levinsohn-Petrin TFP estimation in Stata without proxy

I'm trying to compute TFP by using levpet Stata command. I use revenue estimation, yet it requires proxy option which is intermediate input (usually material ...
0
votes
0answers
16 views

Benchmarking Software Exercises

I am building a website for practicing programming in various languages. The content is user driven; some users write exercises with test frameworks for validating solutions, and other users write ...
1
vote
1answer
34 views

Evaluating a fixed classifier

I have a classifier that is fixed and wish to evaluate its predictive performance using a test dataset. I'm familiar with the situation (e.g. in k-fold CV) where the data is split and the classifier ...
1
vote
0answers
30 views

Does within-group heterogeneity negatively impact random forest classification?

I have two rather conceptual questions about random forest classifiers. Before we get there, I quickly want to lay out the problem I am working on: I have large a large data set consisting of 300 ...
2
votes
0answers
242 views

Unbalanced dataset - ROC curve to compare classifiers?

I use the machine learning software WEKA for data mining on biological data. I would describe my dataset as unbalanced: It comprises around 2000 instances, ...
1
vote
1answer
56 views

Possible values for sensitivity, specificity, precision and accuracy

I have some results where the tester claims the following values: Sensitivity: 0.525, Specificity: 0.925, Precision: 0.516, Accuracy: 0.907 Where Sensitivity=TP/(TP+FN), Specificity=TN/(TN+FP), ...
0
votes
2answers
250 views

How is the “classification error rate” of an artificial neural network calculated?

Frequently I see artificial neural networks compared by their "classification error rates" or "error rates", particularly for multi-class problems like CIFAR-10. What does this error rate actually ...
2
votes
1answer
111 views

For a model like this what performance measures can I calculate and how?

Methods: From the machine learning literature, I understand different parameters can show performance of model in machine learning. I would briefly expand my understanding with confusion matrix: ...
2
votes
0answers
70 views

How do I measure a speedup?

I have two versions of the same program. I apply a number of performance tests to both and measure their response time, for instance https://gist.github.com/valtih1978/d2cc2fe96fbbe1987ada ...
1
vote
1answer
69 views

How to speed up Kernel density estimation

Background A couple of days ago I asked here about: How to reduce number of points for clustering . Instead of reducing number of points, a method "Kernel Density Estimation" (KDE) was suggested to ...
0
votes
0answers
46 views

Performance metric

Let's say I have two people, A and B, who are each performing a different task. They have an aptitude score which designates how well they perform any task; A has apitude score a and B has aptitude ...
0
votes
1answer
278 views

precision recall breakeven point

There are two popular measures that aggregate $Precision$ and $Recall$, there are $F1$ and $\text{Precision Recall Breakeven point}$. $F1$ can be calculated easily by formula, but how to calculate ...
1
vote
1answer
152 views

Multinomial Naive Bayes performance with and without Bigrams/Stopwords

I have a dataset of 300 text documents that I have manually classified into three classes. The first two classes are relatively equal in size; the third class is about 40% larger than each of the ...
3
votes
0answers
302 views

Reduce Random Forest model memory size

I've created a regression model on my data using random forests in R. The output is quite large, I'm wondering if there's any way to reduce this to only the necessary pieces to make a prediction? The ...
2
votes
1answer
156 views

Alternatives to box plots for comparing clustering results

In my research paper, I wish to compare the accuracy of 2 clustering algorithms. I have conducted clustering using these 2 algortihms on 300 proprietary datasets. I use adjusted rand index as the ...
1
vote
2answers
73 views

How to test the robustness and the performance of a novel classification algorithm

Suppose you have a new algorithm that you want to publish. Are there any best practices and methodologies you usually consider in order to test the robustness and performance of a new method? The ...
1
vote
0answers
23 views

Bootstrapped optimism with low prevalence

My dataset is not quite large enough to be split into training and testing sets, so I am using bootstrapped optimism to account for overfitting when reporting model performance. The cases in my data ...
0
votes
0answers
418 views

How to compare the results of two classifiers are statistically significant different?

I am applying kNN and SVM classifiers to my classification problem. Both of the classifiers get over 95% cross-validation accuracy (leave one out cross validation used). Not sure how to tell if the ...
1
vote
0answers
34 views

What would you use as your default model for comparison?

I have a question. Say you are given with like 1500 instances as your dataset, in which instance will be categorized into 1 of the 3 classes. You're supposed to come up with the best model from the ...
2
votes
3answers
323 views

Evaluating a regression model

For classification problems I have been using Neural Networks and measuring Type I and II error using the confusion matrix and its measures as per this resource which is pretty straight forward. ...
1
vote
2answers
120 views

How to report a SVM model to a 3rd party after cross-validation?

I have a binary classification problem. I trained my dataset using a Support Vector Machine (SVM). Now I want to report the model I trained to a 3rd party so that they can use. For the primal probem ...
4
votes
0answers
427 views

Evaluating a regression model's performance using training and test sets?

I often hear about evaluating a classification model's performance by holding out the test set and training a model on the training set. Then creating 2 vectors, one for the predicted values and one ...
1
vote
0answers
121 views

how to calculate precision, recall, and accuracy, if the label is not binary?

I am new in statistics, so maybe my question kind of stupid. I know how to evaluate the prediction performance if the label is binary, such as the classification label is ...