The tag has no wiki summary.

learn more… | top users | synonyms

1
vote
0answers
8 views

Does within-group heterogeneity negatively impact random forest classification?

I have two rather conceptual questions about random forest classifiers. Before we get there, I quickly want to lay out the problem I am working on: I have large a large data set consisting of 300 ...
2
votes
0answers
44 views

Unbalanced dataset - ROC curve to compare classifiers?

I use the machine learning software WEKA for data mining on biological data. I would describe my dataset as unbalanced: It comprises around 2000 instances, ...
1
vote
1answer
26 views

Possible values for sensitivity, specificity, precision and accuracy

I have some results where the tester claims the following values: Sensitivity: 0.525, Specificity: 0.925, Precision: 0.516, Accuracy: 0.907 Where Sensitivity=TP/(TP+FN), Specificity=TN/(TN+FP), ...
0
votes
2answers
24 views

How is the “classification error rate” of an artificial neural network calculated?

Frequently I see artificial neural networks compared by their "classification error rates" or "error rates", particularly for multi-class problems like CIFAR-10. What does this error rate actually ...
2
votes
1answer
61 views

For a model like this what performance measures can I calculate and how?

Methods: From the machine learning literature, I understand different parameters can show performance of model in machine learning. I would briefly expand my understanding with confusion matrix: ...
0
votes
0answers
16 views

A framework for comparing the performance of Expectation Maximization

I have my own implementation of the Expectation Maximization (EM) algorithm based on the following paper http://pdf.aminer.org/000/221/588/fuzzy_k_means_clustering_with_crisp_regions.pdf I would like ...
1
vote
0answers
29 views

How do I measure a speedup?

I have two versions of the same program. I apply a number of performance tests to both and measure their response time, for instance https://gist.github.com/valtih1978/d2cc2fe96fbbe1987ada ...
1
vote
1answer
29 views

How to speed up Kernel density estimation

Background A couple of days ago I asked here about: How to reduce number of points for clustering . Instead of reducing number of points, a method "Kernel Density Estimation" (KDE) was suggested to ...
0
votes
0answers
43 views

Performance of hierarchical clustering for binary data in R

I am trying to use Hierarchical clustering to see how well it performs in classifying a dataset which I previously know its true classification. I am new to clustering in general. I was able to draw ...
0
votes
0answers
32 views

Performance metric

Let's say I have two people, A and B, who are each performing a different task. They have an aptitude score which designates how well they perform any task; A has apitude score a and B has aptitude ...
0
votes
1answer
27 views

precision recall breakeven point

There are two popular measures that aggregate $Precision$ and $Recall$, there are $F1$ and $\text{Precision Recall Breakeven point}$. $F1$ can be calculated easily by formula, but how to calculate ...
1
vote
1answer
30 views

Multinomial Naive Bayes performance with and without Bigrams/Stopwords

I have a dataset of 300 text documents that I have manually classified into three classes. The first two classes are relatively equal in size; the third class is about 40% larger than each of the ...
0
votes
0answers
20 views

Best way to report “average speedup”?

I have a collection of benchmarks that we run against different implementations of a system. Is there a generally-accepted way to make comparisons between multiple sets of benchmark data? Today, we ...
2
votes
1answer
80 views

Alternatives to box plots for comparing clustering results

In my research paper, I wish to compare the accuracy of 2 clustering algorithms. I have conducted clustering using these 2 algortihms on 300 proprietary datasets. I use adjusted rand index as the ...
1
vote
2answers
55 views

How to test the robustness and the performance of a novel classification algorithm

Suppose you have a new algorithm that you want to publish. Are there any best practices and methodologies you usually consider in order to test the robustness and performance of a new method? The ...
1
vote
0answers
18 views

Bootstrapped optimism with low prevalence

My dataset is not quite large enough to be split into training and testing sets, so I am using bootstrapped optimism to account for overfitting when reporting model performance. The cases in my data ...
0
votes
0answers
62 views

How to compare the results of two classifiers are statistically significant different?

I am applying kNN and SVM classifiers to my classification problem. Both of the classifiers get over 95% cross-validation accuracy (leave one out cross validation used). Not sure how to tell if the ...
0
votes
0answers
31 views

Difference between RMSE and Spearman Correlation

I am trying to evaluate model performance (regression problem). In literature, some use RMSe and others use correlation. Is there any difference between both the approaches? Here: What are good ...
1
vote
0answers
30 views

What would you use as your default model for comparison?

I have a question. Say you are given with like 1500 instances as your dataset, in which instance will be categorized into 1 of the 3 classes. You're supposed to come up with the best model from the ...
1
vote
3answers
196 views

Evaluating a regression model

For classification problems I have been using Neural Networks and measuring Type I and II error using the confusion matrix and its measures as per this resource which is pretty straight forward. ...
1
vote
2answers
92 views

How to report a SVM model to a 3rd party after cross-validation?

I have a binary classification problem. I trained my dataset using a Support Vector Machine (SVM). Now I want to report the model I trained to a 3rd party so that they can use. For the primal probem ...
3
votes
0answers
152 views

Evaluating a regression model's performance using training and test sets?

I often hear about evaluating a classification model's performance by holding out the test set and training a model on the training set. Then creating 2 vectors, one for the predicted values and one ...
0
votes
0answers
24 views

Measuring the performance of a classifier that estimates bin counts for a Poisson process

I have some data that results from the convolution of a Poisson random process with a decaying exponential, plus some additive Gaussian noise. I'm working on designing a classifier that returns a ...
1
vote
0answers
62 views

how to calculate precision, recall, and accuracy, if the label is not binary?

I am new in statistics, so maybe my question kind of stupid. I know how to evaluate the prediction performance if the label is binary, such as the classification label is ...
1
vote
1answer
1k views

Interpretation of a WEKA result buffer - confusion matrix and performance

I want to know how to get several performance measurements of a generated WEKA model. Note that I am predicting a two-class variable, Alive or ...
0
votes
1answer
45 views

Performance metric for categorical outcome prediction

I have 3 independent subjects each of whom have one attempt at guessing the number of balls in a hidden tray with 20 compartments. Each subject therefore has to produce 20 guesses, one for each ...
2
votes
0answers
50 views

Performance of regression models

If we have to choose between two regression models, one has a correlation coefficient of 0.95 and the other has a correlation coefficient of 0.75. Is it always the case that the first model is to be ...
1
vote
1answer
430 views

What are the metrics to evaluate a machine learning algorithm

I would like to know what are the various techniques and metrics used to evaluate how accurate/good an algorithm is and how to use a given metric to derive a conclusion about a ML model.
0
votes
1answer
122 views

How can I compare variables with uneven data sets?

I'm doing a stationary sport in which I use different colored equipment for different weather. I am recording my scores along with the color of the equipment and the weather in which I am using it. I ...
0
votes
0answers
23 views

What are the most accurate athletic performance indicators based on scores?

I want to generate statistics for a sport I'm doing. I am collecting data such as my scores and instances in which I miss. Based on this, I calculated averages for my overall performance and for when ...
1
vote
0answers
27 views

Tuning a model for predictive performance in a narrow probability range

Assume we model the probability of disease incidence. When an individual's predicted probability of incident disease, or absolute risk, is greater than a certain threshold, we start preventive ...
0
votes
0answers
29 views

Evaluation of dynamic model

This is kind of a beginner level question. I want to know how to assess the performance of a model I have created. Suppose that we have an identification system aboard a spaceship. The job of the ...
1
vote
0answers
77 views

BIC vs. Out of sample performance

I have two statistical models. Model 1 uses a GLM approach while model 2 uses a time series approach for fitting. I want to compare these two models. Model 1 (i.e. GLM) has a better out of sample ...
0
votes
0answers
21 views

Average of performance estimates

When I learn several classifiers for a task and want to compare their performance on it, I thought about calculating several performance measures, such as: sensitivity specificity positive and ...
7
votes
1answer
171 views

Evaluating Time Series Prediction Performance

I have a Dynamic Naive Bayes Model trained on a couple of temporal variables. The output of the model is the prediction of P(Event) @ t+1, estimated at each ...
1
vote
0answers
378 views

poor prediction performance from (WEKA) classifiers

I have an imbalanced 2-class data set made of two parts - model creation data and prediction data. Model creation data is split into train and test data. Model is trained using the training data and ...
3
votes
2answers
106 views

Comparing employee sales monthly data despite regional differences

I'm not exactly sure how to title this thread, or what tags to use since advanced statistics isn't really my thing right now. Please change the title and tags as needed to improve this question. Say ...
0
votes
0answers
21 views

How to compare performance of probability estimating model [duplicate]

Suppose I have a model that estimates a discrete probability distribution over a set of classes/factors c1, c2, c3. What are the options and appropriate use cases ...
4
votes
5answers
167 views

How to measure a classifier's performance when close to 100% of the class labels belong to one class?

In my data, I have a class variable, denoted as $C$. This class variable values are ${0, 1}$ (binary). Almost all observations of $C$ are 0 (close to 100%, more precisely, 97%). I would like a ...
2
votes
2answers
46 views

How to assess the effect of optimisation on software performance improvement based on five trials with and without optimisation?

[This question is similar to How to assess the improvement from a treatment?] I'm attempting to characterize what appears to be an obvious improvement in performance as a result of some changes to a ...
1
vote
2answers
138 views

Effective validity of AUROC as performance measure: what about “very high” AUROC values?

The Area Under ROC curve (AUROC) is a quantity used to quantify performance of classifiers. I am currently interested in the most basic drawbacks of using AUROC as unique performance measure. I got ...
1
vote
0answers
127 views

Fast/parallel alternative to GLS with nlme?

I am using the gls function from nlme to fit a fixed-effects model yet correct for spatial autocorrelation. My dataset has about 100,000 unique geographic observations, and running the following ...
6
votes
1answer
221 views

Benchmark data for Random Forest evaluation

I would like to make/explore several alternative formulations of random forest (link) and am looking for the current state of the science for evaluating performance. In the reference file Leo ...
3
votes
1answer
822 views

Which performance measure for unbalanced binary classification without an 'active' class?

My datasets have two classes A and B. The classes should be treated equally (there is no "active/inactive"). The datasets are unbalanced, sometimes A is more frequent, sometimes B is more frequent. ...
1
vote
0answers
34 views

Performance worse with new observations

I come from the computer science area but am new to machine learning / stats, so this question may be fundamental and easy. I have time-series data (biological data), and, without getting into the ...
3
votes
2answers
56 views

How to assess the improvement from a treatment?

When optimizing a computer program, I need to determine how much faster the program runs with the latest optimization. To do this, I run the program N times without the optimization and N times with ...
0
votes
0answers
125 views

Cross validation statistics

I have a question about the generation of statistics from cross validation schemes for performance estimation (two classes classification in my case). Before each experiment (K iterations) I randomly ...
0
votes
1answer
88 views

Units for silhouette measure

While drawing graphs that compare the silhouette measure of different clustering algorithms, what unit should I specify for the silhouette width?
2
votes
1answer
52 views

Rates affecting rates

Stats noob here. I've been trying to find out if there is a technique to perform a certain type of analysis but so far a lot of searching hasn't yielded anything. My question is, if a large peak or ...
0
votes
0answers
26 views

What is the multiclass loss used in kaggles for measure performance?

I searched on Internet but cannot find any source about this. What is multiclass loss and what kind of metric is it? If someone can enlighten me, it will be appreciated.