The tag has no wiki summary.

learn more… | top users | synonyms

0
votes
0answers
21 views

How to compare the results of two classifiers are statistically significant different?

I am applying kNN and SVM classifiers to my classification problem. Both of the classifiers get over 95% cross-validation accuracy (leave one out cross validation used). Not sure how to tell if the ...
0
votes
0answers
20 views

Difference between RMSE and Spearman Correlation

I am trying to evaluate model performance (regression problem). In literature, some use RMSe and others use correlation. Is there any difference between both the approaches? Here: What are good ...
0
votes
0answers
5 views

Problems with speeding up loop in R [migrated]

I have a particularly big dataset which consists of 3.7 mio rows and 76 string columns. I want to compare the above row with the below row in terms of whether they match and have written this code. ...
0
votes
0answers
27 views

What would you use as your default model for comparison?

I have a question. Say you are given with like 1500 instances as your dataset, in which instance will be categorized into 1 of the 3 classes. You're supposed to come up with the best model from the ...
1
vote
3answers
190 views

Evaluating a regression model

For classification problems I have been using Neural Networks and measuring Type I and II error using the confusion matrix and its measures as per this resource which is pretty straight forward. ...
1
vote
2answers
78 views

How to report a SVM model to a 3rd party after cross-validation?

I have a binary classification problem. I trained my dataset using a Support Vector Machine (SVM). Now I want to report the model I trained to a 3rd party so that they can use. For the primal probem ...
3
votes
0answers
61 views

Evaluating a regression model's performance using training and test sets?

I often hear about evaluating a classification model's performance by holding out the test set and training a model on the training set. Then creating 2 vectors, one for the predicted values and one ...
0
votes
0answers
18 views

Measuring the performance of a classifier that estimates bin counts for a Poisson process

I have some data that results from the convolution of a Poisson random process with a decaying exponential, plus some additive Gaussian noise. I'm working on designing a classifier that returns a ...
1
vote
0answers
31 views

how to calculate precision, recall, and accuracy, if the label is not binary?

I am new in statistics, so maybe my question kind of stupid. I know how to evaluate the prediction performance if the label is binary, such as the classification label is ...
1
vote
1answer
303 views

Interpretation of a WEKA result buffer - confusion matrix and performance

I want to know how to get several performance measurements of a generated WEKA model. Note that I am predicting a two-class variable, Alive or ...
0
votes
1answer
40 views

Performance metric for categorical outcome prediction

I have 3 independent subjects each of whom have one attempt at guessing the number of balls in a hidden tray with 20 compartments. Each subject therefore has to produce 20 guesses, one for each ...
2
votes
0answers
46 views

Performance of regression models

If we have to choose between two regression models, one has a correlation coefficient of 0.95 and the other has a correlation coefficient of 0.75. Is it always the case that the first model is to be ...
1
vote
1answer
163 views

What are the metrics to evaluate a machine learning algorithm

I would like to know what are the various techniques and metrics used to evaluate how accurate/good an algorithm is and how to use a given metric to derive a conclusion about a ML model.
0
votes
1answer
61 views

How can I compare variables with uneven data sets?

I'm doing a stationary sport in which I use different colored equipment for different weather. I am recording my scores along with the color of the equipment and the weather in which I am using it. I ...
0
votes
0answers
15 views

What are the most accurate athletic performance indicators based on scores?

I want to generate statistics for a sport I'm doing. I am collecting data such as my scores and instances in which I miss. Based on this, I calculated averages for my overall performance and for when ...
1
vote
0answers
27 views

Tuning a model for predictive performance in a narrow probability range

Assume we model the probability of disease incidence. When an individual's predicted probability of incident disease, or absolute risk, is greater than a certain threshold, we start preventive ...
0
votes
0answers
27 views

Evaluation of dynamic model

This is kind of a beginner level question. I want to know how to assess the performance of a model I have created. Suppose that we have an identification system aboard a spaceship. The job of the ...
1
vote
0answers
59 views

BIC vs. Out of sample performance

I have two statistical models. Model 1 uses a GLM approach while model 2 uses a time series approach for fitting. I want to compare these two models. Model 1 (i.e. GLM) has a better out of sample ...
0
votes
0answers
18 views

Average of performance estimates

When I learn several classifiers for a task and want to compare their performance on it, I thought about calculating several performance measures, such as: sensitivity specificity positive and ...
6
votes
0answers
127 views

Evaluating Time Series Prediction Performance

I have a Dynamic Naive Bayes Model trained on a couple of temporal variables. The output of the model is the prediction of P(Event) @ t+1, estimated at each ...
1
vote
0answers
257 views

poor prediction performance from (WEKA) classifiers

I have an imbalanced 2-class data set made of two parts - model creation data and prediction data. Model creation data is split into train and test data. Model is trained using the training data and ...
3
votes
2answers
92 views

Comparing employee sales monthly data despite regional differences

I'm not exactly sure how to title this thread, or what tags to use since advanced statistics isn't really my thing right now. Please change the title and tags as needed to improve this question. Say ...
0
votes
0answers
20 views

How to compare performance of probability estimating model [duplicate]

Suppose I have a model that estimates a discrete probability distribution over a set of classes/factors c1, c2, c3. What are the options and appropriate use cases ...
4
votes
5answers
136 views

How to measure a classifier's performance when close to 100% of the class labels belong to one class?

In my data, I have a class variable, denoted as $C$. This class variable values are ${0, 1}$ (binary). Almost all observations of $C$ are 0 (close to 100%, more precisely, 97%). I would like a ...
2
votes
2answers
43 views

How to assess the effect of optimisation on software performance improvement based on five trials with and without optimisation?

[This question is similar to How to assess the improvement from a treatment?] I'm attempting to characterize what appears to be an obvious improvement in performance as a result of some changes to a ...
1
vote
2answers
117 views

Effective validity of AUROC as performance measure: what about “very high” AUROC values?

The Area Under ROC curve (AUROC) is a quantity used to quantify performance of classifiers. I am currently interested in the most basic drawbacks of using AUROC as unique performance measure. I got ...
1
vote
0answers
103 views

Fast/parallel alternative to GLS with nlme?

I am using the gls function from nlme to fit a fixed-effects model yet correct for spatial autocorrelation. My dataset has about 100,000 unique geographic observations, and running the following ...
6
votes
1answer
175 views

Benchmark data for Random Forest evaluation

I would like to make/explore several alternative formulations of random forest (link) and am looking for the current state of the science for evaluating performance. In the reference file Leo ...
3
votes
1answer
489 views

Which performance measure for unbalanced binary classification without an 'active' class?

My datasets have two classes A and B. The classes should be treated equally (there is no "active/inactive"). The datasets are unbalanced, sometimes A is more frequent, sometimes B is more frequent. ...
1
vote
0answers
29 views

Performance worse with new observations

I come from the computer science area but am new to machine learning / stats, so this question may be fundamental and easy. I have time-series data (biological data), and, without getting into the ...
3
votes
2answers
56 views

How to assess the improvement from a treatment?

When optimizing a computer program, I need to determine how much faster the program runs with the latest optimization. To do this, I run the program N times without the optimization and N times with ...
0
votes
0answers
118 views

Cross validation statistics

I have a question about the generation of statistics from cross validation schemes for performance estimation (two classes classification in my case). Before each experiment (K iterations) I randomly ...
0
votes
1answer
85 views

Units for silhouette measure

While drawing graphs that compare the silhouette measure of different clustering algorithms, what unit should I specify for the silhouette width?
2
votes
1answer
51 views

Rates affecting rates

Stats noob here. I've been trying to find out if there is a technique to perform a certain type of analysis but so far a lot of searching hasn't yielded anything. My question is, if a large peak or ...
0
votes
0answers
25 views

What is the multiclass loss used in kaggles for measure performance?

I searched on Internet but cannot find any source about this. What is multiclass loss and what kind of metric is it? If someone can enlighten me, it will be appreciated.
1
vote
3answers
178 views

Can unsupervised evaluation measures for clustering replace a supervised evaluation measure?

Is it possible to have the same evaluation performances when comparing some clustering algorithms using many unsupervised evaluation measures instead of a supervised one ?
0
votes
1answer
118 views

Estimating maximum predictive power in noisy data

When assessing model performance, one would like to separate prediction errors due to limitations of the model from those errors due to intrinsic noise. For example, in noisy data, AUC in an ROC ...
2
votes
0answers
178 views

Which probabilites are to be supplied to rcorr.cens and improveProb in package Hmisc?

Both functions are great for comparing, for example, survival models. The first especially for computing Harrell's C-index, the second for NRI and IDI. However, it looks like ...
4
votes
1answer
217 views

Predictive models with large numbers of missing values in the features

I have been trying to train an algorithm to predict if an account will close or not using thousands of data points and many features. I am using data from the month before the account closed but ...
3
votes
1answer
183 views

False discovery rate of a Bayesian classifier: scaling based on prior odds?

I am trying to assess the performance of my Bayesian classifier. One measure that I calculate is the false discovery rate (FDR): FP / (FP + TP), where FP = False Positive and TP = True Positive. ...
1
vote
1answer
170 views

How can I know a random small subsample has a good performance?

Sorry, if the title is confusing. not native speaker here. I ran a survey study which had 15 7-point likert scale questions. Each question is ask people to judge how creativity an object in the ...
1
vote
0answers
164 views

How to compare the accuracy of two different models using statistical significance

I am working on time series prediction. I have two data sets $D1=\{x_1, x_2,....x_n\}$ and $D2=\{x_n+1, x_n+2, x_n+3,...., x_n+k\}$. I have three prediction models: $M1, M2, M3$. All of those model ...
2
votes
1answer
928 views

Is my interpretation of Neural Network results correct?

I use a neural network with a topology of 17-30-1 (sigmoid, atan error function, mse as cost function, 5-fold cv) for text classification. (It's closely related to a previous question of mine.) The ...
4
votes
1answer
219 views

Name of mean absolute error analogue to Brier score?

Yesterday's question Determine accuracy of model which estimates probability of event got me curious about probability scoring. The Brier score $$\frac{1}{N}\sum\limits _{i=1}^{N}(prediction_i - ...
9
votes
2answers
377 views

Incremental learning for LOESS time series model

I am currently working on some time series data, I know I can use LOESS/ARIMA model. The data is written to a vector whose length is 1000, which is a queue, updating every 15 minutes, Thus the ...