The performance tag has no wiki summary.
2
votes
1answer
75 views
Which performance measure for unbalanced binary classification without an 'active' class?
My datasets have two classes A and B. The classes should be treated equally (there is no "active/inactive"). The datasets are unbalanced, sometimes A is more frequent, sometimes B is more frequent. ...
1
vote
0answers
19 views
Performance worse with new observations
I come from the computer science area but am new to machine learning / stats, so this question may be fundamental and easy.
I have time-series data (biological data), and, without getting into the ...
3
votes
2answers
45 views
How to assess the improvement from a treatment?
When optimizing a computer program, I need to determine how much faster the program runs with the latest optimization. To do this, I run the program N times without the optimization and N times with ...
0
votes
0answers
6 views
What is a contrastive run?
I keep hearing about contrastive runs for evaluating the statistical performance of a system. I could not find an entry on Wikipedia on the topic and Google returns something but not much.
0
votes
0answers
71 views
Cross validation statistics
I have a question about the generation of statistics from cross validation schemes for performance estimation (two classes classification in my case).
Before each experiment (K iterations) I randomly ...
0
votes
1answer
52 views
Units for silhouette measure
While drawing graphs that compare the silhouette measure of different clustering algorithms, what unit should I specify for the silhouette width?
2
votes
1answer
44 views
Rates affecting rates
Stats noob here. I've been trying to find out if there is a technique to perform a certain type of analysis but so far a lot of searching hasn't yielded anything.
My question is, if a large peak or ...
0
votes
0answers
19 views
What is the multiclass loss used in kaggles for measure performance?
I searched on Internet but cannot find any source about this. What is multiclass loss and what kind of metric is it? If someone can enlighten me, it will be appreciated.
1
vote
3answers
99 views
Can unsupervised evaluation measures for clustering replace a supervised evaluation measure?
Is it possible to have the same evaluation performances when comparing some clustering algorithms using many unsupervised evaluation measures instead of a supervised one ?
0
votes
1answer
107 views
Estimating maximum predictive power in noisy data
When assessing model performance, one would like to separate prediction errors due to limitations of the model from those errors due to intrinsic noise. For example, in noisy data, AUC in an ROC ...
2
votes
0answers
109 views
Which probabilites are to be supplied to rcorr.cens and improveProb in package Hmisc?
Both functions are great for comparing, for example, survival models. The first especially for computing Harrell's C-index, the second for NRI and IDI.
However, it looks like ...
3
votes
1answer
141 views
Predictive models with large numbers of missing values in the features
I have been trying to train an algorithm to predict if an account will close or not using thousands of data points and many features.
I am using data from the month before the account closed but ...
0
votes
0answers
95 views
Measuring responsiveness and memory usage of a web application
We have a in house web-app that we tested for Responsiveness and memory usage. Reason was two fold. One its never been bench-marked and second was that the app itself in its earlier version was slow.
...
3
votes
1answer
156 views
False discovery rate of a Bayesian classifier: scaling based on prior odds?
I am trying to assess the performance of my Bayesian classifier. One measure that I calculate is the false discovery rate (FDR):
FP / (FP + TP),
where FP = False Positive and TP = True Positive.
...
1
vote
1answer
153 views
How can I know a random small subsample has a good performance?
Sorry, if the title is confusing. not native speaker here.
I ran a survey study which had 15 7-point likert scale questions. Each question is ask people to judge how creativity an object in the ...
1
vote
0answers
117 views
How to compare the accuracy of two different models using statistical significance
I am working on time series prediction. I have two data sets $D1=\{x_1, x_2,....x_n\}$ and $D2=\{x_n+1, x_n+2, x_n+3,...., x_n+k\}$. I have three prediction models: $M1, M2, M3$. All of those model ...
2
votes
1answer
606 views
Is my interpretation of Neural Network results correct?
I use a neural network with a topology of 17-30-1 (sigmoid, atan error function, mse as cost function, 5-fold cv) for text classification. (It's closely related to a previous question of mine.)
The ...
3
votes
1answer
146 views
Name of mean absolute error analogue to Brier score?
Yesterday's question Determine accuracy of model which estimates probability of event got me curious about probability scoring.
The Brier score
$$\frac{1}{N}\sum\limits _{i=1}^{N}(prediction_i - ...
9
votes
2answers
352 views
Incremental learning for LOESS time series model
I am currently working on some time series data, I know I can use
LOESS/ARIMA model.
The data is written to a vector whose length is 1000, which is a queue,
updating every 15 minutes,
Thus the ...
