The tag has no usage guidance.

learn more… | top users | synonyms

0
votes
0answers
12 views

Year on year model performance evaluation via bootstrapping

I am interested in monitoring the yearly evolution of model’s performance (a Loss Given Default model more specifically). The yearly additional data, however, is very limited (maximally 5 ...
0
votes
1answer
21 views

Compute accuracy of model evaluation

Say the number of negativ classes is $9990$ and the number of positive classes is $10$. If a model predicts all examples to belong to the negative class, how accurate is this prediction? So what we ...
0
votes
0answers
18 views

In the classification framework, is AUROC a performance measure or metric?

I guess, the title is self-explaining. I have seen both so far and was wondering if there is a correct term or whether it does not really matter.
0
votes
0answers
15 views

Matched pairs versus multiple t-test measurements for task performance over time

I am trying to compare performance on a task over time across one group based on initial performance, but am running into a little trouble. As background, I am examining a group of students' ...
3
votes
2answers
30 views

Creating a rank on performance and size

Ranking is not something that I need to do particularly frequently in my job and its probably the weakest part of my statistics background. I have a dataset containing a list of items and a ...
0
votes
1answer
77 views

How to compare classification methods in terms of performance?

I'd like to compare logistic regression to classification trees. In a first step, I compared the theoretical framework of the two classifiers. In a second step, I compared the performance using a ...
1
vote
0answers
39 views

Why MSE of this MATLAB neural network is so high?

I'm using this code to train a neural netowrk in MATLAB: ...
1
vote
1answer
62 views

Is it valid to get better performance in logistic regression using only a subset of the coefficients?

I have an imbalanced data set containing 12% of the positive class 88% negative. First, I ran a logistic regression with all my coefficients and got an average accuracy of 0.91 (I know that's not ...
0
votes
0answers
11 views

MLP and RBF performance assessment (Matlab)

I hope you really help me. I'm now approaching to the world of data mining and pattern recognition and I need a little help. I have a data set composed by 4 struct of features. Each struct contains 18 ...
1
vote
1answer
22 views

How to account for games and number of teams in share of points?

I am doing research on the relationship of income inequality on team performance in soccer. The team performance I measure through share of points, so the number of points a team got in a particular ...
0
votes
1answer
12 views

Performance of estimators

I have 11 different estimators and try to figure out which of them performs best. The regression line between given and estimated values ideally should have a slope of 1 and an intersect at 0,0; which ...
0
votes
0answers
36 views

Model stability- ratios in same bucket should keep constant over different population?

I need to explain the variation in a validation table for a propensity score to buy insurance. I trained in R a logistic model with 4 months of 2014. Calculated limits of non-overlapping deciles of ...
6
votes
2answers
96 views

How can I assess performance of a semi-supervised learning method?

I'm working with a semi-supervised learning task, where I only have positive and unlabelled data (PU learning). I've tested a few algorithms and would like to assess their performance. For ...
0
votes
1answer
20 views

Best performance evaluation measure for GARCH model?

I want to compare the performance of various volatility models like GARCH, eGARCH, and gjrGARCH from actual volatility (computed using high frequency data). I found three common performance evaluation ...
0
votes
1answer
30 views

How can I compare two mutual funds' performance with a sparse set of data?

I want to compare the performance of two mutual funds. The only data I have is annual returns for the past 7 years. So I have 7 observations for Fund 1 and 7 observations for Fund 2. In addition, I ...
0
votes
1answer
65 views

Calculate thresholds of factor analysis output to classify data to 5 classes

Suppose that we calculate a composite indicator for some companies using Factor Analysis (FA) by combining five features to one output (calculate weights of input features). This is histogram of ...
0
votes
0answers
11 views

How does one show that an extension of an algorithm performs as well or better than the original?

If we have an extension of an algorithm (for example, we take OLS and scale the inputs, and solve it as a reformulated OLS problem), how does one show that the extension performs as well or better ...
0
votes
0answers
12 views

bootstrap simulation for CI

I have generate 4 functions to test the performance of my bootstrap for obtaining 95% confidence intervals on the mean of a univariate sample of data. However, it does not run properly with the last ...
0
votes
0answers
16 views

Performance Evaluation of Search Engines Test

I have to apply double-blind testing method to evaluate the performance of any two search engines.(ex. Google vs. Bing) I'm drawing a blank here. Anyone have any ideas of how I can start?
0
votes
1answer
30 views

Reporting result of classifiers [duplicate]

I am using a binary classifier in my research. I wanna report the obtained result of classifier, but I am not sure about it. The learning prosedure is as follows: ...
0
votes
0answers
20 views

Does this type of diagram showing conditional mean have a name?

I'm trying to show the skill of a prediction $\hat{Y}$ relative to a true value $Y$ beyond simple statistics like MSE or $r^2$. I usually like looking at the scatter plot of $\hat{Y}$ vs $Y$ but in ...
0
votes
0answers
52 views

GBM Performance on different sampling techniques

I am working on a healthcare data set for breast cancer patients. This data set is class imbalances and the distribution of positive and negative classes is 80%/20%. In order to deal with the class ...
0
votes
0answers
24 views

Uncertainty assignment to genetic algorithm solution

I have a black-box function for which I need to find its global maximum. I do this via a genetic algorithm (GA) since I can't solve it analytically. Since the GA can't guarantee to return the true ...
0
votes
1answer
66 views

Which is better, MSE or classification rate, for minimizing the likelihood of overfitting?

During the process of selecting a classification model (with cross validation), if one uses MSE of the predicted posterior probabilities as a selection criterion, then a model ...
1
vote
0answers
24 views

Standard performance measure for regression?

I will perform time-series prediction and I will report the accuracy of my system with a measure like RMSE or MAE. However, the variables I will predict are in different ranges. So let's say one is ...
0
votes
0answers
13 views

Term for results being similar in relation to each other, but differing is absolute magnitude

I have some results of several methods, applied to two tasks. Roughly they are on the lines of Task1 Method A: 20% Method B: 25% Method C: 40% Method D: 70% Method E: 71% Task2 Method A: 40% ...
0
votes
0answers
10 views

Performance metrics, prioritizing based on impact

I apologize in advance for the poorly worded question. If anyone has any suggestions, please advise. Also, help with tagging appropriately would be appreciated. I'm collecting performance metrics ...
1
vote
2answers
61 views

Visually summarising multiple diverse metric results

I have a multivariate time series (climate fluxes), and I have a number of model outputs that I want to asses. I also have number of metrics that I can use to assess the performance of each variable ...
0
votes
0answers
22 views

Performance measure

I am currently trying to show the trend of different team performances based on what percent of customer complaints they receive that they are able to handle within a given month. So if one team ...
2
votes
1answer
96 views

CART with Ordinal Response Variable using rpartScore Stuck

I'm trying to fit a decision tree over some data which has ~40K rows and ~200 features. The response variable, y, is ordinal and takes values ...
2
votes
1answer
47 views

What should the AUROC be on the test set when no positive example is present?

Assume we have a probabilistic, binary classifier. We compute the AUROC on a test set in which no positive example is present (i.e. the ground truth is always 0). What should the AUROC be?
0
votes
0answers
31 views

suitable flexibility of statistical learning method and number of predictors and number of samples

Out of a flexible and an inflexible statistical learning method, which one is expected to perform better when the number of samples is extremely large and the number of predictors is small? the ...
4
votes
1answer
104 views

What's the reasoning behind presenting unvalidated AUC as a measure of model fit or performance?

Often one sees, particularly in the biomedical literature, papers that analyze the performance of a risk prediction model in terms of the AUC or the area under the ROC curve. If the AUC is suitably ...
1
vote
0answers
20 views

Taking longer sample should not reduce the variance of the mean

I have not realized completely why n times longer samples should give you n times shorter variance. However, I measure performance of my programs and there are some fluctuation due to PC state ...
0
votes
1answer
32 views

Relative speed of gaussian process classification

I used gaussian process classification implemented in Matlab gpml toolbox and also in R kernlab. For my problem - 600*14 matrix with two classes - it trains order of magnitude slower than other ...
5
votes
2answers
251 views

How to compare (probability) predictive ability of models developed from logistic regression?

I know some well-known measures are $c$ statistic, Kolmogorov-Smirnov $D$ statistic. However, as far as I know, those statistics take into account only of the rank order of the observations, and is ...
1
vote
0answers
24 views

Which validation method to choose if the number of instances is limited?

I have a question regarding how to measure the performance of a model when the distribution of instances per class is limited. In my scenario I have five data sets $\mathcal{D}_{1}, \mathcal{D}_{2}, ...
0
votes
2answers
123 views

Leave one out cross validation for neural network perfomance

When using leave one out cross validation in neural network, do I have to fix the epoch number for each training model? The test results of these models are averaged to show performance. So can I ...
3
votes
1answer
196 views

Statistical test to compare method performance over multiple problems

I am comparing several methods of ordering training patterns for on-line neural network training. Let's call those methods I-VII. Then, I have a sample of 20 data sets within a given "data set type" ...
0
votes
0answers
43 views

How to build kpi on employee<>customer<>sales

For further analysis i want to build a KPI which describes the "critically" of a employee<>customer relationship. To make it clear, some example data: Employee - Customer - ...
2
votes
0answers
45 views

Is it possible to explain the performance of a classifier by the specific properties of a data set?

I receive the following comment from a reviewer : "I think that the authors could explain in more details the results. For instance, there is no discussion linking the specific properties of the ...
0
votes
1answer
229 views

Bad Performance of PMML Evaluator

I have been working on a proof-of-concept setup of evaluating various PMML-based models. Specifically, I've been using the JPMML evaluator (https://github.com/jpmml/jpmml-evaluator), which is in Java, ...
0
votes
0answers
35 views

Why can I use the posterior probability of a classifier as a new classifier?

I have read that, when doing discriminant analysis, you can use the posterior probability you obtain using your classifier as a new fine-tuned classifier. Can anyone talk me through the rationale of ...
2
votes
1answer
63 views

Representative transaction KPIs - what method to chose (average, median…) / how?

In my dataset i have about 30k of transactional data. For analysis, I want to show the end user some important KPIs: Average of transactions with loyalty cards Average turnover of these transactions ...
0
votes
0answers
101 views

Difference between regression and performance plot of Artificial neural network in MATLAB

I am having problem understanding regression and performance plots of ANN. My data consists of 13 inputs and 3 outputs. Parameters used for simulation can be found here. The problem I am facing is ...
2
votes
1answer
59 views

So many significant explanatory variables and so small auc

Have you ever seen a model with almost every significant variable and such small auc (area under the ROC curve) ? What might be the cause of it? When I saw summary of a model I thought this model will ...
2
votes
1answer
62 views

How to measure loss of performance of clustering by applying dimensionality reduction

Let's suppose I have a given dataset with $n$ features. Having a data-centric approach, I would like to measure the loss of performance of applying a given dimensionnality reduction technique, for a ...
0
votes
0answers
21 views

Usage of precision and recall vs type I and II errors

What is the advantage of using precision and recall, instead of type 1 and 2 errors? How about disadvantage? The two groups can be derived from each other. But it seems that they are used often in ...
0
votes
0answers
48 views

How use F1 score in an unbalanced binary classification problem?

I have two trained models (MLP and SVM) that want check on unbalanced binary samples (out of sampl - True samples =3000, False samples = 200). I found that i can use ...
1
vote
0answers
19 views

paired subject design test of proportions

I am analyzing a performance data set and would like to compare across two different conditions. I have three subjects, which were tested over a number of trials. Each trial had a chance of being in ...