# Tagged Questions

The tag has no usage guidance.

14 views

### How can I compare two mutual funds' performance with a sparse set of data?

I want to compare the performance of two mutual funds. The only data I have is annual returns for the past 7 years. So I have 7 observations for Fund 1 and 7 observations for Fund 2. In addition, I ...
52 views

### Calculate thresholds of factor analysis output to classify data to 5 classes

Suppose that we calculate a composite indicator for some companies using Factor Analysis (FA) by combining five features to one output (calculate weights of input features). This is histogram of ...
10 views

### How does one show that an extension of an algorithm performs as well or better than the original?

If we have an extension of an algorithm (for example, we take OLS and scale the inputs, and solve it as a reformulated OLS problem), how does one show that the extension performs as well or better ...
10 views

### bootstrap simulation for CI

I have generate 4 functions to test the performance of my bootstrap for obtaining 95% confidence intervals on the mean of a univariate sample of data. However, it does not run properly with the last ...
11 views

### Performance Evaluation of Search Engines Test

I have to apply double-blind testing method to evaluate the performance of any two search engines.(ex. Google vs. Bing) I'm drawing a blank here. Anyone have any ideas of how I can start?
25 views

### Reporting result of classifiers [duplicate]

I am using a binary classifier in my research. I wanna report the obtained result of classifier, but I am not sure about it. The learning prosedure is as follows: ...
20 views

### Does this type of diagram showing conditional mean have a name?

I'm trying to show the skill of a prediction $\hat{Y}$ relative to a true value $Y$ beyond simple statistics like MSE or $r^2$. I usually like looking at the scatter plot of $\hat{Y}$ vs $Y$ but in ...
36 views

### GBM Performance on different sampling techniques

I am working on a healthcare data set for breast cancer patients. This data set is class imbalances and the distribution of positive and negative classes is 80%/20%. In order to deal with the class ...
21 views

### Uncertainty assignment to genetic algorithm solution

I have a black-box function for which I need to find its global maximum. I do this via a genetic algorithm (GA) since I can't solve it analytically. Since the GA can't guarantee to return the true ...
58 views

### Which is better, MSE or classification rate, for minimizing the likelihood of overfitting?

During the process of selecting a classification model (with cross validation), if one uses MSE of the predicted posterior probabilities as a selection criterion, then a model ...
23 views

### Standard performance measure for regression?

I will perform time-series prediction and I will report the accuracy of my system with a measure like RMSE or MAE. However, the variables I will predict are in different ranges. So let's say one is ...
13 views

### Term for results being similar in relation to each other, but differing is absolute magnitude

I have some results of several methods, applied to two tasks. Roughly they are on the lines of Task1 Method A: 20% Method B: 25% Method C: 40% Method D: 70% Method E: 71% Task2 Method A: 40% ...
9 views

### Performance metrics, prioritizing based on impact

I apologize in advance for the poorly worded question. If anyone has any suggestions, please advise. Also, help with tagging appropriately would be appreciated. I'm collecting performance metrics ...
57 views

### Visually summarising multiple diverse metric results

I have a multivariate time series (climate fluxes), and I have a number of model outputs that I want to asses. I also have number of metrics that I can use to assess the performance of each variable ...
21 views

### Performance measure

I am currently trying to show the trend of different team performances based on what percent of customer complaints they receive that they are able to handle within a given month. So if one team ...
74 views

### CART with Ordinal Response Variable using rpartScore Stuck

I'm trying to fit a decision tree over some data which has ~40K rows and ~200 features. The response variable, y, is ordinal and takes values ...
44 views

### What should the AUROC be on the test set when no positive example is present?

Assume we have a probabilistic, binary classifier. We compute the AUROC on a test set in which no positive example is present (i.e. the ground truth is always 0). What should the AUROC be?
13 views

### suitable flexibility of statistical learning method and number of predictors and number of samples

Out of a flexible and an inflexible statistical learning method, which one is expected to perform better when the number of samples is extremely large and the number of predictors is small? the ...
98 views

### What's the reasoning behind presenting unvalidated AUC as a measure of model fit or performance?

Often one sees, particularly in the biomedical literature, papers that analyze the performance of a risk prediction model in terms of the AUC or the area under the ROC curve. If the AUC is suitably ...
20 views

### Taking longer sample should not reduce the variance of the mean

I have not realized completely why n times longer samples should give you n times shorter variance. However, I measure performance of my programs and there are some fluctuation due to PC state ...
28 views

### Relative speed of gaussian process classification

I used gaussian process classification implemented in Matlab gpml toolbox and also in R kernlab. For my problem - 600*14 matrix with two classes - it trains order of magnitude slower than other ...
182 views

### How to compare (probability) predictive ability of models developed from logistic regression?

I know some well-known measures are $c$ statistic, Kolmogorov-Smirnov $D$ statistic. However, as far as I know, those statistics take into account only of the rank order of the observations, and is ...
24 views

20 views

### Usage of precision and recall vs type I and II errors

What is the advantage of using precision and recall, instead of type 1 and 2 errors? How about disadvantage? The two groups can be derived from each other. But it seems that they are used often in ...
44 views

### How use F1 score in an unbalanced binary classification problem?

I have two trained models (MLP and SVM) that want check on unbalanced binary samples (out of sampl - True samples =3000, False samples = 200). I found that i can use ...
15 views

### paired subject design test of proportions

I am analyzing a performance data set and would like to compare across two different conditions. I have three subjects, which were tested over a number of trials. Each trial had a chance of being in ...
56 views

### Why added predictors improved performance in decision trees if they don't appear in the model?

although I've been working some months now with decision trees, I still have issues understanding some things and also finding a right source to answer my questions. Maybe I'm not using the right ...
20 views

### Measure performance of a multiples classifier that is a combination of binary support vector machines

I am comparing the performance of multiple schemes to combine binary SVMs into one multiples classifier. I was hoping to use the cross entropy, however I cannot figure out how to apply it. For ...
1k views

### Why isn't the holdout method (splitting data into training and testing) used in classical statistics?

In my classroom exposure to data mining, the holdout method was introduced as a way of assessing model performance. However, when I took my first class on linear models, this was not introduced as a ...
30 views

### does the area under the curve (AUC) has some interpretation? [duplicate]

I know that the ROC curve plots true positives vs false positives, but does it have any other interpretation, or is it just an arbitrary performance measure? Also, in the case of very unbalanced ...
44 views

### Graph generation model effect on performance of Spectral graph algorithms

I use spectral graph algorithms for finding community structures, specifically the Leading Eigenvector Method (http://arxiv.org/abs/physics/0605087). I try analyzing the performance of these ...
25 views

### Ranking performance measure

I have a system able to produce a ranking of some operations according to their anomalous score. I have N anomalous operations inside the dataset, which I expect to be in the first N positions (best ...
728 views

### SVM and SMO main differences

I am unable to clearly see the main differences between SVM & SMO. While I get the fact that SMO provides better algorithm for QP solvers but I see that when I use this in Weka on my MacBook it ...
344 views

### What are good criteria for performance evaluation of algorithms in a regression problem?

I'm classifying different algorithms on the wine quality dataset. The quality ranges from between 0 - 10 based on 11 other attributes. Here is the data. I'm treating this as a regression problem. ...
166 views

### Combining different metrics to calculate composite score

Let's assume I have 3 metrics to evaluate a student. 1)Test Marks (0-50) 2)Weight of the student (KGs) 3)% Attendance (0%-100%) I want to calculate a total composite score which is a function of ...
109 views

### Interpreting high P-Value in ordinal rank correlation measures

I'm currently evaluating a machine-learned ranking task. Two (among other) performance measures are Spearman's Rho and Kendall's Tau (rank or concordance correlation). I'm basically comparing a ...
81 views

### Slow Lasso Performance Using sklearn

I am trying to use scikit-learn's LassoCV and/or ElasticNetCV functions to model a dataset with a large (>800) number of ...
847 views

### Levinsohn-Petrin TFP estimation in Stata without proxy

I'm trying to compute TFP by using levpet Stata command. I use revenue estimation, yet it requires proxy option which is intermediate input (usually material ...