The tag has no usage guidance.

learn more… | top users | synonyms

0
votes
0answers
9 views

pyJAGS vs RJAGS

I was wondering if there is any empirical studies out there to compare performance of different JAGS bindings, specifically RJAGS and pyJAGS. I am mainly interested in speed. Currently I have an ...
2
votes
2answers
52 views

AUROC equal to 1.0 means overfitting?

Evaluating the classifier I implemented for university, I am observing an AUROC (Area under curve of the ROC) of 1.0 (which means a TP rate of 1 and a FP rate of 0.0) The dataset used for training ...
1
vote
0answers
16 views

Determinant of confusion matrix: how useful is it as a performance metric?

I was thinking a bit about confusion matrices and it came to my mind the determinant of a confusion matrix could be an useful performance metric in classification. Indeed, I got some results in ...
1
vote
3answers
41 views

Is K-means performance a bottleneck everywhere?

I've read a paper about a sped-up version of k-means: Ding et al. (2015). Yinyang K-Means: A Drop-In Replacement of the Classic K-Means with Consistent Speedup. Now I wonder, is k-means' ...
0
votes
0answers
18 views

Doing cross-validation when diagnosing a classifier through learning curves

I have a theoretical question on the correct way to make learning curves to diagnose a classifier. To see a generic example of these curves one can refer to this (min 34 onward) lecture by Andrew Ng ...
0
votes
0answers
13 views

PyMC sampling is slow

I'm using pymc2 to estimate the parameters of a normal distribution. My data has shape 50000 x 6. Basically, I have 50K independent distributions and I want to obtain the parameters for each of them, ...
1
vote
0answers
61 views

How to assess logistic regression & reduce computational effort with imbalanced data

I have a highly imbalanced data set (ratio 1:150) with four predictors, where two are correlated. The data can be found here, you can also see the two figures below. I would like to use logistic ...
0
votes
0answers
23 views

How to measure clustering algorithm performance? [duplicate]

For supervised learning, both regression and classification have ground truth. The model performance can be measured against ground truth. For example, $R^2$ in regression or accuracy (0-1) in ...
0
votes
0answers
18 views

Creating one performance measure

I want to create one operational performance score for each day so that we know if the performance was good or bad in our call centers, to be able to compare days, and to be proactive in maximizing ...
4
votes
0answers
86 views

optimism-corrected regression coefficients using Frank Harrell's method?

I used a regularized (LASSO) cox regression to estimate relapse times of patients and used Frank Harrell's bootstrapping method to obtain an optimism-corrected performance estimate of my model. I am ...
1
vote
0answers
22 views

Reference summarizing various machine learning algorithms' computational complexity

For example, suppose you train a linear regression model using the Normal Equation, on a training set $\mathbf{X}$ containing $m$ instances and $n$ features. The Normal Equation requires computing $(\...
2
votes
0answers
59 views

Interpret ROC/AUC values with respect data

I am using R to plot ROC curves. I have a prediction matrix, where each column shows the prediction values corresponding to different approaches. Also, I have a ...
0
votes
0answers
23 views

What is Ideal Point Error (IPE)?

I am working on a rainfall forecasting study where I aim to compare my results against observed values or any other model. I have been asked to develop a new performance measurement which is Ideal ...
0
votes
0answers
11 views
0
votes
0answers
11 views

What is a better approach to create operational performance score, LSM or PCA?

I want to create one operational performance score for each day so that we know if the performance was good or bad in our call centers, to be able to compare days, and to be proactive in maximizing ...
0
votes
0answers
11 views

Selecting a network based on Validation performance or test performance? [duplicate]

Sorry for repeating this question but honestly I could not find the exact answer for that.There are lots of opposite answers and solutions. If the dataset has been divided to 3 subsets training/...
0
votes
0answers
21 views

External validation of a group's machine learning process

Say there's a group (that is opaque to you) that's heavily using machine learning methods to produce some outputs. You don't have access to their input data or code but can ask high level questions ...
0
votes
0answers
26 views

Variable that discriminates but performance metrics don't improve

I have a score adjusted by a logistic regression done with variables let's say X1 to X5. I'm not including a variable X6 to be chosen. Now, if I let the algorithm (stepAIC from R) choose among all of ...
2
votes
0answers
28 views

What would be a proper statistical test for multiple models and model selection?

Suppose I have a data set of N observations (n = 1...N) for out-of-sample estimation and values of ($y_n$). I have also ...
0
votes
0answers
28 views

My predictors have strong collinearities, yet linear regression performs as good as partial least squares. Why?

I am trying to predict a single response from twelve explanatory variables. There exist strong correlations between my variables. The correlation matrix looks as follows, and the data have a ...
0
votes
0answers
5 views

Ranking districts (on basis of % cahnge) after controlling initial conditions

We want to rank Indian districts as good/bad/average performer on the basis of change in access to latrine between 2001 and 2011. We calculate the percentage point change in access to latrine ...
3
votes
2answers
136 views

comparison of SGD and ALS in collaborative filtering

Matrix factorization is widely applied in collaborative filtering, and briefly speaking, it tries to learn the following parameters: $$\min_{q_u,p_i}\sum_{\{u,i\}}(r_{ui} - q_u^Tp_i)^2$$ And we could ...
0
votes
0answers
85 views

Can I add the scores of three principal components

I wish to create one operational performance score for each day so that we know if the performance was good or bad, to be able to compare days, and to be proactive in maximizing performance. I did ...
0
votes
0answers
22 views

Performance metric for one-class SVM classifier with small number of instances

I am using LibSVM one-class classifier for detecting a particular hand gesture from accelerometer data. I have trained two categories of classifiers: (a) user-dependent: data from the same user is ...
5
votes
1answer
62 views

Geometric Interpretation of Whether SVMs are performing well or not

I came across this research paper which contained this figure which talks about the center of mass (presumably, of the training dataset's datapoints?) and represents the solution of an SVM as ...
1
vote
2answers
91 views

R performance limitations [closed]

I attempted to predict crime category from its X and Y coordinates based on San Francisco Crime Statistics (https://www.kaggle.com/c/sf-crime). And it turned out that glmnet is pretty slow for fitting ...
0
votes
0answers
12 views

Comparing single-value and probabilistic online predictions

Let $\mathbf{x}^{(T)} = \{ x_1, x_2, \cdots, x_T \}$ ($x_t \in \mathbb{R}^d$) and $\mathbf{y}^{(T)} = \{ y_1, y_2, \cdots, y_T \}$ ($y_t \in \mathbb{R}$) be two time series. If I had a "single-value",...
7
votes
3answers
424 views

How to select or validate the selection of a clustering method?

One of the biggest issue with cluster analysis is that we may happen to have to derive different conclusion when base on different clustering methods used (including different linkage methods in ...
0
votes
0answers
10 views

Compare accuracy of a new light meter

I'm performing a study where i want to test the accuracy of a new light meter (A) againsta a standard calibrated and validated light meter (B) as a reference. I have measure different light ...
0
votes
0answers
79 views

Year on year model performance evaluation via bootstrapping

I am interested in monitoring the yearly evolution of model’s performance (a Loss Given Default model more specifically). The yearly additional data, however, is very limited (maximally 5 observations)...
0
votes
1answer
39 views

Compute accuracy of model evaluation

Say the number of negativ classes is $9990$ and the number of positive classes is $10$. If a model predicts all examples to belong to the negative class, how accurate is this prediction? So what we ...
0
votes
0answers
20 views

In the classification framework, is AUROC a performance measure or metric?

I guess, the title is self-explaining. I have seen both so far and was wondering if there is a correct term or whether it does not really matter.
1
vote
0answers
16 views

Well defineness of the definition of F1 score

The F1 score is defined as the harmonic mean of precision and recall: $$ F_{1}=2\cdot {\frac {\mathrm {precision} \cdot \mathrm {recall} }{\mathrm {precision} +\mathrm {recall} }}.$$ The above ...
0
votes
0answers
21 views

Matched pairs versus multiple t-test measurements for task performance over time

I am trying to compare performance on a task over time across one group based on initial performance, but am running into a little trouble. As background, I am examining a group of students' ...
3
votes
2answers
43 views

Creating a rank on performance and size

Ranking is not something that I need to do particularly frequently in my job and its probably the weakest part of my statistics background. I have a dataset containing a list of items and a ...
0
votes
1answer
88 views

How to compare classification methods in terms of performance?

I'd like to compare logistic regression to classification trees. In a first step, I compared the theoretical framework of the two classifiers. In a second step, I compared the performance using a ...
1
vote
0answers
184 views

Why MSE of this MATLAB neural network is so high?

I'm using this code to train a neural netowrk in MATLAB: ...
1
vote
1answer
72 views

Is it valid to get better performance in logistic regression using only a subset of the coefficients?

I have an imbalanced data set containing 12% of the positive class 88% negative. First, I ran a logistic regression with all my coefficients and got an average accuracy of 0.91 (I know that's not ...
0
votes
0answers
22 views

MLP and RBF performance assessment (Matlab)

I hope you really help me. I'm now approaching to the world of data mining and pattern recognition and I need a little help. I have a data set composed by 4 struct of features. Each struct contains 18 ...
1
vote
1answer
24 views

How to account for games and number of teams in share of points?

I am doing research on the relationship of income inequality on team performance in soccer. The team performance I measure through share of points, so the number of points a team got in a particular ...
0
votes
1answer
16 views

Performance of estimators

I have 11 different estimators and try to figure out which of them performs best. The regression line between given and estimated values ideally should have a slope of 1 and an intersect at 0,0; which ...
0
votes
0answers
40 views

Model stability- ratios in same bucket should keep constant over different population?

I need to explain the variation in a validation table for a propensity score to buy insurance. I trained in R a logistic model with 4 months of 2014. Calculated limits of non-overlapping deciles of ...
6
votes
2answers
141 views

How can I assess performance of a semi-supervised learning method?

I'm working with a semi-supervised learning task, where I only have positive and unlabelled data (PU learning). I've tested a few algorithms and would like to assess their performance. For ...
0
votes
1answer
40 views

Best performance evaluation measure for GARCH model? [duplicate]

I want to compare the performance of various volatility models like GARCH, eGARCH, and gjrGARCH from actual volatility (computed using high frequency data). I found three common performance evaluation ...
0
votes
1answer
83 views

How can I compare two mutual funds' performance with a sparse set of data?

I want to compare the performance of two mutual funds. The only data I have is annual returns for the past 7 years. So I have 7 observations for Fund 1 and 7 observations for Fund 2. In addition, I ...
0
votes
1answer
79 views

Calculate thresholds of factor analysis output to classify data to 5 classes

Suppose that we calculate a composite indicator for some companies using Factor Analysis (FA) by combining five features to one output (calculate weights of input features). This is histogram of ...
0
votes
0answers
11 views

How does one show that an extension of an algorithm performs as well or better than the original?

If we have an extension of an algorithm (for example, we take OLS and scale the inputs, and solve it as a reformulated OLS problem), how does one show that the extension performs as well or better ...
0
votes
0answers
13 views

bootstrap simulation for CI

I have generate 4 functions to test the performance of my bootstrap for obtaining 95% confidence intervals on the mean of a univariate sample of data. However, it does not run properly with the last ...
0
votes
0answers
30 views

Performance Evaluation of Search Engines Test

I have to apply double-blind testing method to evaluate the performance of any two search engines.(ex. Google vs. Bing) I'm drawing a blank here. Anyone have any ideas of how I can start?
0
votes
1answer
73 views

Reporting result of classifiers [duplicate]

I am using a binary classifier in my research. I wanna report the obtained result of classifier, but I am not sure about it. The learning prosedure is as follows: ...