Questions tagged [method-comparison]

Questions about comparisons of different statistical methods

Filter by
Sorted by
Tagged with
0
votes
2answers
29 views

McNemar-Bowker exact test in R: difficulty interpreting the results

I am trying to use the McNemar-Bowker test to test the difference in performance for 2 classifiers. Since my input matrix is sparse and the sum of some of the symmetric cells is less than 10, I am ...
3
votes
1answer
30 views

Comparing the accuracy of binary classifiers using iterated cross-validation

Let's say that you want to compare two binary classifiers (e.g., LDA and linear SVM) for a given research question, the question being "which one will probably perform best for the problem at ...
1
vote
1answer
31 views

Overlapping quinquennial data - How to compare?

I actually do not know if this is the right place to ask, but I hope someone can help me. I have a dataset that provides the number of newborns in quinquennial periods. For instance, I have the number ...
0
votes
0answers
10 views

covariance estimation method comparison

There seem to be a bunch of ways to estimate the covariance and precision matrices (in sklearn there are several, including MinCovDet and graphical lasso, but also including primitive (which does not ...
0
votes
0answers
5 views

Comparing differences between three or more unpaired groups (DV categorical and non-normal continuous) and describing used tests

I have a made-up table as follows, including a footnote describing given p-values. P-value shows the differences between all subgroups, and it was derived from the Kruskal-Wallis test for continuous ...
1
vote
1answer
33 views

Do XBGoost and LightGBM only use trees as base learners?

From my understanding of GBM is that it can take not only decision trees as base learners, but different weak learners as well (e.g., linear models), since it relies on sequential gradient descent. ...
1
vote
2answers
87 views

Comparison of Bayesian and Classical estimates

Is it correct to compare Bayesian and Classical estimates using Mean Squared Error (MSE)? MSE is a criterion that is used in the classical paradigm. For example: I am comparing the performance of ...
1
vote
0answers
48 views

Which regression analysis to use?

I have percentage change data compared to a baseline of public and private transport activity over 1 year. With the help of one or two independent variables I want to analyze if there is a switch ...
0
votes
0answers
17 views

Need to test heteroscedasticity before using the Kruskal-Wallis test

My question is whether or not I have to test heterodasticity before using the Kruskal-Wallis test. What I understand is that if I want to compare medians, then I have to check for heteroscedasticity, ...
0
votes
0answers
12 views

Compare the means of dependent samples that are not paired? Group size of dolphins sighted?

I'm looking for a statistic way to compare the means of dependent samples which are not paired. Context: I study a dolphin population in which individuals are clearly identified. My data consists in ...
2
votes
5answers
198 views

Interval Estimation for a Change in a Binomial Proportion

I'm not sure how to estimate the confidence interval (CI) for a change in a small sample size binomial proportion using the same sample set both times. I have two methods that I would like to compare (...
0
votes
1answer
26 views

Statistical Significance on shops performances with a marketing promotion

I have data regarding the performances of shops on a certain time period: in this data some shops had an active marketing promotion, and some didn't. I would like to understand whether the shops with ...
0
votes
0answers
124 views

Comparing two clustering methods with adjusted Rand index - How to make labels match

I have a set of data (countries), and I use a clustering method (Kmeans) to create two partitions on them based on different data (diet and covid stats). So I get two groups of partitions, and I want ...
0
votes
0answers
30 views

What methods could I use to compare two methods of quantitative measurement where subjects were measured once by each method?

I used two separate methods for measuring a quantitative value from 196 subjects. One method can be considered a "golden standard" while the other is crude but cheaper. Each subject was ...
9
votes
2answers
150 views

Propensity score matching vs non-parametric regression

I am trying to understand the benefit of propensity matching over non-parametric regression for causal inference from non-experimental data. As background: the way I understand it, parametric ...
1
vote
0answers
35 views

Efficient way to compare algorithms performance with missing samples

I'm currently doing some experiments to compare more than 70 different configurations (different parameters of an evolutionary algorithms) that have 30 runs each (repeated due to stochastic results). ...
0
votes
0answers
20 views

Merge before or after calculation?

I have a set of yearly panel data. For the example, let's say I have 5 dfs each representing one year of survey. The data from each dfs is representative of the entire population. I want to compute ...
1
vote
1answer
112 views

What is the difference between simulated annealing and deterministic annealing?

Not sure if this is the right place, but I was wondering if someone could briefly explain to me the differences & similarities between simulated annealing and deterministic annealing? I know that ...
1
vote
2answers
291 views

How can I tell if images are from the same distribution?

What are some tests I could run on two groups of imagery to see if they're from the same distribution? Imagine, for example, that you want to augment a dataset with another, and you want to make sure ...
1
vote
0answers
1k views

R: Caret package: AUC (“ROC”) standard error from cross validation

This seems like a basic questions, so I'm likely missing the big picture here... I would like to know the AUC of each fold of a cross-validation performed in Caret's train function, in order to ...
0
votes
1answer
51 views

Statistical tests for comparing (black box) model performance on two datasets

Introduction I have two datasets, dataA (real images) and dataB (generated images). I also have a model that has been trained on dataC. dataC might have some overlap with dataA but for certain does ...
2
votes
1answer
170 views

Comparing different model with the same output

I work on developing 3 different models for predicting land surface temperatures in an area (validation is done with images taken by a thermal drone). The models are: A thermodynamic mechanistic ...
0
votes
1answer
142 views

compare correlation between two matrices

We have two algorithms that assign a score to the foreground of how well it fits the background. And we want to know which algorithm is closer to human perception. Then we collect an evaluation set ...
7
votes
2answers
117 views

What is better in Monte Carlo integration: product of means or mean of products?

Let $X$ and $Y$ be two independent continuous random variables with pdfs $f_X$ and $f_Y$, respectively. Let $\varphi_1$ and $\varphi_2$ be two continuous functions from ${\mathbb R}$ to ${\mathbb R}$. ...
0
votes
0answers
49 views

why the result of primal sim is not the same as linear kernel?

I have a data set with 36000 rows and 9 columns. so n<< m. this is multi classification SVM. I solved this by primal model in OSQP. then I used R package e710 and svm with linear kernel. the ...
2
votes
1answer
85 views

When to use the full and the conditional likelihood

In the context of estimating parameters of a time series model, we may consider either the full likelihood or the conditional likelihood. I was wondering in what situations the full ...
0
votes
2answers
41 views

Evaluate methods across datasets based on distance metrics

I want to evaluate $n$ methods, says $M_1,\cdots,M_n$ on $m$ datasets $D_1, \ldots, D_m$. For each method $M$, I have a score function $f(M,D) \in [0, \infty)$ for dataset $D$. A perfect method will ...
1
vote
1answer
18 views

if different methods have difference 0.5 in third decimal place while finding accuracy then what to conclude?

For the comparison of different feature selection methods and evaluating performance metrics. After evaluation when their is only 0.5 difference in third decimal place. Is this difference is ...
2
votes
1answer
766 views

Which method is correct for calculating total deviance explained for boosted regression trees?

I have the following output from a boosted regression trees model and I would like to calculate the total deviance explained. ...
1
vote
1answer
1k views

Can you compare the mean of one data set with the median of another?

I’m having an argument with another numerically literate person. We both saw a news site state that under no circumstances should you compare the median of one set with the mean of another. It’s ...
2
votes
0answers
64 views

comparison of machine learning models

I’m struggling with a statistical problem in machine learning: I’m using event-related fMRI responses (voxels) evoked by graded painful stimulations as feature. The targets are the scores (labels) of ...
3
votes
2answers
504 views

Reasons for different parameters via MoM and MLE

I got a study of 210 samples and I tried fitting gamma distribution to them. I used method of moments and maximum likelihood estimation to calculate the parameters, but parameters came out quite ...
1
vote
0answers
7 views

Evaluating a set of algorithms based on daily performance

I work in a production system which can use three competing algorithms, say $A_1,A_2,A_3$ for a customer on a daily basis. On each day, for a given customer, the production system looks at how much ...
2
votes
1answer
47 views

Multiple classifier comparison with Bayesian statistics

I've been recently working on some classifier comparisons. I've come to realize, the critical distance diagrams introduced $\approx$ 10 years ago might not be the most relevant solution for this ...
2
votes
2answers
530 views

Compare clustering results with different attributes and number of clusters

I used K-means to cluster a large data set that has millions of samples. I tried to create the clusters with different sets of attributes, which, as a result, generated different optimal number of ...
1
vote
0answers
48 views

Statistical difference of machine learning predictions

I am applying a machine learning classification method to two difference scenarios for my boss (in this context, I am using a random forest). Due to the nature of the problem, there is high class ...
2
votes
0answers
80 views

Analysis of Pre-post Likert-type scales: Intervention vs. Control group

I gathered the following data: Intervention group (n= 24, received treatment): pre- and post- scores of different psychometric questionnaires Control group (n=23, inactive): same questionnaires, same ...
1
vote
0answers
38 views

How to compare my model's performance with that of others?

I'm building an audio classifier and would like to compare my model's performance with that of others. I use the Urban Sound Dataset. I searched for already existing results on this dataset but couldn'...
1
vote
0answers
139 views

comparison of two estimators

Assume we have a data set $\mathbf{x}_{n} = (x_{1}, \dots, x_{n})$. Let $\delta_{1}(\mathbf{x}_{n})$ and $\delta_{2}(\mathbf{x}_{n})$ be two consistent estimators of some parameter $\theta \in R^{k}$. ...
2
votes
2answers
1k views

How to compare two levels of one factor

I´ve run a GLMM model with the following variables: Response variable: continuous (with positive, negative and zero values; Explanatory variable: 1 factor with 6 levels (DB, DF, NDB, NDF, BB, BF); ...
8
votes
3answers
1k views

Clarification on how to perform nested cross-validation

I'm currently trying to get lifetables for our population based on death and population counts. My first idea was to follow this paper methodology but after some discussions we are planning to use ...
6
votes
2answers
423 views

Taxonomy/overview of machine learning techniques [duplicate]

My question: I'm looking for a taxonomy/bestiary/overview of machine learning techniques. I would like to learn 1) how these methods relate to each other, and 2) the relative costs and benefits (and ...
2
votes
1answer
4k views

Which method to use to compare two sets of data for similarity? [closed]

I need to validate that my simulation results are correct. So I have two sets of data, my simulation results, and calculations I obtained from an established commercial software to compare against. ...
0
votes
0answers
10 views

Comparing a traditional and technology based tool

I am trying to compare a printed word dictionary with a digital dictionary that included added features like sound and video feedback. For this I have selected 60 participants and asked 5 questions (...
0
votes
1answer
173 views

Model comparing with statistical test

I am comparing few classification models (performance in AUC of ROC curves). I uses 10 times repeated 5 fold cross-validation so I have 50 values of AUC for each model. The key information is, that ...
1
vote
0answers
106 views

Ordered probit model prediction: why highest probabilities and not number of thresholds exceeded?

I'm running ordered probit regression models with polr in R. This question states that for prediction, ...
3
votes
1answer
243 views

Most widely accepted method of SVM model validation?

I am building an SVM based classifier (libsvm) for two distinct balanced classes which are non-linear but separable with an RBF kernel. Both classes have 175 instances each, which must be utilized ...
0
votes
2answers
61 views

How to check significant change in the actual population, not sample [duplicate]

I'm trying to understand how I would decide whether a policy actually had a meaningful impact by comparing the population itself, not samples. An example that comes to mind is comparing the number of ...
0
votes
0answers
231 views

Method(s) to quantify the hysteresis of a data set

I have a data set for two different mixtures of a material with 5 and 6 samples in each data set. I would like to quantify the hysteresis each sample in a "generally accepted" way. The following is my ...
6
votes
1answer
4k views

Most well-known set-similarity measures?

I know of the Jaccard index and the Sørensen-Dice coefficient for computing set similarity, but have been unable to find any other algorithms related to set similarity. This site contains quite a few ...