# Questions tagged [method-comparison]

Questions about comparisons of different statistical methods

106 questions
Filter by
Sorted by
Tagged with
64 views

### Comparing different model with the same output

I work on developing 3 different models for predicting land surface temperatures in an area (validation is done with images taken by a thermal drone). The models are: A thermodynamic mechanistic ...
25 views

### compare correlation between two matrices

We have two algorithms that assign a score to the foreground of how well it fits the background. And we want to know which algorithm is closer to human perception. Then we collect an evaluation set ...
101 views

### What is better in Monte Carlo integration: product of means or mean of products?

Let $X$ and $Y$ be two independent continuous random variables with pdfs $f_X$ and $f_Y$, respectively. Let $\varphi_1$ and $\varphi_2$ be two continuous functions from ${\mathbb R}$ to ${\mathbb R}$. ...
25 views

### why the result of primal sim is not the same as linear kernel?

I have a data set with 36000 rows and 9 columns. so n<< m. this is multi classification SVM. I solved this by primal model in OSQP. then I used R package e710 and svm with linear kernel. the ...
59 views

### When to use the full and the conditional likelihood

In the context of estimating parameters of a time series model, we may consider either the full likelihood or the conditional likelihood. I was wondering in what situations the full ...
37 views

### Evaluate methods across datasets based on distance metrics

I want to evaluate $n$ methods, says $M_1,\cdots,M_n$ on $m$ datasets $D_1, \ldots, D_m$. For each method $M$, I have a score function $f(M,D) \in [0, \infty)$ for dataset $D$. A perfect method will ...
15 views

### if different methods have difference 0.5 in third decimal place while finding accuracy then what to conclude?

For the comparison of different feature selection methods and evaluating performance metrics. After evaluation when their is only 0.5 difference in third decimal place. Is this difference is ...
150 views

### Which method is correct for calculating total deviance explained for boosted regression trees?

I have the following output from a boosted regression trees model and I would like to calculate the total deviance explained. ...
60 views

### Can you compare the mean of one data set with the median of another?

I’m having an argument with another numerically literate person. We both saw a news site state that under no circumstances should you compare the median of one set with the mean of another. It’s ...
60 views

### comparison of machine learning models

I’m struggling with a statistical problem in machine learning: I’m using event-related fMRI responses (voxels) evoked by graded painful stimulations as feature. The targets are the scores (labels) of ...
179 views

### Reasons for different parameters via MoM and MLE

I got a study of 210 samples and I tried fitting gamma distribution to them. I used method of moments and maximum likelihood estimation to calculate the parameters, but parameters came out quite ...
6 views

### Evaluating a set of algorithms based on daily performance

I work in a production system which can use three competing algorithms, say $A_1,A_2,A_3$ for a customer on a daily basis. On each day, for a given customer, the production system looks at how much ...
40 views

### Multiple classifier comparison with Bayesian statistics

I've been recently working on some classifier comparisons. I've come to realize, the critical distance diagrams introduced $\approx$ 10 years ago might not be the most relevant solution for this ...
254 views

### Compare clustering results with different attributes and number of clusters

I used K-means to cluster a large data set that has millions of samples. I tried to create the clusters with different sets of attributes, which, as a result, generated different optimal number of ...
48 views

### Statistical difference of machine learning predictions

I am applying a machine learning classification method to two difference scenarios for my boss (in this context, I am using a random forest). Due to the nature of the problem, there is high class ...
53 views

### Analysis of Pre-post Likert-type scales: Intervention vs. Control group

I gathered the following data: Intervention group (n= 24, received treatment): pre- and post- scores of different psychometric questionnaires Control group (n=23, inactive): same questionnaires, same ...
31 views

### How to compare my model's performance with that of others?

I'm building an audio classifier and would like to compare my model's performance with that of others. I use the Urban Sound Dataset. I searched for already existing results on this dataset but couldn'...
126 views

### comparison of two estimators

Assume we have a data set $\mathbf{x}_{n} = (x_{1}, \dots, x_{n})$. Let $\delta_{1}(\mathbf{x}_{n})$ and $\delta_{2}(\mathbf{x}_{n})$ be two consistent estimators of some parameter $\theta \in R^{k}$. ...
472 views

### How to compare two levels of one factor

I´ve run a GLMM model with the following variables: Response variable: continuous (with positive, negative and zero values; Explanatory variable: 1 factor with 6 levels (DB, DF, NDB, NDF, BB, BF); ...
370 views

### Clarification on how to perform nested cross-validation

I'm currently trying to get lifetables for our population based on death and population counts. My first idea was to follow this paper methodology but after some discussions we are planning to use ...
280 views

### Taxonomy/overview of machine learning techniques [duplicate]

My question: I'm looking for a taxonomy/bestiary/overview of machine learning techniques. I would like to learn 1) how these methods relate to each other, and 2) the relative costs and benefits (and ...
2k views

### Which method to use to compare two sets of data for similarity? [closed]

I need to validate that my simulation results are correct. So I have two sets of data, my simulation results, and calculations I obtained from an established commercial software to compare against. ...
9 views

### Comparing a traditional and technology based tool

I am trying to compare a printed word dictionary with a digital dictionary that included added features like sound and video feedback. For this I have selected 60 participants and asked 5 questions (...
41 views

### Model comparing with statistical test

I am comparing few classification models (performance in AUC of ROC curves). I uses 10 times repeated 5 fold cross-validation so I have 50 values of AUC for each model. The key information is, that ...
102 views

### Ordered probit model prediction: why highest probabilities and not number of thresholds exceeded?

I'm running ordered probit regression models with polr in R. This question states that for prediction, ...
137 views

### Most widely accepted method of SVM model validation?

I am building an SVM based classifier (libsvm) for two distinct balanced classes which are non-linear but separable with an RBF kernel. Both classes have 175 instances each, which must be utilized ...
56 views

### How to check significant change in the actual population, not sample [duplicate]

I'm trying to understand how I would decide whether a policy actually had a meaningful impact by comparing the population itself, not samples. An example that comes to mind is comparing the number of ...
164 views

### Method(s) to quantify the hysteresis of a data set

I have a data set for two different mixtures of a material with 5 and 6 samples in each data set. I would like to quantify the hysteresis each sample in a "generally accepted" way. The following is my ...
2k views

### Most well-known set-similarity measures?

I know of the Jaccard index and the Sørensen-Dice coefficient for computing set similarity, but have been unable to find any other algorithms related to set similarity. This site contains quite a few ...
2k views

### Generating Random Data Sets for Linear Regression with Random Slope and Error Term in R

I want to test the effects of sample size on Deming regression using simulated paired data in R. As the data are paired, the expected slope value should be 1 and the intercept 0. The code I have ...
28 views

### What statistical analysis to use to compare two measurment tools of creativity?

Participants took two different creativity assessment test. I al trying to make a comparison between the two measurment tools and see which one gives more insight about the level of creativity. What ...
122 views

### Specifying the Error Ratio in Deming Regression

We use Deming Regression for method validation in our clinical ELISA laboratory. As we run clinical trials this needs to be performed if we get a new reagent batch, batch of plates, new plate reader ...
11 views

### Validation about chosen graph showing rates with big gap between them

I'm trying to show the variation in time of 4 variables, in which the first one is in the order of 20-30 and the remainder are all smaller than 2. The growth rates for each are: 52, 70, 8.85 and 19.06....
116 views

### weighted sampling from posterior distribution (i.e. regression predictions)

I am comparing measurement instruments A and B to assess their agreement on measuring feature x. Measurements with A are cheap. I have made measurements on thousands of randomly selected objects with ...
109 views

### How to compare effectiveness of methods of parameter estimation on the small samples?

Abstract example. Let's say, I want to compare MLE (Maximum Likelihood Estimation) and LSE (Least Squares Estimation) performance on the small samples for simple linear regression. My idea: to ...
19 views

### Comparing difference of means due to individual variability

So in our statistical test, we are using a dependent t-test because we are comparing the means between pre and post conditions in a physiology lab. Since there is a lot of variability between two ...
562 views

### Determine correct statistical algorithm for mixed-type data

I have a dataset consisting of ~20000 lines and 8 columns. It is a mixture of binary and float data, but one binary variable is the important one and I want to create a model to predict the other ...
641 views

### How to obtain the same results of a random forest model using caret and randomForest?

I am trying to understand how does building a regression model with caret's train () function differs from randomForest(). For my excercise, I am using the iris dataset. As shown in the code below, I ...
313 views

### A good metric to simultaneously compare both mean and variance

I am new to statistics, and have a fairly basic question. I have two sets of data that I want to compare with a third dataset. I am only interested in comparing the means and variances. Turns out ...
185 views

### Visualizing differences between alternative clusterings?

Does anyone know of a good way to visualize differences between alternative clusterings? I know there are some measures which can be used to quantify the similarity between two partitionings, e.g.: ...
6k views

### how can i compare two groups of data?

I asked a question here how can i see if my data are coming from two different population? and seems like I cannot correctly communicate and the person who is writing to me is making me confused. So ...
54 views

### What is the difference between jackknifing and LOOCV? [duplicate]

Leave-one-out cross-validation model technique is very similar to jackknifing resampling, because both omitting each training case and perform retraining of the network on the left-out subset. On ...
73 views

### What are the benefits of a “deep” over a “shallow” architectures? [duplicate]

What are the benefits of a deep neural network architecture over a shallow architecture with many computational elements. What are pros and cons of these two architectures and its limitations?
105 views

### Determining a cut-off value from two populations of unequal sample sizes

I have two groups (healthy and disease, different sample sizes) of data, pooled from the individual subject's measurements of a certain region of the brain. The size of the brain region may be ...
29 views

### Statistical comparison between two stochastic algorithms

I am working on a mechanics problem with variability in material properties. For that I need to analyze the efficiency of two methods for same accuracy (confidence level I guess?), the first one being ...
96 views

### Need a comparison method to compare original data values and data values derived from a method

I have a gene expression dataset for which I have the original data values and I have put this dataset through a few methods to test their accuracy of prediction. I need to compare the results from ...
111 views

### Can an independent t-test be used on paired data when the pairing is unknown?

Suppose the effectiveness of a training course is examined, and performance of each individual in a group is taken both before and after, and the differences are compared in a paired $t$-test. Would ...
81 views

### Correspondence between two different depression scales?

I have become involved in a project where psychiatrists are studying the correspondence between two scales measuring the level of depression among a specific patient group. One of the scales is for ...