Questions tagged [method-comparison]

Questions about comparisons of different statistical methods

Filter by
Sorted by
Tagged with
2
votes
1answer
64 views

Comparing different model with the same output

I work on developing 3 different models for predicting land surface temperatures in an area (validation is done with images taken by a thermal drone). The models are: A thermodynamic mechanistic ...
0
votes
1answer
25 views

compare correlation between two matrices

We have two algorithms that assign a score to the foreground of how well it fits the background. And we want to know which algorithm is closer to human perception. Then we collect an evaluation set ...
7
votes
2answers
101 views

What is better in Monte Carlo integration: product of means or mean of products?

Let $X$ and $Y$ be two independent continuous random variables with pdfs $f_X$ and $f_Y$, respectively. Let $\varphi_1$ and $\varphi_2$ be two continuous functions from ${\mathbb R}$ to ${\mathbb R}$. ...
0
votes
0answers
25 views

why the result of primal sim is not the same as linear kernel?

I have a data set with 36000 rows and 9 columns. so n<< m. this is multi classification SVM. I solved this by primal model in OSQP. then I used R package e710 and svm with linear kernel. the ...
2
votes
1answer
59 views

When to use the full and the conditional likelihood

In the context of estimating parameters of a time series model, we may consider either the full likelihood or the conditional likelihood. I was wondering in what situations the full ...
0
votes
2answers
37 views

Evaluate methods across datasets based on distance metrics

I want to evaluate $n$ methods, says $M_1,\cdots,M_n$ on $m$ datasets $D_1, \ldots, D_m$. For each method $M$, I have a score function $f(M,D) \in [0, \infty)$ for dataset $D$. A perfect method will ...
1
vote
1answer
15 views

if different methods have difference 0.5 in third decimal place while finding accuracy then what to conclude?

For the comparison of different feature selection methods and evaluating performance metrics. After evaluation when their is only 0.5 difference in third decimal place. Is this difference is ...
1
vote
1answer
150 views

Which method is correct for calculating total deviance explained for boosted regression trees?

I have the following output from a boosted regression trees model and I would like to calculate the total deviance explained. ...
1
vote
1answer
60 views

Can you compare the mean of one data set with the median of another?

I’m having an argument with another numerically literate person. We both saw a news site state that under no circumstances should you compare the median of one set with the mean of another. It’s ...
2
votes
0answers
60 views

comparison of machine learning models

I’m struggling with a statistical problem in machine learning: I’m using event-related fMRI responses (voxels) evoked by graded painful stimulations as feature. The targets are the scores (labels) of ...
3
votes
2answers
179 views

Reasons for different parameters via MoM and MLE

I got a study of 210 samples and I tried fitting gamma distribution to them. I used method of moments and maximum likelihood estimation to calculate the parameters, but parameters came out quite ...
1
vote
0answers
6 views

Evaluating a set of algorithms based on daily performance

I work in a production system which can use three competing algorithms, say $A_1,A_2,A_3$ for a customer on a daily basis. On each day, for a given customer, the production system looks at how much ...
2
votes
1answer
40 views

Multiple classifier comparison with Bayesian statistics

I've been recently working on some classifier comparisons. I've come to realize, the critical distance diagrams introduced $\approx$ 10 years ago might not be the most relevant solution for this ...
2
votes
1answer
254 views

Compare clustering results with different attributes and number of clusters

I used K-means to cluster a large data set that has millions of samples. I tried to create the clusters with different sets of attributes, which, as a result, generated different optimal number of ...
1
vote
0answers
48 views

Statistical difference of machine learning predictions

I am applying a machine learning classification method to two difference scenarios for my boss (in this context, I am using a random forest). Due to the nature of the problem, there is high class ...
2
votes
0answers
53 views

Analysis of Pre-post Likert-type scales: Intervention vs. Control group

I gathered the following data: Intervention group (n= 24, received treatment): pre- and post- scores of different psychometric questionnaires Control group (n=23, inactive): same questionnaires, same ...
1
vote
0answers
31 views

How to compare my model's performance with that of others?

I'm building an audio classifier and would like to compare my model's performance with that of others. I use the Urban Sound Dataset. I searched for already existing results on this dataset but couldn'...
1
vote
0answers
126 views

comparison of two estimators

Assume we have a data set $\mathbf{x}_{n} = (x_{1}, \dots, x_{n})$. Let $\delta_{1}(\mathbf{x}_{n})$ and $\delta_{2}(\mathbf{x}_{n})$ be two consistent estimators of some parameter $\theta \in R^{k}$. ...
2
votes
2answers
472 views

How to compare two levels of one factor

I´ve run a GLMM model with the following variables: Response variable: continuous (with positive, negative and zero values; Explanatory variable: 1 factor with 6 levels (DB, DF, NDB, NDF, BB, BF); ...
6
votes
3answers
370 views

Clarification on how to perform nested cross-validation

I'm currently trying to get lifetables for our population based on death and population counts. My first idea was to follow this paper methodology but after some discussions we are planning to use ...
6
votes
2answers
280 views

Taxonomy/overview of machine learning techniques [duplicate]

My question: I'm looking for a taxonomy/bestiary/overview of machine learning techniques. I would like to learn 1) how these methods relate to each other, and 2) the relative costs and benefits (and ...
1
vote
1answer
2k views

Which method to use to compare two sets of data for similarity? [closed]

I need to validate that my simulation results are correct. So I have two sets of data, my simulation results, and calculations I obtained from an established commercial software to compare against. ...
0
votes
0answers
9 views

Comparing a traditional and technology based tool

I am trying to compare a printed word dictionary with a digital dictionary that included added features like sound and video feedback. For this I have selected 60 participants and asked 5 questions (...
0
votes
1answer
41 views

Model comparing with statistical test

I am comparing few classification models (performance in AUC of ROC curves). I uses 10 times repeated 5 fold cross-validation so I have 50 values of AUC for each model. The key information is, that ...
1
vote
0answers
102 views

Ordered probit model prediction: why highest probabilities and not number of thresholds exceeded?

I'm running ordered probit regression models with polr in R. This question states that for prediction, ...
3
votes
1answer
137 views

Most widely accepted method of SVM model validation?

I am building an SVM based classifier (libsvm) for two distinct balanced classes which are non-linear but separable with an RBF kernel. Both classes have 175 instances each, which must be utilized ...
0
votes
2answers
56 views

How to check significant change in the actual population, not sample [duplicate]

I'm trying to understand how I would decide whether a policy actually had a meaningful impact by comparing the population itself, not samples. An example that comes to mind is comparing the number of ...
0
votes
0answers
164 views

Method(s) to quantify the hysteresis of a data set

I have a data set for two different mixtures of a material with 5 and 6 samples in each data set. I would like to quantify the hysteresis each sample in a "generally accepted" way. The following is my ...
4
votes
1answer
2k views

Most well-known set-similarity measures?

I know of the Jaccard index and the Sørensen-Dice coefficient for computing set similarity, but have been unable to find any other algorithms related to set similarity. This site contains quite a few ...
3
votes
1answer
2k views

Generating Random Data Sets for Linear Regression with Random Slope and Error Term in R

I want to test the effects of sample size on Deming regression using simulated paired data in R. As the data are paired, the expected slope value should be 1 and the intercept 0. The code I have ...
1
vote
0answers
28 views

What statistical analysis to use to compare two measurment tools of creativity?

Participants took two different creativity assessment test. I al trying to make a comparison between the two measurment tools and see which one gives more insight about the level of creativity. What ...
1
vote
0answers
122 views

Specifying the Error Ratio in Deming Regression

We use Deming Regression for method validation in our clinical ELISA laboratory. As we run clinical trials this needs to be performed if we get a new reagent batch, batch of plates, new plate reader ...
0
votes
0answers
11 views

Validation about chosen graph showing rates with big gap between them

I'm trying to show the variation in time of 4 variables, in which the first one is in the order of 20-30 and the remainder are all smaller than 2. The growth rates for each are: 52, 70, 8.85 and 19.06....
1
vote
0answers
116 views

weighted sampling from posterior distribution (i.e. regression predictions)

I am comparing measurement instruments A and B to assess their agreement on measuring feature x. Measurements with A are cheap. I have made measurements on thousands of randomly selected objects with ...
0
votes
1answer
109 views

How to compare effectiveness of methods of parameter estimation on the small samples?

Abstract example. Let's say, I want to compare MLE (Maximum Likelihood Estimation) and LSE (Least Squares Estimation) performance on the small samples for simple linear regression. My idea: to ...
1
vote
1answer
19 views

Comparing difference of means due to individual variability

So in our statistical test, we are using a dependent t-test because we are comparing the means between pre and post conditions in a physiology lab. Since there is a lot of variability between two ...
0
votes
1answer
562 views

Determine correct statistical algorithm for mixed-type data

I have a dataset consisting of ~20000 lines and 8 columns. It is a mixture of binary and float data, but one binary variable is the important one and I want to create a model to predict the other ...
1
vote
0answers
641 views

How to obtain the same results of a random forest model using caret and randomForest?

I am trying to understand how does building a regression model with caret's train () function differs from randomForest(). For my excercise, I am using the iris dataset. As shown in the code below, I ...
1
vote
0answers
313 views

A good metric to simultaneously compare both mean and variance

I am new to statistics, and have a fairly basic question. I have two sets of data that I want to compare with a third dataset. I am only interested in comparing the means and variances. Turns out ...
5
votes
1answer
185 views

Visualizing differences between alternative clusterings?

Does anyone know of a good way to visualize differences between alternative clusterings? I know there are some measures which can be used to quantify the similarity between two partitionings, e.g.: ...
3
votes
3answers
6k views

how can i compare two groups of data?

I asked a question here how can i see if my data are coming from two different population? and seems like I cannot correctly communicate and the person who is writing to me is making me confused. So ...
1
vote
0answers
54 views

What is the difference between jackknifing and LOOCV? [duplicate]

Leave-one-out cross-validation model technique is very similar to jackknifing resampling, because both omitting each training case and perform retraining of the network on the left-out subset. On ...
1
vote
0answers
73 views

What are the benefits of a “deep” over a “shallow” architectures? [duplicate]

What are the benefits of a deep neural network architecture over a shallow architecture with many computational elements. What are pros and cons of these two architectures and its limitations?
0
votes
1answer
105 views

Determining a cut-off value from two populations of unequal sample sizes

I have two groups (healthy and disease, different sample sizes) of data, pooled from the individual subject's measurements of a certain region of the brain. The size of the brain region may be ...
1
vote
0answers
29 views

Statistical comparison between two stochastic algorithms

I am working on a mechanics problem with variability in material properties. For that I need to analyze the efficiency of two methods for same accuracy (confidence level I guess?), the first one being ...
0
votes
1answer
96 views

Need a comparison method to compare original data values and data values derived from a method

I have a gene expression dataset for which I have the original data values and I have put this dataset through a few methods to test their accuracy of prediction. I need to compare the results from ...
3
votes
1answer
111 views

Can an independent t-test be used on paired data when the pairing is unknown?

Suppose the effectiveness of a training course is examined, and performance of each individual in a group is taken both before and after, and the differences are compared in a paired $t$-test. Would ...
3
votes
1answer
81 views

Correspondence between two different depression scales?

I have become involved in a project where psychiatrists are studying the correspondence between two scales measuring the level of depression among a specific patient group. One of the scales is for ...
4
votes
1answer
8k views

Minimum “recommended” sample size for boxplots? Boxplots for different sample sizes

What is the minimum "recommended" sample size to generate boxplots? If I'm comparing different methods and each method has a different sample size, is it fine to use boxplots for this comparison? If ...
3
votes
2answers
2k views

SciPy chisquare applied on continuous data?

chi2 seems to be used to compare distributions between categorical features. Is it possible to use SciPy chisquare function to compare two distributions of continuous data ? For example : ...