The model-comparison tag has no wiki summary.
1
vote
1answer
11 views
comparing AIC (or BIC or whatever) between different SETS of models
Suppose I have $m$ competing models. Suppose also that I could classify these models into $s$ sets. For example, I could classify models of migration behavior conditional on climatic conditions by the ...
0
votes
0answers
36 views
How to statistically compare two fibonacci algorithms?
I have two Fibonacci recursive algorithms and I want to check the efficiency comparing them with each other.
Does anyone know how to do it statistically?
Here's my data.
Recursive_1:
Recursive_2 ...
0
votes
0answers
26 views
Compare the predictive power of a model between datasets
I have two sets of continuous response data for the same group of species, but in different areas (area a and area b). I am building a model for each area separately, to predict the area-specific ...
0
votes
2answers
156 views
Regression and simple SEM: identification and model comparison
I have two problems I am trying to solve. I am using MPLUS, but I have very general questions for which MPLUS knowledge will not be necessary.
Terminology: y = DV, x = IV
(1) Simple regression ...
0
votes
0answers
53 views
Get the percentage of two data sets compability
I'm new here and I don't have special background on math / stat. I'm a computer programmer and need to create a program that involved statistic.
I want to search the closest of two set data, which is ...
2
votes
1answer
55 views
Comparison of two simulated curves for given experimental data
Physicist here with a naive question,
I've got a data set (points $x_i,y(x_i),i\in \mathbb N$) and two different smooth curves $f(x),g(x)$ from my simulation. I can't quite judge with my eye which ...
0
votes
0answers
49 views
How to compare Arellano-Bond models?
I want to compare different Arellano-Bond models, but I don't know exactly how to do it and which statistics to consider. For instance, should I compare $\chi^2$ of Sargan tests? What other options ...
0
votes
0answers
77 views
Model comparison when models differ in complexity and the data used fit them
Let's say I have two models and two data sets. Lets call them $M_1$, $M_2$ and $D_1$, $D_2$. We fit $M_1$ to $D_1$ and $M_2$ to $D_2$.
Now I would like to compare how well they did. When doing that I ...
3
votes
2answers
223 views
Prerequisites for AIC model comparison
What are exactly the prerequisites, that need to be fulfilled for AIC model comparison to work?
I just came around this question when I did comparison like this:
...
0
votes
0answers
76 views
What's the difference between AUC and R square
I understand the meaning of AUC and R-square. Either one is a good statisic for evaluating models. But which is better? If I comparing two models A and B, A has higher R square but B has higher AUC, ...
2
votes
1answer
56 views
Comparing two Bayesian models
To compare two Bayesian models for the same data, I plotted the fitted values of the first model versus the fitted values of the second model. The plot indicates a perfect similarity.
Relying on this ...
0
votes
0answers
119 views
How to compare a frequentist model with a Bayesian one
I analyzed my data by using maximum likelihood estimation and a Bayesian approach. Now, I want to see which model has a better fit. How could I do this using either plots or numbers? Please be ...
2
votes
0answers
140 views
When to use likelihood ratio test vs. incremental F-test in nested nonlinear model selection?
I have two nested nonlinear models and I want to know which provides a better fit to some data. I see descriptions of both the likelihood ratio test and of the incremental F-test (also called the ...
0
votes
0answers
83 views
Comparing Algorithms using AUC, Effect Size, or Box-plot?
I want to compare the performance of two algorithms (one semi-supervised and the other one is an unsupervised algorithm like SVM or Naive Bayes) based on Hedges' effect size calculation).
For this ...
0
votes
0answers
70 views
Comparing multiple regression models on different dimensions
I have a linear regression model with IVs=A+B+C+D
And DV=E
I also have a model with IVs=A+B+C+D
and DV=F
F is composed of observations from a subpopulation of E (Fn+~Fn=En)
For this comparison, ...
0
votes
0answers
48 views
Comparing model adequacy across different dependent variables
I have a three-level factor that I want to model as a response. I can collapse this into a two-level factor and build a logistic regression model. I can also keep the three-way coding and build a ...
1
vote
1answer
217 views
How to calculate absolute fit indices (RMSEA, GFI…) from relative ones (AIC, BIC…)?
I have conducted an IRT analysis with Conquest in order to compare two models (1-dimensional vs. 8-dimensional) applied to a given data set (41 items of a questionnaire, N=195).
Comparing the ...
0
votes
0answers
99 views
Comparing multiple models using the same explanatory variables [closed]
I have 5 GAMs, each one using a subset of my data, defined by quantiles. The response/explanitory variables are the same in each case, but the deviance explained by each in each model differs, along ...
6
votes
3answers
335 views
Similarity of two discrete fourier tranforms?
In climate modelling, you're looking for a models that can adequately portray the Earth's climate. This includes showing patterns that are semi-cyclical: things like the El Nino Southern Oscillation. ...
0
votes
1answer
123 views
Group comparison: joint model vs. separate model, contradictory result
Any hint on this issue would be really welcome: The goal is to employ a two-country comparison analysis. When running two separate OLS regressions, one for each country, the coefficient of variable x ...
3
votes
0answers
106 views
Comparing model fit with heteroscedastic data
I am developing a physiological test using R that requires some parameters optimised. In comparing the new method against the existing method, the values of individual readings correlate in a linear ...
2
votes
0answers
117 views
Open source data mining package comparsion
Is there sources where I can see performance of different ML models applied to same data?
Currently I want to understand performance percision/recall difference between mahout online logistic ...
1
vote
2answers
279 views
Impact of regression normality-assumption on model comparison & prediction?
This question is a continuation of the discussion here:
How to test the statistical significance for categorical variable in linear regression?
Following Macro's suggestion, I started a new thread.
...
1
vote
0answers
109 views
How to compare the performence between different models?
Given a data, I'd like to utilize it by two different models, such as generalized linear model (GLM) and generalized partial linear model (GPLM), and my problem is how to compare the performance of ...
1
vote
1answer
306 views
Modification of "corrected repeated k-fold cv test” when also averaging Random Forest results across multiple sampling seeds?
I would be very grateful for any ideas concerning the following problem. I would be even more grateful if someone could point me to a literature reference proposing a solution for a scenario like the ...
3
votes
0answers
89 views
Comparing two non-linear biological models: differences in y vs. differences in slope
I'm an ecology PhD student, but unfortunately not a very mathematically sophisticated one. I am interested in ways to compare two non-linear models. I was really impressed with this answer, which ...
2
votes
0answers
201 views
How do I compare 2 models?
I would like to show that a new biomarker (let's say percent tumor volume reduction at 3 weeks of treatment estimated with automatic methods) performs better than an old one (same measure, but based ...
2
votes
1answer
360 views
Comparing logistic regressions
What is an appropriate way to test whether two logistic regressions are significantly different from one another?
Essentially, I have two similar logistic regressions made from two different sets of ...
2
votes
1answer
140 views
Logical extensions after logistic regression
I have a number of predictors to use for a binary (Classes 0 and 1) classification task. Let us call them $x_1, x_2, x_3, ... x_n$. The way these are calculated, my naive heuristic assumption is that ...
7
votes
1answer
121 views
How can I compare my model to a technically invalid model?
I've created nice little nonlinear model relating survival probability to length in salmon. I fit it assuming binomial errors and minimizing the negative log likelihood. I've been asked to compare it ...
3
votes
3answers
900 views
Can you compare p-values of Kolmogorov Smirnov tests of normality of two variables to say which is more normal?
I have applied the one sample Kolmogorov Smirnov test of normality to two variables and one has a larger p value but both are greater than .05.
e.g.,
$x_1$ (p-value) = 0.09
$x_2$ (p-value) = 0.06
...
4
votes
0answers
578 views
Comparison of positive predictive value between two models
Does anybody know how to compare two positive predictive value (PPV) from two different predictive models? I found
some papers reporting $\chi^2$-test or Fisher exact test but I can´t figure out how ...
4
votes
2answers
512 views
Number of parameters in Markov model
I want to use BIC for HMM model selection:
BIC = -2*logLike + num_of_params * log(num_of_data)
So how do I count the number of parameters in the HMM model. ...
2
votes
3answers
1k views
Model comparison between an ARIMA model and a regression model
I'm really having trouble finding out how to compare ARIMA and regression models. I understand how to evaluate ARIMA models against each other, and different types of regression models (ie: ...
5
votes
1answer
900 views
Conducting planned comparisons in mixed model using lmer
When deconstructing my mixed effects model, I found a three-way significant interaction. I calculated my p-value by using maximum likelihood ratio tests allowing for a comparison of the fit of the two ...
8
votes
2answers
774 views
Comparing regression models
I recently fit 4 multiple regression models for the same predictor/response data. Two of the models I fit with Poisson regression.
...
13
votes
3answers
712 views
Pitfalls of linear mixed models
What are some of the main pitfalls of using linear mixed-effects models? What are the most important things to test/watch out for in assessing the appropriateness of your model? When comparing models ...
4
votes
1answer
93 views
Rating how closely one graph models another
I'm using a genetic algorithm to generate a string that produces certain results I map into a line/bar graph. I'm trying to rate how closely the results produced by the genetic string compares to a ...
6
votes
1answer
95 views
How should I objectively test my program results?
I have developed two differing methods in MATLAB which aim to analyse a pop song and then automatically create a 30 second audio thumbnail (a preview clip) containing part of the chorus section.
Both ...
10
votes
5answers
642 views
When to use multiple models for prediction?
This is a fairly general question:
I have typically found that using multiple different models outperforms one model when trying to predict a time series out of sample. Are there any good papers ...
