Skip to main content
13 votes
Accepted

What is the difference between MLP and RBF?

MLP: uses dot products (between inputs and weights) and sigmoidal activation functions (or other monotonic functions such as ReLU) and training is usually done through backpropagation for all layers (...
rcpinto's user avatar
  • 1,703
13 votes
Accepted

What is the relationship between ANOVA to compare means of several groups and ANOVA to compare nested models?

In my understanding, the abstract intuition of ANOVA is the following: One decomposes the sources of variance of the observed variable in various directions and investigates the respective ...
bmbb's user avatar
  • 308
13 votes

What if there is no true data-generating process?

Have you heard the "all models are wrong, but some are useful" quote? It's one o the most famous quotes in statistics. Let's use human language as an example. What you say, is a result of ...
Tim's user avatar
  • 139k
12 votes

Is it wrong to compare multiple models on the same test set and choose the best model?

You could consider the model itself a hyperparameter as well. If you optimize the hyperparameter using the test set, and then choose the best model, you overfit with the human in the loop. I like the ...
Nikolas Rieble's user avatar
10 votes

What if there is no true data-generating process?

Looking at it the other way, if there were no true data generating process, how did the data get generated? The inability of standard estimating techniques to accurately approximate the true data-...
Dikran Marsupial's user avatar
9 votes

Prerequisites for AIC model comparison

Transformation of Variable In the analysis of time series it is common to try some kind of transformation on the variable. The decision on the choice of the transformation can be realized very simply ...
bjd's user avatar
  • 91
9 votes
Accepted

Mixed-effect model single term deletion -- should I change my random effects?

As a rule for lme4 and other packages with a similar parameterization (at least at the level of the user interface), it does not make sense to have random slopes ...
Livius's user avatar
  • 2,146
8 votes
Accepted

Is overfitted model with higher AUC on test sample better than not overfitted one

This will depend on how your training and test sets are composed. If the test set is rather big and reflects the "application case" data diversity correctly, I would not argue like this. But if the ...
geekoverdose's user avatar
  • 3,901
8 votes
Accepted

Should I cross-validate metrics that were not optimised?

It is a good idea to bootstrap or cross-validate (e.g., 100 repeats of 10-fold cross-validation) indexes that were not optimized. For example, I recommend optimizing on a gold standard such as log-...
Frank Harrell's user avatar
8 votes

How to compare fitted survival model with covariates vs. Kaplan-Meier?

You can calculate the residuals from the accelerated failure time model fitted by survreg() and compare them with the assumed distribution. You will need to account ...
Dimitris Rizopoulos's user avatar
8 votes
Accepted

why are models compared over multiple datasets?

When we study a classification, prediction, forecasting or any other method, we are typically less interested in whether it works well on one specific dataset. Rather, we want to understand whether it ...
Stephan Kolassa's user avatar
8 votes
Accepted

How to determine the best fitted model by AIC between lm(y~x),lm(log(y)~x), drc(y~x) in R

You can't compare AIC between a model fitted to the original data and another model fitted to logarithmized data. In addition, Be extremely careful about comparing AICs between models fitted using ...
Stephan Kolassa's user avatar
7 votes

What is the relationship between ANOVA to compare means of several groups and ANOVA to compare nested models?

If you are doing one-way ANOVA to test if there is a significant difference between groups, then implicitly you are comparing two nested models (so there is only one level of nesting, but it is still ...
Sextus Empiricus's user avatar
7 votes
Accepted

How to test significance in shift of a variable taken an other variable into the model (suest) in R?

I was able to replicate Stata's suest using geepack in R. I based my replication on: Zhuan ...
Jonathan's user avatar
  • 603
7 votes
Accepted

How is the relationship between two variables $X$ and $Y$ supposed to "explain" $R^2\text%$ of the variation of the data?

It's an expression that is often used as short-hand or conventional jargon. Anyone who finds it puzzling should feel that way! Some people say "accounts for" instead as a usage supposedly a ...
Nick Cox's user avatar
  • 57.5k
7 votes
Accepted

Why do you need non-linear regression if you can use a linear one to fit any kind of curvature to your data?

Model Parsimony If you have a sine curve, you can approximate it to arbitrary accuracy with its series expansion. I’d probably rather estimate the two parameters of $\mathbb E[y]= A\sin(Bx)$ than the ...
Dave's user avatar
  • 63.7k
7 votes
Accepted

Compare GLM AICs with different likelihoods?

The Aikake information criterion (AIC) is derived by minimizing the Kullback–Leibler (KL) divergence between the data-generating distribution $g(y)$ and an approximating model $f_\theta(y)$ with ...
dipetkov's user avatar
  • 10.2k
7 votes
Accepted

When dealing with data imbalance, shouldn't we never compare models based on validation loss, or at least weight it?

You should use a loss that accurately reflects the "real world loss" you are trying to minimize by using your model (in the context of subsequent decisions). Then the "problem" ...
Stephan Kolassa's user avatar
7 votes

Comparing models with main effects and interactions

Ignore "significance". Pre-specify the full model, heavily informed by subject matter knowledge. Get point and interval estimates of interest through pre-specified contrasts. Use ...
Frank Harrell's user avatar
6 votes
Accepted

Model comparison with AIC based on different sample size

All the information criteria are based on likelihood function, which in its turn depends on sample size. The larger sample size is, the smaller likelihood becomes and as a result the greater IC ...
Ivan Svetunkov's user avatar
6 votes

Is overfitted model with higher AUC on test sample better than not overfitted one

If the focus is purely on predictive accuracy, then the overfitted model is most probably better. Take e.g. a random forest: On the training data set, by construction, it extremely overfits. Still the ...
Michael M's user avatar
  • 11.8k
6 votes
Accepted

Over-parameterization in Bayesian Hierarchical Model

Important to note first: Bayesian inference does NOT automatically guard against overfitting. Adding additional variables will pretty much result in the same problems as in an non-Bayesian analysis. ...
Florian Hartig's user avatar
6 votes
Accepted

AIC/BIC for a segmented regression model?

In principle yes. If you know the segments (i.e., if they are given exogenously), then the segmented regression model just corresponds to a certain kind of interaction model and information criteria ...
Achim Zeileis's user avatar
6 votes

Logistic regression BIC: what's the right N?

The BIC (and the AIC) are relative measures for comparing models. However, it makes no sense to compare what is otherwise the same model between using an aggregated vs. a disaggregated response. Nor ...
gung - Reinstate Monica's user avatar
6 votes

How is the relationship between two variables $X$ and $Y$ supposed to "explain" $R^2\text%$ of the variation of the data?

Say you fit the model: $$ Y_i=\beta_0+\beta_1X_i $$ and you get: $$ R^2=0.81 $$ This means that your independent variable $X$ accounts for 81% of the variability in your dependent variable $Y$. Or, in ...
Adrià Luz's user avatar
  • 1,024
6 votes

Valid model comparison/selection? Poisson, negative binomial, zero-inflated poisson, z-i negative binomial with AIC

By doing model selection before doing inference on the selected model, you are distorting your final inferences. Any inference you do on the final model, say calculating a confidence interval on a ...
Dave's user avatar
  • 63.7k
5 votes

How do I compare the performance of random forests for regression?

I like the @hxd1011 answer, and this is only to expand on it slightly. Here is my code: ...
EngrStudent's user avatar
  • 9,430
5 votes

Tests of Forecast Accuracy for Nested Models

The Diebold-Mariano test is well suited for the purpose it was designed for: to assess whether two forecasts produce loss of equal size in population. This holds regardless of the models that were ...
Richard Hardy's user avatar
5 votes
Accepted

Can one give an example(s) of when non-nested AIC model comparison is not useful for model selection?

Nesting is when all of the models tested can be derived by eliminating parameters from a parent model. Non-nesting is when the models contain parameters that are not in a set with subset(s) format. ...
Carl's user avatar
  • 13.2k
5 votes
Accepted

Specifying multilevel model structure when random effects exhaust the population

The idea that the clusters are drawn from a population is one story we tell ourselves. Another story is that the random effects approach simply allows us to estimate coefficients for each cluster in ...
Heteroskedastic Jim's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible