Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

Filter by
Sorted by
Tagged with
15
votes
4answers
3k views

Comparing mixed effect models with the same number of degrees of freedom

I have an experiment that I'll try to abstract here. Imagine I toss three white stones in front of you and ask you to make a judgment about their position. I record a variety of properties of the ...
17
votes
7answers
2k views

What is the definition of “best” as used in the term “best fit” and cross validation?

If you fit a non linear function to a set of points (assuming there is only one ordinate for each abscissa) the result can either be: a very complex function with small residuals a very simple ...
20
votes
7answers
13k views

Measures of model complexity

How can we compare complexity of two models with the same number of parameters? Edit 09/19: To clarify, model complexity is a measure of how hard it is to learn from limited data. When two models fit ...
6
votes
3answers
3k views

Algorithm for choosing the number of clusters when using pam in R?

I am clustering a dataset using the pam command (from {cluster} package), and I wish to decide on the number of clusters to use. I was able to implement The_Elbow_Method in R (see wiki) for doing ...
4
votes
2answers
142 views

How good is my error? [closed]

I'm trying to calculate how good are my measurements in machine learning! Let's say that I have five choices, and that error is 4, 2, 0.002, 3, 6. Naturally, I will pick third one for the hit, but I ...
2
votes
1answer
2k views

Panel data and selection models issue [closed]

I'm working with a panel dataset, I've used many models, homogeneous (fixed effect, pooled ols and Driscoll and Kraay) heterogeneous (swamy random coefficients) and would like to do a post-...
2
votes
3answers
418 views

How to model and test a decision support system (e.g. a terrorist warning system)?

I have reformulated the problem from a "dog barking warning system" to something else which hopefully, has less ambiguity. Instead, I will repose the problem as follows: Let's assume that my ...
24
votes
5answers
3k views

When are Shao's results on leave-one-out cross-validation applicable?

In his paper Linear Model Selection by Cross-Validation, Jun Shao shows that for the problem of variable selection in multivariate linear regression, the method of leave-one-out cross validation (...
10
votes
3answers
2k views

Computing best subset of predictors for linear regression

For the selection of predictors in multivariate linear regression with $p$ suitable predictors, what methods are available to find an 'optimal' subset of the predictors without explicitly testing all $...
13
votes
3answers
675 views

Using information geometry to define distances and volumes…useful?

I came across a large body of literature which advocates using Fisher's Information metric as a natural local metric in the space of probability distributions and then integrating over it to define ...
85
votes
14answers
6k views

Why haven't robust (and resistant) statistics replaced classical techniques?

When solving business problems using data, it's common that at least one key assumption that under-pins classical statistics is invalid. Most of the time, no one bothers to check those assumptions so ...
4
votes
3answers
999 views

What is the interpretation/meaning of confidence intervals in misspecified models?

Consider the following model $Y_i = f(X_i) + e_i$ from which we observe n iid data points $\left( X_i, Y_i \right)_{i=1}^n$. Suppose that $X_i \in \mathbb{R}^d$ is a $d$ dimensional feature vector. ...
5
votes
4answers
3k views

Incorporating boolean data into analysis

I have a data set of about 3,000 field observations. The data collected is divided into 20 variables (real numbers), 30 boolean variables, and 10 or so look up variables and one "answer" variable ...
251
votes
12answers
174k views

Is there any reason to prefer the AIC or BIC over the other?

The AIC and BIC are both methods of assessing model fit penalized for the number of estimated parameters. As I understand it, BIC penalizes models more for free parameters than does AIC. Beyond a ...
43
votes
5answers
85k views

Negative values for AICc (corrected Akaike Information Criterion)

I have calculated AIC and AICc to compare two general linear mixed models; The AICs are positive with model 1 having a lower AIC than model 2. However, the values for AICc are both negative (model 1 ...
679
votes
11answers
792k views

How to choose the number of hidden layers and nodes in a feedforward neural network?

Is there a standard and accepted method for selecting the number of layers, and the number of nodes in each layer, in a feed-forward neural network? I'm interested in automated ways of building neural ...

1
31 32 33 34
35