Tagged Questions

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

0
votes
0answers
4 views

AIC or similar selection techniques for Variograms?

I have a very basic question: how does one choose the "best" variogram? It is possible to fit different models to an empirical variogram, e.g. nugget, ...
1
vote
1answer
34 views

Which p-value to report in a comparison of different logistic regression models using marginal effects?

I am running logistic regression models to compare the impact of different indicators using Stata. As these comparisons may lead to false conclusion due to confounding and rescaling if log-odds or ...
1
vote
1answer
49 views

Simple model selection example in PYMC

I am currently experimenting with PYMC and I am trying out a simple example so that I start learning how things work (I am also a Python beginner, but an experienced machine learner). I have set ...
0
votes
0answers
6 views

How to specify that the model must include a particular independent variable in r package glmulti

I am trying to find the best model(lowest AICc value) using the package glmulti in r. This model must include TVIS as an independent variable but I am unsure how to specify this in the script. ...
1
vote
1answer
25 views

How to correlate categorical personality and music genre preference scores?

I'm currently a third year Biology student and I've annoyingly screwed myself over by not following the golden rule of stats, always know how to analyze your data prior to conducting the experiment. ...
0
votes
0answers
20 views

Model averaging with MuMIn. What's the mean of pvalue?

the summary() of a model.avg made with MuMIn in R, give a lot of interesting results, in particular model averaged coefficients (estimate, standard error, adjusted standard error and a z value with a ...
1
vote
0answers
22 views

Fitting a model to data for prediction - best choice for data

I have some data I need to fit a model to that can be used for prediction (interpolation). The data is summarized by the plot below. The black line is x=y. I want to be able to fit a model so as I ...
2
votes
1answer
47 views

c-index for parametric links in binary regression

I am conducting a binary regression using different sorts of parametric links (logistic, Pregibon, Aranda-Ordaz, ... see) and I would like to compare their predictive and classification perfomance in ...
0
votes
0answers
3 views

How to account for different ratio of samples during training and detection using a support vector machine (svm)?

Consider the following object recognition case: Detection of objects in an image using a sliding window approach in combination with a svm model. During sliding window search using multiple scale ...
0
votes
0answers
12 views

How to find a good model for an object recognition case using a support vector machine (svm)?

Consider the following example of an object recognition case: I'm trying to detect objects in an image using histograms of oriented gradients (hog) features. The feature vector resulting from hog is ...
2
votes
2answers
43 views

Using K-fold cross validation to select a model's parameters

I think I understand completely the concept of cross validation, but there is one aspect I've never seen detailed. Let's assume I have a logistic regression model with four parameters I want to train. ...
0
votes
0answers
5 views

Relative variable importance for simple model set

I am evaluating models based on AIC. I started by running the simplest models and the dot model (no covariates) is the best model, with little support for any others. When reporting the relative ...
2
votes
1answer
43 views

Correct numerical result in Bayesian model comparison

I was wondering how to calculate the following Bayesian model comparison. Suppose you have a couple of models: $$M_{1}: x \sim Bin(n, \pi); p \sim Be(1,1)$$ $$M_{2}: x \sim Bin(n, \pi); p \sim ...
0
votes
0answers
4 views

multimodel inference when using rms package

I would be glad to have some advise about how to proceed with multimodel inference to obtain weighted estimates based on AICc after running ordinal logistic analyzes with "rms' package. I used the ...
2
votes
0answers
27 views

How to determine which covariates to use

This may seem like a basic question but here goes... I am looking at the effect of brain stimulation on skill acquisition across several timepoints. I have several measures that may be useful as ...
1
vote
0answers
26 views

Cross validation with unequal sample size for the left out sets

I am trying to do cross validation on several (20) subsets of samples, which all have unequal sample size. I cannot subsample so that sizes are equal. Example: batch 1: 500 samples batch 2: 400 ...
0
votes
1answer
12 views

How to approach hurdle models with multiple covariates in R

I have count data that I standardized into continuous data (density) because the area surveyed varied among sites. I have several sites in which the count was zero. The probability of observing a ...
9
votes
4answers
446 views

Model building and selection using Hosmer et al. 2013. Applied Logistic Regression in R

This is my first post on StackExchange, but I have been using it as a resource for quite a while, I will do my best to use the appropriate format and make the appropriate edits. Also, this is a ...
0
votes
1answer
27 views

Goodness of fit: observed vs simulated data

I have a set of 2-dimensional "observed" data of sample size N: $$O = \{(x_1, y_1), (x_2, y_2), ..., (x_N, y_N)\}$$ The hypothesis is that $O$ is a realization of ...
0
votes
0answers
9 views

Select final model after many Dev-Val splits

We have a dataset that is quite heterogeneous, and we established that different Dev-Val splits affect the outcome quite bit, even after using stratified sampling. So what we did write a loop around ...
2
votes
1answer
85 views

How to prove predictive use of a biomarker?

I have a binary endpoint (cured/not cured) and a continuous biomarker measured on each patient. Every patient recieved one of two treatments. The biomarker predicts the effect of the treatment, if ...
0
votes
0answers
14 views

Is there “infinite” universal model selection ? and Structural Risk Minimization

I ask these because I come up with an idea : If I have infinite and universal model set, then there must exist model that totally fits my data and no parameter for the model so the complexity ...
0
votes
0answers
13 views

model post-event misinformation effect on eyewitness?

I'm trying to model post event misinformation. The questions i have is the factors affecting it. and a psychological way of assessing it.
1
vote
0answers
17 views

How does one adjust for data snooping when using ACF and PACF?

ACF and PACF are routinely used for approximate identification of a time series model, e.g. as described here. Say, one takes a look at the plots and guesses that it's something like ...
0
votes
0answers
10 views

analysis strategy for selecting and/or transforming correlated continuous biomarkers to predict binary endpoint

I am given a simulation task to come up with several analysis strategies and compare their relative performances. The horizon is wide open; I appreciate all recommendations of methods and references. ...
0
votes
4answers
175 views

Can predictive power be inferred from only in-sample modelling results?

I wonder if one can tell anything about predictive power of a model if model selection and estimation was done using all available data. That is, there was no data left for "out of sample" prediction ...
0
votes
0answers
35 views

Generate mixture model from data with features

I want to build a mixture model from my data, but using features of my data to calculate each component in the model. The data: For each point I have 34 associated features. Each feature is a boolean ...
0
votes
0answers
81 views

Model selection using an artificially insignificant covariate

This is continued from my other post on model selection. Let me provide more details first. 1) I have a factorial design. Factor A has 5 levels, B has 2 levels, C has 2 levels. Let us assume that ...
1
vote
0answers
8 views

Selecting optimal model when only smoothed data available?

I have a graph of some (highly nonlinear) experimental spectrum which is obtained by smoothing results of several repeated measurements obtained by different experimental methods. The graph also ...
1
vote
0answers
22 views

Follow-on to “Training with the full dataset after cross-validation” - sequential parameter estimation

Background: Here is the background for the question, both the question itself and the answer given by Dikran Marsupial. Training with the full dataset after cross-validation? It asks about after ...
0
votes
0answers
22 views

How to develop a robust procedure to select a predictive model

Imagine you have a matrix, M, of n input variables and m values per variable. There's also a ...
4
votes
2answers
180 views

Should train and test datasets have similar variance?

If variance of test dataset is lower than the one of the train dataset is it worth splitting the data? Since we know our dataset will always be limited is it fair to select models under the above ...
0
votes
0answers
8 views

Compressed Population Complexity in Minimum Description Length (MDL)

I am studying the MDL and found it is sum of model complexity and compressed population complexity. To my understanding, model complexity refers to number of bits to encode the model, which can be ...
2
votes
0answers
15 views

Posterior Predictive Checks

I understand what the posterior predictive distribution is, and I have been reading about posterior predictive checks, although it isn't clear to me what it does yet. What exactly is the posterior ...
1
vote
1answer
28 views

Empirical Bayes vs “non-informative” priors

I am familiar with the mechanics with both methods, but don't know what factors I should consider when choosing between these two approaches for adjusting a prior. I would imagine that, on a case by ...
1
vote
1answer
24 views

model comparison when alternatives are not all nested within one another

I am running a glmm with three fixed effects: opponent 1 size ("1") opponent 2 size ("2") opponent 1 size - opponent 2 size ("diff") I am unable to run all three variables in the model at once ...
0
votes
0answers
19 views

Determine which variable or variables is/are the most efficient to predict the outcome

I have a small dataset (n=74) with a +/- 50 variables, not the best data but I have to work with it. The variables are used to select a product. I want to determine which variable or variables is/are ...
0
votes
2answers
42 views

Logistic regression, Chi-square, and study design

I have a study in which I have developed a new predictor (binary) for a disease (also a binary variable). The study has two parts. In the first part, I want to test if my predictor is strongly ...
3
votes
1answer
69 views

Model selection: where to start? [closed]

For a general modeling problem, there are literally at least a dozen choices of statistical and algorithmic models to choose from. Off the top of my head, choices could be: regression (and its ...
1
vote
1answer
40 views

Comparing two models

I am interested in comparing two logistic regression models. The two models are nested: model 1 contains all predictors, and model 2 contains all predictors except 1. My goal is to test if removing ...
0
votes
0answers
30 views

selecting a link function for GLM's

If you don't care about using GLM model parameters to predict anything, but simply want to select the best-fitting model for your data, is it necessary to get into the theoretical debate as to which ...
1
vote
0answers
30 views

the role of AIC versus p-values in model selection

Let's say you are trying to choose between two models. One has two significant fixed effects. The other includes only one of the two fixed effects from the aforementioned model but has a lower AIC ...
1
vote
0answers
33 views

Model selection using mean AIC for very huge data sets

I want to select a model which best performs for a very huge data set. However, the data set is too large to calculate a model within reasonable time. If this is the case, is the following a ...
3
votes
1answer
120 views

How to choose between exponential and gamma distributions

I have same data and I would like to choose a model for it. To start with I fit an exponential distribution and a gamma distribution. Now I wanted to do a simple likelihood ratio test . However, I ...
0
votes
0answers
2 views

True discovery rate or intrinsic sensitivity?

I'm trying to evaluate my species distribution model by the rate of discovered\predicted localities. For example, my model at a stated threshold predicts 50 grid suitable cells. Then, after field ...
0
votes
0answers
44 views

GEE in SPSS: the case of ever-decreasing QICc values

I've been using GEEs in SPSS 22 to analyze my dissertation data and have discovered an interesting problem: when trying to figure out which subset of model factors have the lowest QICc, and therefore ...
0
votes
1answer
26 views

Determining if time has influence on correlation between x and y

Is there a sort of statistics model I could use to determine if time has an influence on the correlation between two sets of data? For example say I have a column $X$ and a column $Y$, both just ...
1
vote
2answers
114 views

Why is Lasso regression for high dimensional data better than Stepwise AIC?

I know Lasso eventually set some parameters to zero, acting like variable selection. I also read from paper talking about automated variable selection method like Stepwise AIC can be troublesome. So ...
0
votes
0answers
27 views

How to deal with categorical features.

Recently I am playing in the famous Big-Data website Kaggle. There is a Display Advertising Challenge. In this competition, you are giving a training file which include huge records. the records is ...
0
votes
0answers
24 views

Why is $R^2$ poor for AR model selection used for forecasting?

There is a related question here, about how to calculate the R-squared on a regression with ARIMA errors. I found the answer quite useful, and hoped for some elaboration, particularly on Rob's closing ...