Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

0
votes
0answers
25 views

Calculating Log-Likelihood from Simulated Distribution

I want to perform some sort of model evaluation of a multivariate distribution with the property that it is difficult/impossible to calculate the likelihood (of the whole model, you can do it for ...
0
votes
1answer
24 views

Machine Learning approach to solve this selection problem?

I have a set of items S. items can be joined to groups consisting of up to x items. For a group of items i can derive a score Y using some unknown performance measure. What would be the most efficient ...
1
vote
1answer
33 views

Model uncertainty (model averaging) and R-Squared (R2)

Is it possible to calculate r-squared for an "average model"? Lets say I have 4 different response variables that I want to model to a set (or subset) of 4 independent variables. I'd then like to ...
8
votes
2answers
268 views

Best suggested textbooks on Bootstrap resampling?

I just wanted to ask which are in your opinion the best available books on bootstrap out there. By this I don't necessarily only mean the one written by its developers. Could you please indicate ...
1
vote
2answers
75 views

What model should I use for this?

Consider a dataset with 30 samples. We have a response $Y$ and 12 potential predictors $X_1, \cdots, X_{12}$. We fit two models. The first model $M_1$ includes only $X_1$ and $X_2$ and has an RSS of ...
0
votes
0answers
28 views

How to evaluate variable contribution to a prediction

I need to produce a logistic regression model that: Gives a ranked list of the most important factors Allows you to break out most important factors for each new observation scored Given that I'm ...
0
votes
0answers
11 views

Dummy and Heckman [migrated]

I'm using Heckman Selection Model which are two consist of 2 equation. i'm using Probit as a selection equation and multiple regression as a result equation. how can put in dummy variables in those ...
1
vote
1answer
160 views

what if response variable is 'yes or no' in R?

How to analyze above the data to predict the probability that people have disease with a model? Factors thought to influence infection include city, age, and diet. BUT, I don't know how to do ...
4
votes
1answer
101 views

Justifying and choosing a proper scoring rule

Most resources on proper scoring rules mention a number of different scoring rules like log-loss, Brier score or spherical scoring. However, they often don't give much guidance on the differences ...
0
votes
1answer
33 views

What is the correct order of these hierarchical priors?

I'm quite new to Bayesian data analysis, so this is most likely am easy question. I have the following model: a function f has two exponential rate parameters $\lambda_1$ and $\lambda_2$ and for some ...
0
votes
1answer
45 views

Structural equation modelling: model selection

I am currently trying to fit a structural equation model in R with the Lavaan package. I have this model that fits my data pretty good. This model is what I consider the full model, it has all paths ...
0
votes
0answers
24 views

time series rolling cross-validation for parameter selection and model comparison

I want to do two things using rolling cross-validation for time series (as in the famous Hyndman's post): select parameters for model A, and compare it's predictive performance with a model B. I'm ...
1
vote
0answers
24 views

Comparing a model with a latent variable to one without

I would like to test whether my 3 dependent variables all load onto an underlying latent variable or if a latent variable is not necessary to explain the relationship between the IVs and the DVs. In ...
3
votes
2answers
50 views

How to compare predictions from MLE-based regression Vs. predictions from bayesian regression?

Say I have two linear regression models that I want to use for predictions. Linear regression: \begin{equation} \mathbf{y} \sim \mathcal{N}(\mathbf{X^Tb}, \Sigma_y) \end{equation} Bayesian linear ...
0
votes
0answers
38 views

AIC rankings: Why would a global model rank lower than an intercept-only model?

I'm working with some real-world (i.e. potentially messy) data on the nesting ecology of several bird species. I'm attempting to relate the daily survival rate of nests to vegetation characteristics ...
1
vote
0answers
24 views

$R^2$ for mixed models = ICC?

I will be referring here to Nakagawa and Schielzeth (2013). As those authors state, $R^2$ for OLS regression could be defined as follows: $$R^2 = \frac{\sum^n_{i=1}(\bar{y} - ...
0
votes
0answers
10 views

Averaging against mixed-effect model

I have experimental data that contains information about 50 participants who performed a task in five different conditions (different set sizes). The result is the time spent on the task. My data is ...
0
votes
0answers
13 views

With posterior inclusion probability how do I settle on the final predictive model?

After using the spike-and-slab prior to perform Bayesian model selection, I get the posterior distribution of my variables, from which I calculate the inclusion probability for each variable. How do ...
1
vote
0answers
29 views

Selecting an appropriate VAR model

I would like to receive critical comments on an idea explained below. Suppose I have variables $x_1$ through $x_K$, and this is a time series setting. My aim is to forecast variable $x_1$. I know ...
0
votes
0answers
44 views

Advice for feature selection or feature extraction with semi-supervised learning

I am trying to solve a semi-supervised learning problem using LaplacianSVM. However, before applying LapSVM I would like either to perform feature selection or feature extraction. Furthermore, after ...
3
votes
0answers
48 views

Explain log likelihood behaviour

(This question is related to a previous one I made, here) I have a set of 2D observations (measured data) of sample size $N$: $$O = \{(x_1, y_1), (x_2, y_2), ..., (x_N, y_N)\}$$ I also have a model ...
1
vote
1answer
66 views

Multiple Linear Regression Variable Selection

Using all possible subsets we consider the adjusted $R^2$, Akaike's Information Criterion (AIC), corrected AIC ($AIC_c$), and Bayesian Information Criterion. The model with the highest adjusted $R^2), ...
1
vote
0answers
12 views

Heckman's selection / switching model- selection variable as a regressor

I am conducting some analyses on the effects of membership and the membership rank on income. Everyone in the sample can choose to apply to join an organization. The membership is given upon ...
7
votes
2answers
145 views

How to summarize knowledge about importance of variables?

Assume that we have 30 features (inputs) each of which can potentially influence the result (output). We try to use the available ("observed") mapping from features to targets to develop a predictive ...
0
votes
1answer
39 views

Good Literature about Problems with R squared

A question from a newbie. Recently, I was told that R squared or adjusted R squared can not used as a criteria to select a good regression model (model selection) due to, for example, overfitting . I ...
0
votes
2answers
51 views

Multiple linear regression, backward selection : Normality of the residuals?

I need to create a Multiple Linear regression model on those data explaining max03 T9 T12 T15 Ne9 Ne12 Ne15 Vx9 Vx12 Vx15 maxO3v !My data 1 My first intuition was to make a backward selection : ...
0
votes
0answers
32 views

Akaike information criterion for Cox proportional hazard models

I am conducting an analysis of survival data using Cox proportional hazard (CPH) models, to figure out what is the best model to use. The models I am comparing are non-nested. My plan is to compute ...
1
vote
0answers
17 views

Confidence intervals for the Log Loss metric for model comparison?

Quite a few Kaggle competitions have used or are using the Logarithmic Loss metric as the quality measure of a submission. I'm wondering if there are other ways besides N-fold cross-validation to ...
0
votes
0answers
4 views

Modelling for budget allocation experiment

I'm trying to figure out the best way to model the following process: An individual is given $k$ points and asked to fully allocate it between $m$ items on a menu. The items all share upto $n$ ...
0
votes
0answers
33 views

Comparing Two classification models using F1-score

I am trying to compare the results of 2 classifiers trained with SVM using the F1 score. Some papers that I have read and that do this have made me a bit confused. I have trained the 2 classifiers ...
0
votes
0answers
38 views

R: selecting appropriate fit statistics from GBM output

I'm using the gbm.step function in the dismo package in R to evaluate the contribution of three continuous variables on my ...
0
votes
1answer
77 views

Variable Selection for Logistic regression

I am performing logistic regression. I understand assumptions of logistic regression - Outliers, Multicollinearity. What i didn't understand how to select variables at beginning of model preparation. ...
0
votes
0answers
18 views

Sparse PLS: algorithm for variable selection and model fitting

In the spls package in R (based on the manuscript by Chun and Keleş [1]), there is a separate specification for the variable selection and fitting in the main function, ...
0
votes
0answers
15 views

How to select and compare a subset of models from a big set

I have a model-selection question, any help appreciated. I generate curves with known and well-defined properties. I have a set of 20 thousand candidate models that I use to do non-linear fitting on ...
-1
votes
1answer
37 views

Deciding the Optimal Number of Factors [closed]

In practice, is there generally a difference between having 100 factors and 1000 factors in a model? Is there a well-researched 'upper-bound' to how many factors a given model should have?
0
votes
0answers
11 views

Should the rivals models include the same number of observed variables?

The question is about comparing models that include different number of observed variables. For example consider I have an 80-items questionnaire and I want to do confirmatory factor analysis (CFA) in ...
0
votes
0answers
11 views

AIC or similar selection techniques for Variograms?

I have a very basic question: how does one choose the "best" variogram? It is possible to fit different models to an empirical variogram, e.g. nugget, ...
1
vote
1answer
50 views

Which p-value to report in a comparison of different logistic regression models using marginal effects?

I am running logistic regression models to compare the impact of different indicators using Stata. As these comparisons may lead to false conclusion due to confounding and rescaling if log-odds or ...
1
vote
1answer
84 views

Simple model selection example in PYMC

I am currently experimenting with PYMC and I am trying out a simple example so that I start learning how things work (I am also a Python beginner, but an experienced machine learner). I have set ...
0
votes
0answers
7 views

How to specify that the model must include a particular independent variable in r package glmulti

I am trying to find the best model(lowest AICc value) using the package glmulti in r. This model must include TVIS as an independent variable but I am unsure how to specify this in the script. ...
1
vote
1answer
48 views

How to correlate categorical personality and music genre preference scores?

I'm currently a third year Biology student and I've annoyingly screwed myself over by not following the golden rule of stats, always know how to analyze your data prior to conducting the experiment. ...
0
votes
0answers
41 views

Model averaging with MuMIn. What's the mean of pvalue?

the summary() of a model.avg made with MuMIn in R, give a lot of interesting results, in particular model averaged coefficients (estimate, standard error, adjusted standard error and a z value with a ...
1
vote
0answers
33 views

Fitting a model to data for prediction - best choice for data

I have some data I need to fit a model to that can be used for prediction (interpolation). The data is summarized by the plot below. The black line is x=y. I want to be able to fit a model so as I ...
2
votes
1answer
54 views

c-index for parametric links in binary regression

I am conducting a binary regression using different sorts of parametric links (logistic, Pregibon, Aranda-Ordaz, ... see) and I would like to compare their predictive and classification perfomance in ...
0
votes
0answers
9 views

How to account for different ratio of samples during training and detection using a support vector machine (svm)?

Consider the following object recognition case: Detection of objects in an image using a sliding window approach in combination with a svm model. During sliding window search using multiple scale ...
0
votes
0answers
24 views

How to find a good model for an object recognition case using a support vector machine (svm)?

Consider the following example of an object recognition case: I'm trying to detect objects in an image using histograms of oriented gradients (hog) features. The feature vector resulting from hog is ...
2
votes
2answers
63 views

Using K-fold cross validation to select a model's parameters

I think I understand completely the concept of cross validation, but there is one aspect I've never seen detailed. Let's assume I have a logistic regression model with four parameters I want to train. ...
0
votes
0answers
7 views

Relative variable importance for simple model set

I am evaluating models based on AIC. I started by running the simplest models and the dot model (no covariates) is the best model, with little support for any others. When reporting the relative ...
2
votes
1answer
44 views

Correct numerical result in Bayesian model comparison

I was wondering how to calculate the following Bayesian model comparison. Suppose you have a couple of models: $$M_{1}: x \sim Bin(n, \pi); p \sim Be(1,1)$$ $$M_{2}: x \sim Bin(n, \pi); p \sim ...
0
votes
0answers
6 views

multimodel inference when using rms package

I would be glad to have some advise about how to proceed with multimodel inference to obtain weighted estimates based on AICc after running ordinal logistic analyzes with "rms' package. I used the ...