Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

0
votes
0answers
9 views

Correct evaluation/ comparison between undercomplete and overcomplete representations

Suppose I'm performing Unsupervised Feature Learning method to learn a representation of the data that is under-complete (e.g. 100 features) and use another algorithm to learn an over-complete ...
3
votes
0answers
28 views

Model selection: OLS vs TLS

I have two sets of real-valued data and I am interested in their correlation. From my perspective, there appear to be errors both variables, so I am inclined to perform a regression with TLS (Total ...
0
votes
1answer
36 views

VAR/VECM/ARDL optimal lag selection

Question 1: Is it necessary to consider AIC and the BIC criteria when selecting the lag for a VAR, VECM or ARDL model OR can I use something else? Example: Can I pick 12 lags because the model simply ...
0
votes
0answers
27 views

Model Selection - adding categorical covariates to mixed linear models

I have a dataset consisting of genotypes (crop lines) being grown in multi-replication trials across environments. Here is the mixed linear model I've been working with: $$ Y_{ijk} = \mu + G_i + E_j ...
3
votes
3answers
77 views

How to best model interaction effect of two continuous predictor variables?

Consider the following problem: In a logistic regression model, we believe that two continuous predictor variables $X_1$ and $X_2$ impact the probability of event. It is hypothesized that the ...
0
votes
0answers
5 views

Picking a particular model from regsubsets

I ran regsubsets in r from the 'leaps' library. I have gotten some 16 models in their order of which is best according to certain criterion. How do I select, say, model no.14 from this order and run ...
2
votes
0answers
33 views

GLMs with transformed response variable

I wonder if use of generalized linear models (GLMs) with transformed response variables is correct. My particular case: I compared goodness of fit of several GLMs with response variable transformed ...
1
vote
0answers
38 views

model fitting of data to multiple distributions

I have a set of numbers $ X = \{x_1, x_2,\ldots,x_n\}$ and I am interested in finding the most fitting combination of these numbers to multiple exponential distributions. Using predefined rules, I ...
0
votes
0answers
8 views

model validation with few samples and high variance

I have to choose a model using a set of 90 observations, which I divide between validation and test. First I select the model and the probability threshold that have the highest accuracy on the ...
0
votes
0answers
30 views

What kind of study and statistical test to use for this data?

I'm currently writing my bachelor thesis in CS in which I wrote a program which weights words in a text in a certain way (resulting in a list of words sorted by their weights) and displays N words ...
0
votes
0answers
9 views

Nested cross validation vs. split dataset into train, validation and test for parameter selection and performance evaluation

The goal is to get the unbiased performance estimation of the 'algorithm' (or model), e.g. precision and recall. And get a final model for practical usage. From what I read online, nested cross ...
3
votes
1answer
46 views

Post Model Selection Inference problems - which remedies exist?

Recently, Hannes Leeb from Yale University and Benedikt Pötscher from the University of Vienna have published a series of papers dealing with what they call Post Model Selection Inference problems.* ...
0
votes
1answer
18 views

Justify choice of polynomia based on statistically significant result?

I am using an OLS. The variable of interest is Nth day to the end of the year (discrete variable). I would like to represent the relationship between y and Nth day to the end of the year with a ...
0
votes
0answers
16 views

What kind of bias am I introducing when I include Validation set (kept for model selection) in Train data?

As usual, I have three sets of data: Train, Validation and Test. So I use train data for model selection, where I select the model which would perform best on validation data. After selecting the best ...
0
votes
0answers
36 views

Use AUC for model comparison but what is the optimal threshold for final prediction

We can compare the performance of different models using AUC ROC and pick the one with large AUC. Then, we still need to choose and use specific threshold to predict the label for the test data. I ...
0
votes
0answers
11 views

How to calculate AIC for multiple participants

I have a number of competing models that I fit separately to each participant's data. What is the correct way to calculate AIC in this case? Can I just write $AIC = 2nk-2\sum_{i=1}^n{LL_i}$, where $n$ ...
0
votes
0answers
17 views

What is the best statistical model for my binary outcome variable? Logistic or Structural equation model?

I have one dependent variable with binary outcome. List of independent explanatory variables that are categorical. I need to find out whether the list of explanatory variables influence the dependent ...
0
votes
0answers
18 views

Poisson Regression - Robust Standard Error, Log-Likelihood

am interested in determining the association between an outcome and several predictors using relative risk ratios. I would like to use Poisson regression. As outlined in this article ...
0
votes
0answers
18 views

How to do model selection with models from vglm() function?

I create models with function vglm() (Package VGAM), and I wish to select them with ...
0
votes
1answer
27 views

Size of hidden layer in neural networks for learning specific logical rules

According to this answer, a general rule of thumb is that your hidden layer size should be between your input and output sizes. In developing my JavaScript neural network, this has proven to be about ...
0
votes
1answer
46 views

Choosing Random Forests' parameters

I am new to machine learning and would like to know if it makes sense to fix the number of estimators and the maximal depth of a random forest with cross validation ? My intuition would be that yes, ...
3
votes
1answer
40 views

Model selection for random effects: can unselected random effects be used as fixed effects?

I am working on a mixed effects model. What I would consider random effects are year, sampling transect, and sampling location. There are multiple collections taken along each transect, and multiple ...
0
votes
0answers
23 views

KS statistic in credit scorecard modeling

I work on developing scorecard models in credit risk, and typically model performances are measured and comparisons done using the KS statistic (and c statistic). Is there a way in which this ...
0
votes
0answers
15 views

Selecting best model in GLM logistic with AICc, p-values for best are insignificant?

I have one potentially causal predictor and a number of covariates that I tested via AICc model selection in logistic GLM. I found that alone, the causal predictor has a low AICc (~19) and a ...
1
vote
0answers
40 views

Convergence analysis for forward stagewise regression?

Forward stagewise regression is a simple model selection algorithm related to least angle regression and LASSO. (see e.g. the LARS paper) It repeats the following steps, initializing a predictor ...
0
votes
0answers
34 views

Cross validation with model selection (in-sample fit and out-of-sample prediction)

I currently have 3 models to predict Y from a linear combination of independent variables: Model 1: Y ~ A + B Model 2: Y ~ A + C Model 3: Y ~ A + D Now, I want to compare their in-sample fitting ...
1
vote
2answers
33 views

How can I determine the ARIMA orders ($p$,$d$,$q$) from this correlogram?

I need help for understanding how can I interpret this correlogram in order to determine the $p$, $d$ and $q$ orders for ARIMA model. I use Stata, and I am analysing a time series with really few ...
1
vote
2answers
81 views

Interpretation of AIC value

Typical values of AIC that I have seen for logistic models are in thousands, at least hundreds. e.g. On http://www.r-bloggers.com/how-to-perform-a-logistic-regression-in-r/ the AIC is 727.39 While ...
1
vote
0answers
23 views

HMM Model Selection

What is the process for selecting a model for an HMM? Say the data is time sequences, where each time sequence represents a class. I can used Baum-Welch to train, but I don't know how to determine ...
1
vote
0answers
16 views

Evaluating model performance - different possible metrics

I am comparing the root mean squared error (RMSE) of different regression models that were fit to simulated data. I have RMSE from 30 different simulated data sets for each regression model. Some of ...
0
votes
0answers
17 views

Model selection for nested count data

What is the best tool-box for model selection when working with nested count data? Is AICc appropriate for comparing Poisson and negative binomial mixed models? Is there anything special to the ...
0
votes
1answer
29 views

Variable selection for regression and classification

This might be noob question, because I've just started to learn data analytics. Why should we or should we not include a correlated predictor in our model? While selecting predictors, for a class ...
0
votes
0answers
9 views

Model selection across multiple criteria (qualitative and quantitative)

I have two linear regression models on the same data, but where the response variable has been transformed using respectively the BoxCox transformation and the logit transformation. Therefore, I ...
3
votes
0answers
16 views

Comparing confirmatory factor analysis models with different number of observed variables

I have a 14 item scale that is hypothesized to measure self-esteem. The first model I specify uses all 14 items and predicts one latent factor (i.e., self-esteem). I would then like to specify a ...
1
vote
0answers
21 views

Adjusting for spatial autocorrelation

I have a data set on sand martin population sizes along a stretch of river over 40 years. The river is split into sections and the number of birds per section was counted. I have been trying to ...
0
votes
0answers
19 views

Model comparison with different predictors

I have a conducted a series of experiments and manipulated a variable (X) that - from the literature - I know is relevant in this context. For theoretical reasons, I am now convinced that the ...
3
votes
1answer
105 views

Why is training error a better performance metric than cross-validation error?

What I have learned is that I should use cross-validation performance for selecting the best model. Currently, selecting the models based on cross-validation performance gives lower test performance ...
0
votes
0answers
8 views

How to see the adjR-square in Lasso Regression?

After doing lasso, the final parameters are only 6, but I have 200 covariates originally, is it too literally? And how to see the correspond adj R-square in Lasso Regression? By the way, I also tried ...
0
votes
2answers
40 views

Selecting best ARIMA model with regressors and dummy variable

I have data on GDP, employment rate, inflation and production on two countries and I like to make some ARIMA models. I have done this before, but not with including regressors. Also, the time period ...
0
votes
1answer
53 views

2 Models. Feature distributons are same, though quality is lower

The situation is following: I took ML algorithm, say gradient boosting decision trees. In october my data collector generated me train data A. I did crossvalidation and got 96% of accuracy. In ...
0
votes
1answer
59 views

Testing significance of RMSE of models

I repeatedly trained two neural network models and calculated the RMSE for each run (split validation). Which statistical test is most useful in this case for testing if the difference of the models ...
1
vote
0answers
19 views

Inferring values based on rankings

I'm looking to infer items' values from only ranked lists of these items. I'm assuming that each item has some value, for which a higher value makes it more likely to appear first. This value can be ...
0
votes
0answers
38 views

Understand outer cross validation with caret in R

I have a question that I cannot solve. Sorry if it is too naive, I am a beginner. I have a data set from wich I would like to predict a continuous variable Y based on a set of features. By now I ...
0
votes
0answers
82 views

Comparing AIC/BIC Between Continuous (CFA) and Categorical (LCA) Latent Models

Some colleagues and I have a set of variables that we would like to represent more parsimoniously/latently. Originally, my colleagues used an exploratory and confirmatory factor-analysis approach to ...
0
votes
0answers
7 views

Model selection for order of ranked data in unequally distributed groups?

I'm wondering what type of analysis or statistical model would work best for this particular type of scenario. I've encountered once before (I'll explain that at the bottom) but the example ...
1
vote
1answer
47 views

How to control trade-off between precision and recall?

I applied different classification algorithms in combination with different sampling techniques to a dataset and I get > 100 different models with different performances. I can choose a model for ...
1
vote
0answers
15 views

Do variable selection methods change based on the type of response variable?

The diagnostic tests for different models would be different, but how would the variable selection change? For instance, how much different is it to fit a Poisson regression model versus a logistic ...
0
votes
1answer
35 views

Mixture of Priors/Algebra?

Can someone explain how the author gets to the expression after the words "This leads to:"
0
votes
0answers
40 views

Use of MuMIn::dredge with an offset term

I discovered that the dredge function in the MuMIn package in r cannot handle offset terms ...