New answers tagged model-selection
0
By the rough rules of thumb in section 8.5 of Forecasting: Principles and Practice (2nd ed.) by Athanasopoulos & Hyndman, this indeed looks like an AR(1) process:
The data may follow an ARIMA(p,d,0) model if the ACF and PACF plots of the differenced data show the
following patterns:
the ACF is exponentially decaying or sinusoidal;
there is ...
answered 23 hours ago
S. Kolassa - Reinstate Monica
58.9k10 gold badges113 silver badges216 bronze badges
1
Stepwise model selection, particularly forward stepwise, is not very reliable. This page provides much general discussion.
With Cox models the problem is even worse than for ordinary linear regression.
In ordinary linear regression there is no bias if you omit a predictor that is uncorrelated to the predictors you include. In other types of regressions ...
0
I am actually doing this right now too! :) I have 3 model classes, logistic, random forest and GP.
My design is this (with 5-fold crossvalidation):
training data set - optimize parameters and hyperparameters (not sure if we have the same definition of hyperparameters; in my case these are the length-scales for GP covariance matrix).
validation data set - ...
0
There is nothing wrong whatsoever in using computerized aids as long as you understand the strengths and the shortcomings. Building a model is much like washing both sides of your face .. one needs to deal with both potential auto-projective (memory/arima) structure and deterministic structure ( pulses, level shifts , seasonal pulses and local time trends ). ...
1
Another drawback of CV (in addition to those in Frans Rodenburg's excellent answer) is when there is a dependency between samples, such as in a time series. In that case you can split into train/valid such that no training data has a dependency on any validation data. (E.g. in the case of time series, your validation data has the later timestamps.)
4
The auto.arima() algorithm from the forecast package for R does look at the characteristic roots and will not return a model with near-unit-roots even if it has a low AICc. This is explained at the very end of the link you cite.
1
Why should the above formula for AIC be the same regardless of the dimensionality of the data? Shouldn't there be some way (e.g., an additional term) dependent on the number of dimensions [of the data]?
There is no need for an additional term, since the maximised value of the likelihood function (which appears in the AIC formula) already maximises over all ...
2
I am assuming you have more than one prediction point on which to compare models M1 and M2 -- I wouldn't make a model selection based on a single prediction point. It really depends on your goal.
I would suggest summarizing your models' predictions in the following ways on the validation dataset:
Empirical coverage of the 95% prediction intervals -- that ...
6
Prior to the revival of deep learning in the last few years, hyperparameter tuning used to be called model selection. The purpose of the validation set is to choose among several model candidates. It shouldn't make a difference whether these models have the same architecture with different hyperparameters or are completely different architectures.
So no, ...
9
Cross-validation as an alternative in case of a lack of training data is quite an understatement. Unless your sample size is very large, validation performance can vary widely among different random splits.
Cross-validation suffers less from this, since it considers the results from multitude folds. Even better would be to average over multiple runs of ...
1
how can we get a sense of how our model may perform in the outer loop before actually bringing it to the outer loop?
If there wasn't any hyperparameter tuning using the inner CV results, they'd be a very good estimate for outer loop test results (just slightly pessimistically biased due to having slightly less training data).
The optimistic bias is caused ...
answered Nov 1 at 20:02
cbeleites supports Monica
27.4k2 gold badges59 silver badges110 bronze badges
1
The benefit is that you get a better estimate of the model performance as
it is based on absolutely more tested cases (the n : p ratio isn't relevant here, n is relevant)
you can check the stability of your modeling approach: if the 10 outer folds arrive essentially at equal models, then your training is stable. If these models vary wildly (not all ...
answered Nov 1 at 19:35
cbeleites supports Monica
27.4k2 gold badges59 silver badges110 bronze badges
1
Should we estimate performance in the outer loop only for the best model?
yes. more precisely: in each of teh outer loop's folds, for the model chosen by its inner CV.
What if I estimate performance in the outer loop for the logistic regression model, the best LASSO model and best SVM model from the inner loop?
Then you know that these three ...
answered Nov 1 at 19:28
cbeleites supports Monica
27.4k2 gold badges59 silver badges110 bronze badges
1
ARIMA(original_data, order=(0,1,0))?
The d term should be chosen after you analyze the autocorrelation in your data and before you difference the data. You can also conduct the stationarity test to determine whether you need to take differences. And if you log-transform the raw data in the first place, then after calculating your forecast, you should make a ...
0
Let's look at your final question:
Alternatively, is there another form of regression better suited to
this data, given that it is continuous and has non-normal residuals?
Yes, there is. You could use quantile regression. This makes no assumptions about the distribution of the residuals. It also allows you to look at more questions than regular OLS ...
answered Oct 29 at 10:08
1
Since you are evaluating the error on the holdout subsample, there is no reason to adjust twice the estimated negative log-likelihood for overfitting by adding $2p$. This adjustment would introduce a bias in your estimated twice the negative log-likelihood without adding any benefits to the estimate. Hence, I would not do as you do.
However, you could ...
0
In most cases, residuals will never be perfectly normal. In the first QQ plot you posted, the residuals (assuming that's what the plot if of) follow normality except for your two outlying points at the end. I really wouldn't sweat that too much, outliers happen, and it's not the worst departure from normality I've ever seen.
I'd check a scatter plot of ...
0
find that hyperparameter that gives the smallest error using the validation set.
If you use a validation set to optimize hyperparameters, that data is part of your training data (regardless of the name validation set).
In order to avoid confusion with the terminology (because it is not used for validation of the actual model), I'm going to use a new name ...
answered Oct 27 at 8:52
cbeleites supports Monica
27.4k2 gold badges59 silver badges110 bronze badges
0
I see this question has been asked long time ago; however, no answer points out to the significant shortcomings and misunderstandings in the question yet.
Please note:
You state that R^2 = ESS/TSS = 1 - RSS/TSS. This is only true in a linear context. The equality TSS = RSS + ESS holds true only in linear regression with intercept. Thus you can not use ...
0
Determining optimal lags is a double-edged sword. I agree with the procedure mentionend in the comments in principle. The advantage is that you will save degrees of freedom.
Other suggetions:
-You could try to seasonal adjust your data. This might reduce the "needed" lags.
Some generel comments on your Var:
-Do you checked for cointegration? Might worth ...
0
Generally , if the pure AR model has p coefficients AND the pure MA model has q coefficients select/use the AR model if p = < q OTHERWISE use the MA model.
Top 50 recent answers are included
