LRT & AIC comparison, selecting a model fitted by weighted LS

Question

I am a completely newbie in statistics and I want to ask a couple of questions about Akaike Information Criteria & Likellihood Ratio Test for particular application.

I am trying to fit data using specialized software based on Generalized linear least squares (GLLS) methodology. I have a dataset, let's say $y_{i} = f(x_{i})$, and for each datapoint uncertainty of $\sigma_{i}$ is reported. Those uncertainty values are used during fitting, so the optimization of parameters is done minimizing $\chi^2 = \sum (y_{i} - f(x_{i}))^2 / \sigma_{i}^2$.

Models (which parameters are estimated during regression) use the sum of individual contributions of independent components. Each model can have at least 1 component, and each component is characterized by 3 independent parameters. When we have 2 components in the model - we actually fit 6 parameters to the given datasets, 1 component - 3 parameters, 3 components - 9 parameters, 100 - components - 300 parameters varied in the model, etc.

After each fitting routine (that is done in a separate process) - I am trying to find the "best model" in the meaning of fitting quality but with a minimal number of parameters. Best means - minimum chi2 value for ((Fit_result - Data) / unc_data )^2.

Note, in my case for each observation I have reported uncertainty in the data. It's important for the next discussion.

The process of model selection is done like this - we just delete components each iteration to get the best model with N-1 components (-3 parameters of the model on each step).

As an example - we have got: initial model with

Model(N_initial): 100 components, 300 parameters.

And we want to check if any of candidate_models - a set of 100 models, with one excluded component (N-1 components) is the best.

Candidate_models_set = [
candiadate_model_0: {
          Components: N-1 = 99,
          excluded_component: 0,
          parameters_num: 297}, 

candiadate_model_1: {
          Components: N-1 = 99,
          excluded_component: 1,
          parameters_num: 297},
...

candiadate_model_99: {
          Components: N-1 = 99,
          excluded_component: 99,
          parameters_num: 297},  
]

So creating candidate_models set is actually refitting the given data but with the models with lower number of parameters (-1 component, -3 parameters w.r.t. the initial model).

I was using AIC/AICc to compare models. We selected models with AIC comparable to the initial model e.g.,

AIC_candidate_model = AIC_initial_model +- AIC_threshold, 

or models that have much lower AIC then initial model.

In my case AIC_threshold was selected as 14 - just selected it manually, seeing good enough result.

From candidate_models with N-1 components that have good enough AIC ("passed AIC test"), we select the model with the minimum value of chi2 to move until we find the model with good AIC compared to the initial model with N parameters.

Questions:

0. If I have reported uncertainty (so we have much more information than simple least squares), does the AIC equation still works?

I mean, if we know the uncertainty in the data, and it was used during the fitting procedure, does it change something for the likelihood of the model -> AIC calculation?

because here (as I understood) the maximum value of a model's log-likelihood function is just calculated based on the residuals and number of parameters (SSE / (number of observations - num of parameters):

${AIC} =2k-2\ln({\hat {L}})=2k+n\ln({\hat {\sigma }}^{2})-2C$

where ${\hat {\sigma }^{2}=\mathrm {RSS} /\nu }$ - maximum likelihood estimate for the variance of a model's residuals distributions is the reduced chi-squared statistic.

But with provided uncertainty and without - we will have different models - I think Likellihood must be redefined for this case? How then? My derivation tells me that for negative Likellihood we will have:

$ -2 \ln(\hat {L}) = \sum_{i=1}^{n} \left( \frac{y_i - \text{model}(x_i)}{\sigma_i} \right)^2 $

1. Can you explain or give a link on the derivation of how to select the AIC threshold for my case can be done? Simple theoretical explanations are preferred for better understanding.

I have read here: p. 70-71 about AIC & AIC differences, but there is a very tense phrase there:

Models with $\Delta_{AIC}$ > 10 have either essentially no support, and might be omitted from further consideration, or at least those models fail to explain some substantial explainable variation in the data. These guidelines seem useful if R is small (even as many as 100), but may break down in exploratory cases where there may be thousands of models. The guideline values may be somewhat larger for nonnested models, and more research is needed in this area (e.g., Linhart 1988).

Please correct me if I am wrong: AIC can be used to compare non-nested models.

2. Is there any difference - if we use a Likelihood Ratio Test (LRT) for the same purpose? (Just comparing models one by one with the initial model and doing selection in the same way just using model with minimal chi2).

Is there actually a difference in AIC and LRT? Can we prove that for the described purpose they will select exactly the same models (if we select the same thresholds)?

If there are limitations to use LRT if deleting one component change parameters number by 3? (Do the models can be considered as nested in this case? note, after deletion of component happens - we do refit, so all the parameters can change their values).

What is the difference & advantage of AIC compared to the application of LRT for this case?

Thanks for your answers & comments, and excuse me for my poor understanding of "simple" things.

Answer 1

You have a pretty good take on the situation. LRTs work great for simultaneously testing > 1 parameter. Such tests are often called “chunk tests”.

But the premise of your questions is model selection, and your are implicitly assuming that model selection is a good idea. Because of model uncertainty model selection sometimes causes huge underestimation of standard errors due to “phantom degrees of freedom”. This is especially true if more than 2 or 3 models are being compared. Model uncertainty distorts statistical inference, and can also bias model parameter estimates. It is better to use general principles, limited by the effective sample size, to pre-specify a model, as detailed here.

There are at least 3 ways to transform the likelihood ratio $\chi^2$ (LRT) for a model, and thinking about these may help in understanding.

• LRT with $p$ degrees of freedom ($p$ parameters excluding intercepts): used to quantify overall predictive information in the model and to test for presence of an association; does not penalize for overfitting, but $p$-values do take multiplicity into account by essentially penalizing for degrees of freedom. $p$-values are not very interesting though, since we would fall off our chairs if nothing was associated with the outcome.
• LRT - $p$: this is the major component of $R^{2}_\text{adj}$ as discussed briefly here and is accounting for overfitting within your sample
• LRT - $2\times p$: this is essentially AIC, which estimates out of sample model performance. This measure penalizes both for overfitting in the original sample, and for the fact that new samples will have different predictor matrices with different variances and covariances (collinearities)