0
$\begingroup$

I have a simple data set to find out about the effect of cultivation period length on soil organisms. The main factor of interest is age_class, a categorical variable defining the age of a field under investigation. But, I also measured several environmental variables. I use R for the analysis.

My approach to select the best model would be to calculate all possible models using MuMIn::dredge(). I forced all models to include the main factor (age_class), since I want to make post-hoc pairwise comparisons to get compact letter display of differences between the age-classes.

Normally with this approach one would do model averaging and not simply define the best model, since other models might be as good as the best fit. However, the averaged model object of MuMIn can not be used in post-hoc tests and I really hate the cumbersome presentation of coefficients for the levels of a categorical variable. Whereas compact letter display is easy to understand for everybody. Besides, many journals want it that way and I read that post hoc comparisons based on confidence intervals of point estimates are not recommended

Furthermore the problem of including age_class as fixed variable gives rise to the problem that the relative importance of this parameter will be fixed at 1 Grueber et al 2011 (model selection and model averaging). In this paper the problem was stated, but no solution was given to it.

I was wondering if I could simply use multimodel calculation as tool to detect the best model and then give the strength of evidence (p. 26) for this particular model (which is given as weights in the dredge() Output?).???

Improve this question
$\endgroup$
5
  • $\begingroup$ How do you define the "best" model? $\endgroup$
    – Tim
    Commented Apr 6, 2016 at 10:25
  • $\begingroup$ After reading the Grueber paper, it sounds incorrect to me to state that the "relative importance" of age_class will be fixed at 1 if you follow their prescription. My reading of that was that averaging of the parameters and their standard errors should be on a weighted basis. Given that and across models, weightings could be based on a repercentaged F- or t-statistic for that parameter, repercentaged across all the models included in the averaging. $\endgroup$
    – user78229
    Commented Apr 6, 2016 at 11:05
  • $\begingroup$ @Tim The best model would be defined as the model with the lowest AICc. $\endgroup$
    – Pharcyde
    Commented Apr 7, 2016 at 9:28
  • $\begingroup$ @DJohnson: I must appologize I don't understand your last sentence. $\endgroup$
    – Pharcyde
    Commented Apr 7, 2016 at 9:29
  • $\begingroup$ No worries. It was merely a methodological proposal to weight the contribution of the each parameter by the magnitude of the F- or t- statistics associated with that parameter -- summed and repercentaged (to 100%) across all of the models. $\endgroup$
    – user78229
    Commented Apr 7, 2016 at 10:19

1 Answer 1

Reset to default
2
$\begingroup$

There is no real problem with getting a model-averaged estimate and confidence interval (and p-value, if you must) of the effect of age_class by model averaging the estimates and standard errors. The question you refer to deals with other concerns. I would assume sensible journals would be fine with you presenting the model averaged inference regarding age_class (the other covariates are presumably nuissance parameters, I assume?).

If you insist to look at how much weight models with the variable age_class in receive versus those without, then you need the models without the variable for comparison.

Regarding what you consider at the end: Ignoring the model selection (or only stating that it has taken place) invalidates any confidence intervals, p-values, estimates you present.

Btw. since you did not state what software you use, I have no idea what those commands you intersperse mean.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ (+1) At least the dredge command seems to be honestly named! $\endgroup$
    – Scortchi ♦
    Commented Apr 6, 2016 at 10:35
  • $\begingroup$ I use R with the MuMIn package $\endgroup$
    – Pharcyde
    Commented Apr 7, 2016 at 9:29
  • $\begingroup$ I don't understand what you mean by "ignoring the model selection" and how tht would invalidate my results, if I would use stepwise model selection I would do neither Averaging over component models nor documentation of them. But when I understand you right, you say that if I claculate all candidate models I have to average over them? $\endgroup$
    – Pharcyde
    Commented Apr 7, 2016 at 9:34
  • 1
    $\begingroup$ To take the stepwise model selection example, then yes, if you do stepwise model selection and then fit the final selected model as if it had been the single pre-specified model, then 95% confidence intervals will not cover true parameter values 95% of the time, p-values will be uninterpretable and estimates will be biased. The same applies to other methods of selecting models, unless something is done to account for the model selection. Some methods have a correction already included in the method, but e.g. stepwise selection, or picking the model with the best information criterion do not. $\endgroup$
    – Björn
    Commented Apr 7, 2016 at 15:54
  • $\begingroup$ @Björn Can you recommend a book or paper that explain how the model selection procedure should be accounted, when doing either Stepwise-selection or Model averaging? Is there a "bullet point" to ease the search for correction procedures with regard to corrections of P-values? $\endgroup$
    – Pharcyde
    Commented Apr 8, 2016 at 7:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.