Can you remove a factor from your model if it has a significant effect, but the removal improves AIC and R square?

I have a complex problem but the title sums it up pretty easily.

I have four types of cages that manipulate water flow, but I also have an actual measure of water flow from inside the cages. I'm wondering if I can just use one or the other, if I should include both, or if I should nest them.

The best fit seems to be with just the actual measure of flow, but if I use them both, the cage type has a significant effect.

Any tips?

• It should not be possible to increase $R^2$ by removing a variable. It is possible to improve (i.e. reduce) AIC, though. – Richard Hardy Apr 13 '18 at 17:23
• Sorry, you're right, but the difference between the Rsquare and the adjusted Rsquare does decrease with the removal of the factor, which I thought was a good sign...right? – Nathan Haag Apr 13 '18 at 18:36
• No, I do not think this would indicate anything interesting. Simply the adjustment factor changes as there are fewer variables. – Richard Hardy Apr 13 '18 at 18:38
• The answer has to depend on your analytical objectives. If you're trying to determine whether cage type is associated with flow, then you don't want to drop cage type. If you're only trying to predict flow, or if you will later encounter many more cage types than appear in your data, then you might not use cage type--but you wouldn't use statistical significance as a criterion to determine that. You would use some appropriate measure of predictive accuracy. – whuber Apr 13 '18 at 19:13
• Some prominent researchers like Rob J. Hyndman (here) argue that statistical significance should not be used as a criterion for variable selection, while AIC may be, especially if the goal is forecasting. – Richard Hardy Apr 13 '18 at 19:19

As @whuber suggested, you may be primarily interested in (1) the effect of cage type on your dependent variable (DV), (2) the effect of water flow on your DV or (3) predicting your DV.

For (1), you can report this effect with or without adjusting for water flow.

For (2), you can report this effect with or without adjusting for cage type.

The reported effects will have different meaning - for instance, assuming cage type and water flow have fixed effects, the effect of water flow on the DV will apply to all cages of the same type (with adjustment for cage type) or to all cages irrespective of their type (without adjustment).

So, for (1) or (2), you really need to figure out what you are interested in reporting at the end of the day and go from there.

For (3), @whuber already gave you valuable hints.

• The basic question of this chapter of my PhD is comparing how 2 mussel species react under different water flow regimes, so I guess I'm leaning toward (1) or (2). I personally don't care about the cage types because they're not an industry standard, I just made them out of ABS piping (although they're easy to make, so if anyone wants to repeat it, there could be some novelty in knowing the effect they have). My advisor suggested just using the actual measure of water flow, but I wanted to be sure I wasn't eliminating important information. – Nathan Haag Apr 14 '18 at 23:30