Let's say that I have one categorical variable with six levels, and I then create five indicator variables in order to represent the six levels. If two of the five variables are insignificant, then do I drop these two? I assume not, but I was not sure. I was thinking that it might be better to test the full (all five variables) versus the reduced (just the three significant variables) model and, if that was not significant, then just leave all five of the variables in. I was not sure what to do. Oh and I meant for this to be in the context of fitting a logistic regression model.
You should leave all five indicator variables in. Dropping predictors because they are non-significant leads to biased estimates for regression coefficients and inflated p-values.
A good reference that discusses this at length is Frank Harrell's Regression Modeling Strategies. You can find a summary of the problems with dropping insignificant features in section 4.3 there.
The latter approach (comparing two models with and without the five variables and decide if you should keep them as a set) is better.
The problem with dropping the indicator is that you'll change the p-values of the remaining levels as well, as you're shifting the intercept (aka the reference group.) Given a model:
$y = b_0 + b_1 Lv2 + b_2 Lv3 + b_3 Lv4 + b_4 Lv5 + b_5 Lv6 + \epsilon$
The intercept represents the mean of $y$ for group $Lv1$. Now, if we drop, say, the last two terms:
$y = b_0 + b_1 Lv2 + b_2 Lv3 + b_3 Lv4 + \epsilon$
Because you only drop the variable and not the cases, subjects in levels 5 and 6 need a place to go: notice that your intercept is now picking the groups $Lv5$ and $Lv6$ as well, representing the mean $y$ for levels 1, 5, and 6.
So, two major points: 1: your reference group can change and such change is not always sensible. 2: you may be surprised to see the significant results you wish to keep may be gone, due to the reference group mean has also changed.
Here's my two cents. I can't say with full certainty, but, I guess, it very much depends on a model and data. If I understand this answer correctly, @gung advises to test your model(s) after dropping all and then some levels. However, the details on how exactly to perform the testing are rather fuzzy (at least, to me). Perhaps, he will be kind enough to expand on that for beginners like me.
You may also find relevant and useful this course notes document on logistic regression (in
R) by Professor Christopher Manning (Stanford University). Among other things, he describes dropping whole categorical variables (factors in
R terminology) and manipulations with categorical variable levels, such as collapsing several levels into a single one and other manipulations, as well as the impact of those actions on quality of regression models and interpretations of analysis results.