From the research I've done the common answer is that you can not remove insignificant dummy variables from a regression. I'm having a hard time finding academic papers or books that back up this thought. If anyone knows of any papers dealing with this issue or a good place to search through statistics papers that would be great!
I fear that your question might be considered ambiguous. Let us say you have a regression of the form: $yY_i = a . X_i + b2 . D2_i + b3 . D3_i + \epsilon_i$ Where D2 and D3 are dummies variable respectively equal to 1 if the underlying variable $D_i==2$ (respectively $D_i==3$). Reference category is $D_i ==1$.
Do you ask whether you should withdraw: - case A: both D2 and D3 if (b2 and b3) prove jointly insignificant? - case B: D2 if for instance b2 proves insignificant?
Case A : I do not think there is something special with dummy variables. You should ask another question. (My own understanding would be: why not? But please note that it changes the interpretation of the coefficient a. More has to be known regarding your objectives, I imagine.)
Case B: why not? But this will change the interpretation of coefficient b3, as reference category is now: $D_i==$ 1 or 2.