So there's two points in your question I want to respond too.
- I know that if my categorical variable had only 2 levels (one dummy variable), I would drop this interaction term from the model if it were not significant, and run a main effects model.
In general, I wouldn't do this. My classical statistics training taught me that this was the 'standard' method to follow. However as I started to learn more independently I discovered that this is not really a statistically rigorous method to follow for a variety of reasons (i.e. p-values are not a particularly good way to rank models). Your reduced model (the one without the interaction term) is nested within the full model, and therefore any hypotheses about it can be tested and evaluated with the full model. Many people like to run the reduced model as it allots you more degrees of freedom and therefore more likely to get a significant p-value. This obviously comes with its issues (evaluate your model, don't keep evaluating every model possible until you find one that is significant).
- Should I drop only the non-significant interaction terms, or keep all the interactions in the model?
So this now follows from the point I made above. No, you should not drop the non-significant terms.
I'm assuming you have built this model because you have a specific statistical hypothesis you wish to test. This hypothesis will be testable with the full model, so that is the model you should use.