If I am using growth curve analysis with linear, quadratic and cubic time, and I want to include linear time as a random effect, do I need to also include quadratic and cubic time as random effects? Is it ever justified not to?
I assume you you are referring to random slopes for the time variables.
do I need to also include quadratic and cubic time as random effects?
You don't need to include them at all.
By fitting random slopes you are saying that you want each participant, subject, or whatver your grouping variable is, to have their own linear, quadratic and cubic terms as offsets from the global averages (fixed effects) for them. This may make sense conceptually, but practically speaking there are some issues. In particular, it is quite common for the variance in the higher order terms to be orders of magnitude smaller than than for the linear terms. This often means that unless you have a very large sample size, these small variances cannot be estimated well in a frequentist paradigm, and a singular fit often results. A Bayesian approach may help if you really want to fit random slopes for the higher order terms.
Is it ever justified not to?
Yes, when the data do not support such a complex random structure.