PCA unfortunately only makes sense with numerical data. For instance, if you have a variable that can take the values {'car', 'train', 'boat'}, it will be difficult to do arithmetic with them. If we replace them with $\{0, 1, 2\}$, we can do arithmetic, but that arithmetic will really not achieve what we want, because we'll be treating these labels like real numbers. As if (train + train = boat), or even train being closer to car than boat is. However we have another option, to create dummy variables, one variable for each value that this categorical variable can take, setting them equal to 1 or 0 accordingly.
(Note: some people would use one less variable than the number of values, letting all 0 indicate the remaining variable, but geometrically this is troubling because the all-zero-variable is closer to all the other variables than they are to eachother, so I would not do that with PCA).
These values can be rescaled and PCA can be applied. It may not be ideal, but at least it's not crazy!
Edit:
I had not noticed the word Likert in your post some how. Likert scales are ordinal, rather than categorical variables. In fact, typically one assumes Likert scales are interval, essentially meaning that the difference between 1 and 2 is the same as the difference between 4 and 5. Because of this scale absolutely makes sense! I have changed your title to use the word "ordinal" rather than "categorical."