Take my answer as a comment more than a true answer (I am a new contributor so i cannot comment yet). If you can compute the varcov of the variables, then you can use PCA on that varcov matrix: of course you can compute the covariances between random variables even when they are binomial variables that numerically represent a categoriacal variable referred to two categories only (the same holds for multi-category variables). So be sure that you are representing your categorical variable via numbers (0-1 for a binomial categoriacal variable or 0-1-2,... for a categorical variable with more than 2 categories) and calculating the varcov correctly. Having said that, personally, I would prefer to keep them outside the PCA, especially if they are binomial and especially if you have just a few of them compared to the total number of features: for example, transform the set of non-categorical features via PCA to obtain a set of orthogonal features, then add the categorical variables to the set of simplified orthogonal features.
Code in python for the spectral decomposition of a simple varcov matrix obtained from two binomial variables
import numpy as np
import scipy.linalg as la