"PCA" based on distance metric other than $L_2$ PCA is based on $L_2$ distance and is maximizing variance along the PC axes. 
What if we try a different distance measure (something else than $L_2$)?
Do any methods corresponding to PCA but with different distance measures exist?
Can they be more useful than the vanilla PCA under some settings?
 A: "What if we try a different distance measure (something else than $L_2$)?"
There are many ways of addressing that answer this question but the two obvious routes are:

*

*"Norm/reconstruction-focused" approaches with$L_1$-norm principal component analysis being and obvious example we minimize an $L_1$ instead of an $L_2$ norm of the reconstruction error.

*"Specific measurement types" i.e. model-based PCA (the most common being binary or count data but generally speaking in the exponential family). Collins et al. (2001) "A Generalization of Principal Components Analysis to the Exponential Family" is the seminal reference here.

So for example, if we use $L_1$ we effectively have a more robust version of PCA such that the influence of outliers is amortised. Aside $L_1$, the "nuclear norm" (sometimes called Schatten 1-norm) is also used in a similar effect for denoising; I liked Gu et al. (2014) "Weighted Nuclear Norm Minimization with Application to Image Denoising" as it was easy to follow and descriptive, the Wikipedia link contains a lot of signal processing references on this subject too.
Similarly, in a model-based PCA we minimise the divergence associated with the distribution at hand; Salmon et al. (2014) "Poisson noise reduction with non-local PCA", for example, shows a very nice application for Poisson data where the work out what is the appropriate Bregman divergence for their application. We need to note here that under model-based approaches matrix factorization is a by-product rather than a necessity.  We still compute projections obviously but in our "original space" these are projections are often non-orthogonal.
"Do any methods corresponding to PCA but with different distance measures exist?"
Yes, multiple ones as noted above; we don't even touch on the
robust PCA field where there is a lot of work there and even $L_0$ comes into play to promote sparsity. (So for example if we have both $L_1$ and $L_0$ we have versions of robust and sparse decompositions.)
"Can they be more useful than the vanilla PCA under some settings?"
Yes. Moving back to model-based approaches, when working on analysing questionnaires or datasets where the assumption of multivariate Gaussianity is obviously unfounded. I personally used this logistic PCA when working with symptoms data collected from medical databases where all my variables were indicators ("Do you have symptom $x$?") and we used Bernoulli deviance; the relevant reference for us was "Dimensionality reduction for binary data through the projection of natural parameters" by Landgraf & Lee (2020). I have also used probabilistic PCA as an imputation preprocessing step, I didn't find it great but it was a fun application too. From a reconstruction perspective, depending on your work, denoising use cases (as mentioned above) can also be quite relevant.
