As Jeremy pointed out, EFA, CFA, and IRT models scores would usually be in close agreement. This is especially true in the case of unidimensional scales, or 2nd-order factor models (since this will take you back to almost the same configuration when working on the higher-order factor). Moreover, PCA, which does not take into account measurement error but is often used to select the number of relevant factors, will also be highly correlated to those factor scores, like is the case for raw summated scale score provided the scale is truly unidimensional --- after all, a simple or weighted sum of all item scores is all what is desired to summarize a latent trait. In the case of multi-dimensional scales, you can consider each scale separately, if this makes sense.
For an illustration, here is one of the three subscales of the Holzinger & Swineford (1939) study, available in
lavaan. I choose a simple correlated factor model, although several other CFA models could be built (and would be equally valid). I used the principal axis factoring for extracting factors in the case of (oblique) EFA. Both the EFA and CFA models were estimated on all items (3 subscales). For PCA, I restrict the computation on the single "visual" subscale (to avoid rotation after PCA).
As can be seen in the picture below (EFA factor scores are on the horizontal axis and PCA or CFA scores on the vertical axis), the correlation is above 0.95 in both cases.
Of course, there are many ways to construct factor scores in the EFA framework, see, e.g., Understanding and Using Factor Scores: Considerations for the Applied Researcher, by DiStefano et al. I'm almost sure I came across papers dealing with the correlation between EFA and CFA scores but I can't get my hands on it anymore.
It's not so "weird to try both and select the one that works the best" --- what is really problematic is to force a factor structure without testing its relevance on independant samples, this is just capitalizing on chance, IMO: I would simply suggest to use CFA factor scores if the factor structure is already defined, or use EFA scores if the interest is simply in feature reduction (as you would use PCA scores in PCR in a regression context). Differences between EFA and CFA are often overstated as both methods are useful, even in an exploratory approach (CFA has model fit indices, which may be helpful, or not).