How reliable is factor analysis? I've read some papers that used factor analysis to find latent factors in data. How reliable is this technique in general for correcting identifying related factors? Are spurious results likely? Is it easy for two or more unrelated factors to be conflated into one factor?
 A: I've been uneasily watching this question, hoping for a hapless CVr other than me to bite. I was wrong. Apologies in advance (i.e., spoiler alert) to "Jane" in making this observation as I'm sure I sound like a total prig but, querying the CV horde about FA's "reliability," is one of those naive, red herring kind of questions one can get from someone completely new to multivariate statistics, even the classic, old-school stuff like factor analysis. 
The key thing about factor analysis, factorization, projection pursuit, whatever you care to call it, of any flavor or specification is that it's a dimension reducing technique intended to push correlated content together, be that attributes, features, data, whatever, into a new latent dimension, component, combination that "explains" the variance between and among the input variables. In writing that sentence, I'm aware of subsuming all kinds of technical nuances, e.g., between CFA, PCA, SVD, Harris-Kaiser image analysis, continuously and "normally" distributed variables, mixture models of dummy and continuously scaled variables, orthogonal vs oblique rotations, linear vs nonlinear metrics, sparse, dense, extreme valued data, yada yada yada. 
Given that, "reliability" isn't the issue with FA for the simple reason that the results are as "reliable" as those available from any statistical technique on the face of the earth, with all of the appropriate caveat emptors. For instance, conjoint analysis is "reliable" but depending on how the trade-offs are elicited can be quite unstable. CART-type classification trees are "reliable," but given the many criticisms of their instability and poor predictive power, Breiman developed random forests as a workaround. 
The issue with FA -- and, in point of fact, any and all dimension-reducing techniques -- are the dozens of subjective decisions that have to be made en route to a solution. While there are heuristics, conventions and rules of thumb to guide the analyst in making these decisions, their use are all, without exception, a function of that analyst's training and comprehension...or the team and culture with which and in which they are working.
The thing that surprised me about this question was the use of tags for "categorical-data." That tag opens up a whole other kettle of fish having to do with PCA of categorical information as leveraged by SVD-driven methods such as correspondence analysis, homogeneity analysis, and so on. 
With respect to "Jane's" other questions: "spurious results" are no more likely than from any other multivariate technique. "Conflation" is a function of whether one chooses to generate orthogonal (unassociated) vs oblique (associated) factor structures. 
And so it goes.
