What are the differences between Factor Analysis and Principal Component Analysis? It seems that a number of the statistical packages that I use wrap these two concepts together. However, I'm wondering if there are different assumptions or data 'formalities' that must be true to use one over the other. A real example would be incredibly useful. 
 A: In a paper by Tipping and Bischop the tight relationship between Probabalistic PCA (PPCA) and Factor analysis is discussed. PPCA is closer to FA than the classic PCA is. The common model is
$$\mathbf{y} = \mu + \mathbf{Wx} + \epsilon$$
where $\mathbf{W} \in \mathbb{R}^{p,d}$, $\mathbf{x} \sim \mathcal{N}(\mathbf{0},\mathbf{I})$ and $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{\Psi})$.


*

*Factor analysis assumes $\mathbf{\Psi}$ is diagonal. 

*PPCA assumes $\mathbf{\Psi} = \sigma^2\mathbf{I}$


Michael E. Tipping, Christopher M. Bishop (1999). Probabilistic Principal Component Analysis, Journal of the Royal Statistical Society, Volume 61, Issue 3, Pages 611–622
A: From my response here:
Is PCA followed by a rotation (such as varimax) still PCA?
Principal Component Analysis (PCA) and Common Factor Analysis (CFA) are distinct methods. Often, they produce similar results and PCA is used as the default extraction method in the SPSS Factor Analysis routines. This undoubtedly results in a lot of confusion about the distinction between the two.
The bottom line is that these are two different models, conceptually. In PCA, the components are actual orthogonal linear combinations that maximize the total variance. In FA, the factors are linear combinations that maximize the shared portion of the variance--underlying "latent constructs". That's why FA is often called "common factor analysis". FA uses a variety of optimization routines and the result, unlike PCA, depends on the optimization routine used and starting points for those routines. Simply there is not a single unique solution.
In R, the factanal() function provides CFA with a maximum likelihood extraction. So, you shouldn't expect it to reproduce an SPSS result which is based on a PCA extraction. It's simply not the same model or logic. I'm not sure if you would get the same result if you used SPSS's Maximum Likelihood extraction either as they may not use the same algorithm.
For better or for worse in R, you can, however, reproduce the mixed up "factor analysis" that SPSS provides as its default. Here's the process in R. With this code, I'm able to reproduce the SPSS Principal Component "Factor Analysis" result using this dataset. (With the exception of the sign, which is indeterminate). That result could also then be rotated using any of R's available rotation methods.
data(attitude)
# Compute eigenvalues and eigenvectors of the correlation matrix.
pfa.eigen <- eigen(cor(attitude))
# Print and note that eigenvalues are those produced by SPSS.
# Also note that SPSS will extract 2 components as eigenvalues > 1 = 2.
pfa.eigen$values
# Set a value for the number of factors (for clarity)
kFactors <- 2
# Extract and transform two components.
pfa.eigen$vectors[, seq_len(kFactors)]  %*% 
  diag(sqrt(pfa.eigen$values[seq_len(kFactors)]), kFactors, kFactors)

A: There are numerous suggested definitions on the web. Here is one from a on-line glossary on statistical learning:

Principal Component Analysis
Constructing new features which are
  the principal components of a data
  set. The principal components are
  random variables of maximal variance
  constructed from linear combinations
  of the input features. Equivalently,
  they are the projections onto the
  principal component axes, which are
  lines that minimize the average
  squared distance to each point in the
  data set. To ensure uniqueness, all of
  the principal component axes must be
  orthogonal. PCA is a
  maximum-likelihood technique for
  linear regression in the presence of
  Gaussian noise on both inputs and
  outputs. In some cases, PCA
  corresponds to a Fourier transform,
  such as the DCT used in JPEG image
  compression. See "Eigenfaces for
  recognition" (Turk&Pentland, J
  Cognitive Neuroscience 3(1), 1991),
  Bishop, "Probabilistic Principal
  Component Analysis", and "Automatic
  choice of dimensionality for PCA".choice of dimensionality for PCA".
Factor analysis
A generalization of PCA which is based
  explicitly on maximum-likelihood. Like
  PCA, each data point is assumed to
  arise from sampling a point in a
  subspace and then perturbing it with
  full-dimensional Gaussian noise. The
  difference is that factor analysis
  allows the noise to have an arbitrary
  diagonal covariance matrix, while PCA
  assumes the noise is spherical. In
  addition to estimating the subspace,
  factor analysis estimates the noise
  covariance matrix. See "The EM
  Algorithm for Mixtures of Factor
  Analyzers".choice of dimensionality
  for PCA".

A: The top answer in this thread suggests that PCA is more of a dimensionality reduction technique, whereas FA is more of a latent variable technique. This is sensu stricto correct. But many answers here and many treatments elsewhere present PCA and FA as two completely different methods, with dissimilar if not opposite goals, methods and outcomes. I disagree; I believe that when PCA is taken to be a latent variable technique, it is quite close to FA, and they should better be seen as very similar methods. 
I provided my own account of the similarities and differences between PCA and FA in the following thread: Is there any good reason to use PCA instead of EFA? Also, can PCA be a substitute for factor analysis? There I argue that for simple mathematical reasons the outcome of PCA and FA can be expected to be quite similar, given only that the number of variables is not very small (perhaps over a dozen). See my [long!] answer in the linked thread for mathematical details and Monte Carlo simulations. For a much more concise version of my argument see here:  Under which conditions do PCA and FA yield similar results?
Here I would like to show it on an example. I will analyze the wine dataset from UCI Machine Learning Repository. It is a fairly well-known dataset with $n=178$ wines from three different grapes described by $p=13$ variables. Here is how the correlation matrix looks like:  

I ran both PCA and FA analysis and show 2D projections of the data as biplots for both of them on the figure below (PCA on the left, FA on the right). Horizontal and vertical axes show 1st and 2nd component/factor scores. Each of the $n=178$ dots corresponds to one wine, and dots are colored according to the group (see legend):

The loadings of the 1st and 2nd component/factor onto the each of the $p=13$ original variables are shown as black lines. They are equal to correlations between each of the original variables and the two components/factors. Of course correlations cannot exceed $1$, so all loading lines are contained inside of the "correlation circle" showing maximal possible correlation. All loadings and the circle are arbitrarily scaled by a factor of $3$, otherwise they would be too small to be seen (so the radius of the circle is $3$ and not $1$).
Note that there is hardly any difference between PCA and FA! There are small deviations here and there, but the general picture is almost identical, and all the loadings are very similar and point in the same directions. This is exactly what was expected from the theory and is no surprise; still, it is instructive to observe.
PS. For a much prettier PCA biplot of the same dataset, see this answer by @vqv.
PPS. Whereas PCA calculations are standard, FA calculations might require a comment. Factor loadings were computed by an "iterated principal factors" algorithm until convergence (9 iterations), with communalities initialized with partial correlations. Once the loadings converged, the scores were calculated using Bartlett's method. This yields standardized scores; I scaled them up by the respective factor variances (given by loadings lengths).
A: You are right about your first point, although in FA you generally work with both (uniqueness and communality).
The choice between PCA and FA is a long-standing debate among psychometricians. I don't quite follow your points, though. Rotation of principal axes can be applied whatever the method is used to construct latent factors. In fact, most of the times this is the VARIMAX rotation (orthogonal rotation, considering uncorrelated factors) that is used, for practical reasons (easiest interpretation, easiest scoring rules or interpretation of factor scores, etc.), although oblique rotation (e.g. PROMAX) might probably better reflect the reality (latent constructs are often correlated with each other), at least in the tradition of FA where you assume that a latent construct is really at the heart of the observed inter-correlations between your variables. The point is that PCA followed by VARIMAX rotation somewhat distorts the interpretation of the linear combinations of the original variables in the "data analysis" tradition (see the work of Michel Tenenhaus). From a psychometrical perspectice, FA models are to be preferred since they explicitly account for measurement errors, while PCA doesn't care about that. Briefly stated, using PCA you are expressing each component (factor) as a linear combination of the variables, whereas in FA these are the variables that are expressed as linear combinations of the factors (including communalities and uniqueness components, as you said).
I recommend you to read first the following discussions about this topic:


*

*What are the differences between Factor Analysis and Principal Component Analysis

*On the use of oblique rotation after PCA -- see reference therein

A: None of these response is perfect. Either FA or PCA has some variants. We must clearly point out which variants are compared. 
I would compare the maximum likelihood factor analysis and the Hotelling's PCA.
The former assume the latent variable follow a normal distribution but PCA has no such an assumption. This has led to differences, such as the solution, the nesting of the components, the unique of the solution, the optimization algorithms. 
A: Differences between factor analysis and principal component analysis are:
• In factor analysis there is a structured model and some assumptions. In this respect it is a statistical technique which does not apply to principal component analysis which is a purely mathematical transformation.
• The aim of principal component analysis is to explain the variance while factor analysis explains the covariance between the variables. 
One of the biggest reasons for the confusion between the two has to do with the fact that one of the factor extraction methods in Factor Analysis is called "method of principal components". However, it's one thing to use PCA and another thing to use the method of principal components in FA. The names may be similar, but there are significant differences. The former is an independent analytical method while the latter is merely a tool for factor extraction.
A: Principal component analysis involves extracting linear composites of observed variables.
Factor analysis is based on a formal model predicting observed variables from theoretical latent factors.
In psychology these two techniques are often applied in the construction of multi-scale tests
 to determine which items load on which scales.
They typically yield similar substantive conclusions (for a discussion see Comrey (1988) Factor-Analytic Methods of Scale Development in Personality and Clinical Psychology).
This helps to explain why some statistics packages seem to bundle them together.
I have also seen situations where "principal component analysis" is incorrectly labelled "factor analysis".
In terms of a simple rule of thumb, I'd suggest that you:


*

*Run factor analysis if you  assume or wish to test a theoretical model of latent factors causing observed variables.

*Run principal component analysis If you want to simply reduce your correlated observed variables to a smaller set of important independent composite variables.
A: For me (and I hope this is useful) factor analysis is much more useful than PCA. 
Recently, I had the pleasure of analysing a scale through factor analysis. This scale (although it's widely used in industry) was developed by using PCA, and to my knowledge had never been factor analysed. 
When I performed the factor analysis (principal axis) I discovered that the communalities for three of the items were less than 30%, which means that over 70% of the items' variance was not being analysed. PCA just transforms the data into a new combination and doesn't care about communalities. My conclusion was that the scale was not a very good one from a psychometric point of view, and I've confirmed this with a different sample. 
Essentially, if you want to predict using the factors, use PCA, while if you want to understand the latent factors, use Factor Analysis.
A: A quote from a really nice textbook (Brown, 2006, pp. 22, emphasis added).
PCA = principal components analysis
EFA = exploratory factor analysis
CFA = confirmatory factor analysis

Although related to EFA, principal components analysis (PCA) is frequently miscategorized as an estimation method of common factor analysis. Unlike the estimators discussed in the preceding paragraph (ML,
  PF), PCA relies on a different set of quantitative methods that are
  not based on the common factor model. PCA does not differentiate
  common and unique variance. Rather, PCA aims to account for the
  variance in the observed measures rather than explain the correlations
  among them. Thus, PCA is more appropriately used as a data reduction
  technique to reduce a larger set of measures to a smaller, more
  manageable number of composite variables to use in subsequent
  analyses. However, some methodologists have argued that PCA is a
  reasonable or perhaps superior alternative to EFA, in view of the fact
  that PCA possesses several desirable statistical properties (e.g.,
  computationally simpler, not susceptible to improper solutions, often
  produces results similar to those of EFA, ability of PCA to calculate
  a participant’s score on a principal component whereas the
  indeterminate nature of EFA complicates such computations). Although
  debate on this issue continues, Fabrigar et al. (1999) provide several
  reasons in opposition to the argument for the place of PCA in factor
  analysis. These authors underscore the situations where EFA and PCA
  produce dissimilar results; for instance, when communalities are low
  or when there are only a few indicators of a given factor (cf.
  Widaman, 1993). Regardless, if the overriding rationale and
  empirical objectives of an analysis are in accord with the common
  factor model, then it is conceptually and mathematically inconsistent to conduct PCA; that is, EFA is more appropriate if the stated
  objective is to reproduce the intercorrelations of a set of
  indicators with a smaller number of latent dimensions, recognizing the
  existence of measurement error in the observed measures. Floyd and
  Widaman (1995) make the related point that estimates based on EFA are
  more likely to generalize to CFA than are those obtained from PCA in
  that, unlike PCA, EFA and CFA are based on the common factor model.
  This is a noteworthy consideration in light of the fact that EFA is
  often used as a precursor to CFA in scale development and construct
  validation. A detailed demonstration of the computational differences between PCA and EFA can be found in multivariate and factor analytic textbooks (e.g., Tabachnick & Fidell,
  2001).

Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York: Guilford Press.
A: Expanding on @StatisticsDocConsulting's answer: the difference in loadings between EFA and PCA is non-trivial with a small number of variables. Here's a simulation function to demonstrate this in R:
simtestit=function(Sample.Size=1000,n.Variables=3,n.Factors=1,Iterations=100)
{require(psych);X=list();x=matrix(NA,nrow=Sample.Size,ncol=n.Variables)
for(i in 1:Iterations){for(i in 1:n.Variables){x[,i]=rnorm(Sample.Size)}
X$PCA=append(X$PCA,mean(abs(principal(x,n.Factors)$loadings[,1])))
X$EFA=append(X$EFA,mean(abs(factanal(x,n.Factors)$loadings[,1])))};X}

By default, this function performs 100 Iterations, in each of which it produces random, normally distributed samples (Sample.Size$=1000$) of three variables, and extracts one factor using PCA and ML-EFA. It outputs a list of two Iterations-long vectors composed of the mean magnitudes of the simulated variables' loadings on the unrotated first component from PCA and general factor from EFA, respectively. It allows you to play around with sample size and number of variables and factors to suit your situation, within the limits of the principal() and factanal() functions and your computer.
Using this code, I've simulated samples of 3–100 variables with 500 iterations each to produce data:
Y=data.frame(n.Variables=3:100,Mean.PCA.Loading=rep(NA,98),Mean.EFA.Loading=rep(NA,98))
for(i in 3:100)
{X=simtestit(n.Variables=i,Iterations=500);Y[i-2,2]=mean(X$PCA);Y[i-2,3]=mean(X$EFA)}

...for a plot of the sensitivity of mean loadings (across variables and iterations) to number of variables:

This demonstrates how differently one has to interpret the strength of loadings in PCA vs. EFA. Both depend somewhat on number of variables, but loadings are biased upward much more strongly in PCA. The difference between mean loadings these methods decreases as the number of variables increases, but even with 100 variables, PCA loadings average $.067$ higher than EFA loadings in random normal data. However, note that mean loadings will usually be higher in real applications, because one generally uses these methods on more correlated variables. I'm not sure how this might affect the difference of mean loadings.
A: One can think of a PCA as being like a FA in which the communalities are assumed to equal 1 for all variables.  In practice, this means that items that would have relatively low factor loadings in FA due to low communality will have higher loadings in PCA.  This is not a desirable feature if the primary purpose of the analysis is to cut item length and clean a battery of items of those with low or equivocal loadings, or to identify concepts that are not well represented in the item pool.
A: There many great answers for this post but recently, I came across another difference. 
Clustering is one application where PCA and FA yield different results. When there are many features in the data, one may be attempted to find the top PC directions and project the data on these PCs, then proceed with clustering. Often this disturbs the inherent clusters in the data - This is a well proven result. Researchers suggest to proceed with sub-space clustering methods, which look for low-dimensional latent factors in the model.  
Just to illustrate this difference consider the Crabs dataset in R. Crabs dataset has 200 rows and 8 columns, describing 5 morphological measurements on 50 crabs each of two colour forms and both sexes, of the species - Essentially there are 4 (2x2) different classes of crabs. 
library(MASS)
data(crabs)
lbl <- rep(1:4,each=50)
pc <- princomp(crabs[,4:8])
plot(pc) # produce the scree plot
X <- as.matrix(crabs[,4:8]) %*% pc$loadings
library(mclust)
res_12 <- Mclust(X[,1:2],G=4)
plot(res_12)


res_23 <- Mclust(X[,2:3],G=4)
plot(res_23)

Clustering using PC1 and PC2:

Clustering using PC2 and PC3:

#using PC1 and PC2:
     1  2  3  4
  1 12 46 24  5
  2 36  0  2  0
  3  2  1 24  0
  4  0  3  0 45

#using PC2 and PC3:
    1  2  3  4
  1 36  0  0  0
  2 13 48  0  0
  3  0  1  0 48
  4  1  1 50  2

As we can see from the above plots, PC2 and PC3 carry more discriminating information than PC1. 
If one tries to cluster using the latent factors using a Mixture of Factor Analyzers, we see much better result compared against using the first two PCs. 
mfa_model <- mfa(y, g = 4, q = 2)
  |............................................................| 100%
table(mfa_model$clust,c(rep(1,50),rep(2,50),rep(3,50),rep(4,50)))

     1  2  3  4
  1  0  0  0 45
  2 16 50  0  0
  3 34  0  0  0
  4  0  0 50  5

A: From Factor Analysis Vs. PCA (Principal Component Analysis)





Principal Component Analysis
Factor Analysis




Meaning
A component is a derived new dimension (or variable) so that the derived variables are linearly independent of each other.
A factor (or latent) is a common or underlying element with which several other variables are correlated.


Purpose
PCA is used to decompose the data into a smaller number of components and therefore is a type of Singular Value Decomposition (SVD).
Factor Analysis is used to understand the underlying ‘cause’ which these factors (latent or constituents) capture much of the information of a set of variables in the dataset data. Hence, it is also known as Common Factor Analysis (CFA).


Assumption
PCA looks to identify the dimensions that are composites of the observed predictors.
Factor analysis explicitly presumes that the latent (or factors) exist in the given data.


Objective
The aim of PCA is to explain as much of the cumulative variance in the predictors (or variables) as possible.
FA focuses on explaining the covariances or the correlations between the variables.


How much variation is explained?
The components explain all the variance in the data. PCA captures the maximum variance in the first component, then in the second component, and henceforth followed by the other components.
The latent themselves are not directly measurable, and they do not explain all the variance in the data. Hence, it results in an error term that is unique to each measured variable.


Process
In PCA, the components are calculated as the linear combinations of the original variables.
In factor analysis, the original variables are defined as the linear combinations of the factors.


Mathematical representation
Y =  W1* PC1 + W2* PC2+…  + W10 * PC10 +C Where, PCs are the components and Wis are the weights for each of the components.
X1 = W1F + e1 X2 = W2F + e2 X3 = W3*F + e3   Where, F is the factor, Wis are the weights and eis are the error terms. The error is the variance in each X that is not explained by the factor.


Interpretation of the weights
The weights are the correlation between the standardized scores of the predictors (or variables) and the principal components, also known as the factor loadings. For example, in PCA, the weights indicate which component contributes more to the target variable, Y, as the independent variables are standardized.
The weights in the factor analysis express the relationship or association of each variable (X) to the underlying factor (F). These are also known as the factor loadings and can be interpreted as the standardized regression coefficients.


Estimation of the weights
PCA uses the correlation matrix of the variables, which generates the eigenvectors (or the components) and estimates it as the betas (or the coefficients).
The process of factor analysis ascertains the optimal weights.


Pecking order
In PCA, the variables are specified and then estimate the weights (coefficients or betas) through regression.
In factor analysis, the latent (or the factors) are first specified and then estimate the factor returns through regression.





Use Cases and Applications of PCA
The use cases of PCA are:

*

*PCA is highly used for image processing. It has wide applications in domains such as facial recognition, computer vision. Image
processing is a method to perform operations on an image to either
enhance an image or extract and determine information or patterns from
it.

*It has its use in the field of investment to analyze stocks and predict portfolio returns. Also, it can be used to model the yield
curves.

*PCA also has its applications in the area field of bioinformatics. One such use case is the genomic study done using gene expression
measurements.

*Both the banking sector and marketing have vast applications of PCA. It can be used to profile customers based on their demographics.

*PCA has been extensively used to conduct clinical studies. It is used in the healthcare sector and also by researchers in the domain of
food science.

*In the field of psychology, PCA is used to understand psychological scales. It can be used to understand statically the
ineffective habits that we must have broken yesterday!

Use Case and Applications of Factor Analysis
Some of the business problems where factor analysis can be applied
are:

*

*You may have heard of the old saying, “Don’t put all your eggs in one basket.” In case you have a stock portfolio, then you know what I
am referring to. Investment professionals rely on factor analysis to
diversify their stocks. It is used to predict the movement across
stocks in a consolidated sector or industry.

*In the space of marketing, factor analysis can be used to analyze customer engagement. It is a measure of how much a product or brand is
interacting with its customers during the product’s life cycle.

*The HR managers can employ factor analysis to encourage employee effectiveness. It can be done by identifying the features that have
the most impact on employee productivity.

*Factor analysis can be applied to group (or segment) the customers based on the similarity or the same characteristics of the customers.
For example, in the insurance industry, the customers are categorized
based on their life stage, for example, youth, married, young family,
middle-age with dependents, retried. Another example is of restaurants
that would frame their menu to target customers based on the
demographics. For example, a fine dining restaurant in an upper
locality will not have the same menu as a tea stall near a college
campus.

*Schools, colleges, or universities also apply factor analysis to make their decisions as the class curriculum would be dependent on the
difference in the levels of the classes. This ultimately determines
the salary and staffing limits of the teachers.

*This technique is also handy for exploring the relationships in the category of socioeconomic status, dietary patterns.

*Like PCA, factor analysis can also be used to understand the psycholo


