4
$\begingroup$

Let's say I want to build a factor model to explain (excess) stock returns--think Fama-French, for instance. Obviously one could use OLS to fit a model, but I've seen PCA used as well. What are the practical differences between the two methodologies? I know OLS minimizes squared residuals, while PCA sort of minimizes the (not sure if squared or absolute) errors diagonally (or orthogonally to the variance-covariance matrix eigenvector with the highest eigenvalue, to be more specific). This is shown graphically here: http://www.r-bloggers.com/principal-component-analysis-pca-vs-ordinary-least-squares-ols-a-visual-explanation/? So what are the implications of doing this vs. running an OLS regression? I'm not quite sure when you'd use one methodology over the other.

Another concern I have is the model output. You have a very clear model output with OLS: $\hat y= \hat \beta_0 + \hat\beta_1X_1 + ... +\hat\beta_nX_n+\epsilon$

And there are straightforward methods to assess the fit of your model (e.g., $R^2$ or t-tests).

I guess you could represent a PCA model similarly to how you'd represent an OLS model (as above) using the main eigenvector I referred to earlier as sort of your "line of best fit". But from there, are there simple tests the likes of $R^2$ or t-tests that are available?

More importantly, when would you use PCA as opposed to OLS generally speaking? What makes you decide whether your PCA model provides a good fit?

I looked at this blog post and it intrigued me quite a bit, but I have no clue what's going on: http://www.calculatinginvestor.com/2013/03/18/pca-factors-vs-fama-french-factors/. I'm not sure what the writer means by "PCA factors" and whether or not his use of $R^2$ is appropriate for PCA. Could you perhaps briefly comment on how PCA could be used in the context of factor analysis specifically, whether you're testing three known factors or trying to find them from a pool of, say, 10?

I hope my questions aren't all over the place, but I'd really appreciate any insight you might have on the whole OLS vs. PCA issue.

$\endgroup$
1
  • 3
    $\begingroup$ You can use OLS with PCA. For instance, instead of running a regression with 100 Macroeconomic drivers, you could extract a few PCA components from the independent variables, and use them as variables in OLS. Also, if you have to forecast 20 highly correlated variables separately or in vector regression, you could extract 3 PCA factors, regress them separately on variables, then invert PCA to get your 20 variables back. $\endgroup$
    – Aksakal
    Jan 27, 2016 at 22:17

2 Answers 2

2
$\begingroup$

PCA is a data transform tool that transforms the data to a new coordinate system. It is used to reduce dimension. For example, you can apply PCA to Fama-French's 3 factor to transform three time series into one time series. Then use OLS tech to find the $\beta$. I do not know exactly why people may use PCA before OLS to reduce dimension of the independent variables. Maybe reduced time series gives a better prediction.

$\endgroup$
3
  • 1
    $\begingroup$ One example does come up. Some of the risk model have say 500 factors, but your observation may only have say two years of data which is only 600 observation. So you can't use OLS before you reduce dimension. $\endgroup$
    – JOHN
    Jan 27, 2016 at 21:36
  • 1
    $\begingroup$ I'm not sure what you mean exactly. If I'm visualizing this correctly, when you're reducing your data to a single dimension, it's no longer being mapped to the variable you're trying to explain (returns in our case). You just have a single line that sort of shows you how volatile the variable is. I'm looking here: setosa.io/ev/principal-component-analysis to think this through. I must be mistaken or misunderstanding.. $\endgroup$ Jan 27, 2016 at 22:03
  • 1
    $\begingroup$ PCA is orthogonal transform and it will indeed cause the lost of economic meaning. The example of apply PCA to Fama-French is definitely a bad example. No one do that. You can try to look at Quantitative Equity Investing for some example. It has a chapter to introduce why and how to use PCA. $\endgroup$
    – JOHN
    Jan 28, 2016 at 16:44
1
$\begingroup$

To the distinction when use what: "This is the linear case of what is known as Orthogonal Regression or Total Least Squares, and is appropriate when there is no natural distinction between predictor and response variables, or when all variables are measured with error. This is in contrast to the usual regression assumption that predictor variables are measured exactly, and only the response variable has an error component." from https://www.mathworks.com/help/stats/examples/fitting-an-orthogonal-regression-using-principal-components-analysis.html

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.