9
$\begingroup$

In this thread @ttnphns writes that

Because it is regression coefficients [...] I insist that it is better to say "factor loads variable" than "variable loads factor".

I learned from here that a factor analysis model is the system of equations

$V_1 = a_{1I}F_I + a_{1II}F_{II} + E_1$

$V_2 = a_{2I}F_I + a_{2II}F_{II} + E_2$

$...$

$V_p = …$

where coefficient a is a loading, F is a factor [...], and variable E is regression residuals.

However, I don't get how it follows that we should say "factor loads variable" rather than vice-versa. What is it about the term "loading" that makes it follow?

I also don't know why we need the term "loading" at all, when we already had the term "regression coefficient". Is it because sometimes the regression coefficients are also correlation coefficients, and statisticians wanted a generic term to cover both cases?

I'm hoping that the answer to this question will make it easier for students to remember that factors load observed variables, rather than the other way around.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ I can never remember what is supposed "to load" (or to load on? or to be loaded by? etc.) what, and so I avoid using the verb "to load" in this context. I do say "loadings" though. However, note that FA is not a regression, and there are no "regression coefficients" here, at least not in the usual sense of the word. I think it would only be confusing to call $A$ regression coefficients. Either don't call it anything at all, or call it "loadings". But they are not regression coefficients! $\endgroup$
    – amoeba
    Commented Jun 14, 2016 at 0:01
  • $\begingroup$ @amoeba, saying FA is not a regression you are both correct and not correct. FA as extraction procedure is of course not a regression procedure. But FA model is a regressional model. If we were able to know true Fs values (instead of approximate scores) and the fit was excellent (wrt reproduction of correlations by the loadings), and we decide to regress the Vs by those Fs' values, the loadings will come out as our parameter estimates. $\endgroup$
    – ttnphns
    Commented Jun 14, 2016 at 0:26

1 Answer 1

Reset to default
7
$\begingroup$

I don't get how it follows that we should say "factor loads variable" rather than vice-versa

Abstract explanation. If a point seen as object has a coordinate on an axis seen as feature then the coordinate is how much the feature loads the point, how much it charges, by itself, that point. If my height is 1.86 m then this is how I'm loaded by height (not how much height is loaded by me). Note that loading is variable's coordinate on factor-as-axis on the loading plot.

Latent-trait explanation. Factor is conceptualized as an entity which plays "in" the variables or "behind" them and which makes them correlate. Therefore "load" is intuitively a good word to express the degree how strongly the variable is dependent on, driven by, the latent factor. Factor analysis model is regressional model whereby factors explain or "influence" observed variables. Any regression coefficient (not only factor analytic) may be labeled a "loading": regressional coefficient = regressional weight = regressional loading. More reason to call a factor's coefficient "loading" comes from the fact that in the factor model, factors $F$s are set standardized, each unit-variance, while a variable $V$ isn't necessarily standardized. There comes therefore that the effect on $V$ is realized/expressed completely and only via the loading coefficients. Whenever in regressional model a standardized variable predicts a potentially unstandardized one - call the coefficient "loading".

Why we need the term "loading" at all, when we already had the term "regression coefficient"

We actually don't need. Word "loading" is simply a tradition stemming from psychologists' liking for figurative sense (FA started to develop a century ago among psychologists). Moreover, the term "loading" may have somewhat different statistical meaning in other related multivariate methods (such as discriminant analysis). In general, some people in some cases call "loadings" regression coefficients, while other or in other cases - correlation coefficients. So the term is confusing. It is not a statistical term, ultimately.

If you don't like the word, don't use it. You may also say "variable loads (on) factor" if you want; to me, it is simply a thoughtless speech, not a vice.

P.S. I've just looked in an English dictionary (English isn't my language) and observed that to load may have meanings as (1) "I loaded the cart" (by a bag, or by myself as embarked); (2) "the ship loads (up) many passengers (on it)". If to follow the second word usage, it would be quite OK to say "the variable loads the factor (on itself, the variable) well".

Improve this answer
$\endgroup$
8
  • $\begingroup$ +1. But I don't like calling loadings "regression coefficients". To me, "regression" implies that there are dependent and independent variables and the goal is to find regression coefficients. In latent variable models such as FA there is only one set of variables, and the goal is to find latent variables. This is not a regression; the math and the computations are entirely different (one usually needs something like that expectation-maximization to find an FA solution; there is no need for EM in regression). I would keep the term "regression coefficient" for regression. $\endgroup$
    – amoeba
    Commented Jun 13, 2016 at 22:06
  • 1
    $\begingroup$ This is not a regression @amoeba, I don't object your way of making sense of these terms, albeit my way is slightly different from yours. That discrepancy of ours is of minor importance. Note that saying "regressional (= regression-like) model" isn't exactly as saying "regression (analysis)". FA isn't a regression, obviously, because there isn't external Y variable(s). Still, it models the Xs by the somehow extracted Fs as if the Xs were those Ys. Explaining FA in terms of "regressional model" to students is a smooth pass. $\endgroup$
    – ttnphns
    Commented Jun 13, 2016 at 23:40
  • $\begingroup$ Does this terminology affect the following interpretation of loadings? The interpretation of the loading is: 1. The higher the loading of a PC, the more influence it has in the formation of the variable. 2. The higher the loading of a variable, the more influence it has in the formation of the principal component score. 3. Both ? If it matters, full post here. $\endgroup$
    – user_anon
    Commented Sep 15, 2018 at 12:22
  • 1
    $\begingroup$ OK now. 1. Correct (well, usually we don't take a PC for a honest factor though). 2. Correct, but not straightforward. Please see how component score coefficients are computed from loadings: stats.stackexchange.com/a/126985/3277, and also this discussing the difference of those coef-s from loadings: stats.stackexchange.com/a/191332/3277 $\endgroup$
    – ttnphns
    Commented Sep 15, 2018 at 13:00
  • 1
    $\begingroup$ @user_anon, 3). However, both 1 and 2 are also valid if you are speaking in terms of random variables rather than a dataset. Normal distribution isn't required, generally. $\endgroup$
    – ttnphns
    Commented Sep 15, 2018 at 16:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.