Warning: I might be forgetting basic statistics here. Please edit title if it can be improved.

This paper, seemingly summarized in the fancy ZUI slideshow here, points out a possible "critique" in its... methodology or hypothesis choice or something (slides 26-27)?

The paper creates a regression with credit score (FICO) against some variables. Here is the model (It strangely doesn't have a $\beta_0$):

$FICO = -28.56(OtE) + 45.97(C) - 11.79(E) - 35.12(A) + 8.61(N) + 0.003(TP) + 0.002(OCBO) + 0.002(OCBI) + 0(PD) + 0(PA)$

The paper has a hypothesis for each variable being positively or negatively correlated with credit score (FICO), except for OtE, which was not expected to have any link (it says "no hypothesis", but I suspect this is layman for hypothesis of no statistical significance).

  1. The correlation between OtE and credit score (FICO) is -0.17.

  2. The $\beta$ of OtE is -28.56.

  3. The correlation is deemed significant with $p < 0.05$.

So what exactly is the critique here?

It's a significant (despite having no expectation of any significance, whether positive or negative) negative correlation, but it's not a high negative correlation?

Maybe they meant that it is a critique because it is their only hypothesis that is significantly not true (All the others are either significantly true or not significantly true, I think) ?

If not...

I recall testing for statistical significance is testing $\beta = 0$. Is the correlation being significant equivalent to statistical significance ($\beta \neq 0$) ?

It seems like they are instead saying that $\beta = 0$ is false from the calculated -28.56. I don't recall being able to conclude significance of a coefficient from its number. 28.56 may be far from 0 relative to 6 or 0.01, but it may be near relative to 90 or 1,000,000.


The "critique" in the linked slide show seems to represent an apparent discrepancy: a significant correlation between FICO and OtE (evidently a measure of "openness") while there is no significant relation between OtE and FICO in the multiple regression model. These findings are in the context of the authors having hypothesized a relation between FICO and E (a measure of "extraversion") that was not supported by the multiple regression. They did not hypothesize a relation of OtE to FICO.

The absolute values of the coefficients aren't the important issue in these significance tests. A correlation coefficient is restricted to the range [-1,1]; the observed correlation coefficient of -0.17 between OtE and FICO would be significantly different from zero, given the number of cases, if those variables had a joint normal distribution. So that's considered "significant."

The $\beta$ coefficients have no such restriction so that their magnitudes depend on the measurement scales. For example, a car with a certain fuel efficiency will have different values depending on whether you express that efficiency as miles per gallon, kilometers per liter, etc. What's important in evaluating significance of a regression coefficient is the ratio of the coefficient to its estimated standard error; in the ratio the measurement scale cancels out. Table 2 of the paper (not in the slideshow) reports that the standard error of the -11.79 E regression coefficient was 11.69, and for the -28.56 OtE regression coefficient was 16.91. Neither coefficient reached the ratio of about 2 between coefficient and standard error needed to reach statistical significance.

So how to deal with the apparent discrepancy between a "significant" correlation between FICO and OtE and a "non-significant" coefficient in multiple regression? This is explained as follows. A correlation coefficient just looks at two variables. A regression coefficient looks at the relation of a predictor variable to the outcome variable when all the other variables are taken into account. The reason the authors did a multiple regression was because they has some hypotheses to test about relations of certain variables to FICO, and knew that they had to take the effects of other variables into account to try to isolate the relations they were testing.

A potential problem with the paper is that OtE and E are significantly correlated with each other ($r = 0.27$ from Table 1, in slide show), and both have nominally negative correlations with FICO. The point of "taking other variables into account" is to try to correct for correlations between variables that are measuring different things. But if both OtE and E are related to some deeper personality characteristic, then they might be two ways of measuring the same thing and this multiple regression could end up as a type of over-correction. The multiple regression perhaps couldn't determine which of these 2 variables to credit with the relation to FICO, so that both multiple regression coefficients ended up with standard errors too large to be considered significant. When predictor variables are correlated this can even lead to changes in the signs of coefficients between single comparisons and multiple regression.

So at a superficial level the authors did nothing wrong. These coefficients were not "statistically significant" in the multiple regression. There is a good chance, however, that had they approached this problem in a more sophisticated way they might have found different results. Trying to examine that many variables (11) with so few cases (142) can lead to problems, which the authors did not seem to address. There can be better ways to deal with these issues, such as principal components regression, ridge regression, and LASSO, which the authors evidently did not explore.

  • 1
    $\begingroup$ In this case, the authors may have overcontrolled by including two correlated predictors, so there was no room for evidence of a link of either to FICO in the multiple regression that they ran. I would not be surprised if they performed a different type of analysis like LASSO they might have found one of these 2 to be related to FICO. $\endgroup$ – EdM Aug 29 '15 at 1:58
  • 1
    $\begingroup$ Standard errors are in table 2 of the original paper, to which I have access, but not shown in the slideshow. Extraversion (E) and openness (OtE) are variables 4 and 6 in table 1 (under "critique" in the slideshow); they have a significant correlation with each other of 0.27, and the sign of the correlation coefficient for each is negative. In this case trying to "control for" OtE in testing the relation of E to FICO by linear regression might have masked a true relation of these variables to FICO. For beta the ratio to the standard error is what matters for significance, not the value itself. $\endgroup$ – EdM Aug 30 '15 at 2:24
  • 1
    $\begingroup$ I edited the answer to include some of this discussion. I may have over-explained from your perspective, but on this site we try to post answers that will be helpful to others who might come across this question later and might not be so sophisticated. $\endgroup$ – EdM Aug 31 '15 at 15:32
  • 1
    $\begingroup$ In analyzing binary outcomes the rule of thumb is to have no more than 1 predictor for each 10 to 20 cases of the least-frequent outcome. It's hard to give a similar rule of thumb for linear regression as in this paper, since so much depends on the structure of the particular data set. I tend to start worrying as the ratio of cases to predictors approaches a lower limit of 10, which is why I would have appreciated some effort by the authors of this paper to validate their model. $\endgroup$ – EdM Sep 18 '15 at 13:13
  • 1
    $\begingroup$ This was a linear regression, not a logistic regression, so a strict rule of thumb isn't appropriate. See this CV question and answers for an introduction to a more thorough discussion. $\endgroup$ – EdM Sep 18 '15 at 13:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.