What exactly is the critique in this regression?

Question

Warning: I might be forgetting basic statistics here. Please edit title if it can be improved.

This paper, seemingly summarized in the fancy ZUI slideshow here, points out a possible "critique" in its... methodology or hypothesis choice or something (slides 26-27)?

The paper creates a regression with credit score (FICO) against some variables. Here is the model (It strangely doesn't have a $\beta_0$):

$FICO = -28.56(OtE) + 45.97(C) - 11.79(E) - 35.12(A) + 8.61(N) + 0.003(TP) + 0.002(OCBO) + 0.002(OCBI) + 0(PD) + 0(PA)$

The paper has a hypothesis for each variable being positively or negatively correlated with credit score (FICO), except for OtE, which was not expected to have any link (it says "no hypothesis", but I suspect this is layman for hypothesis of no statistical significance).

1. The correlation between OtE and credit score (FICO) is -0.17.

2. The $\beta$ of OtE is -28.56.

3. The correlation is deemed significant with $p < 0.05$.

So what exactly is the critique here?

It's a significant (despite having no expectation of any significance, whether positive or negative) negative correlation, but it's not a high negative correlation?

Maybe they meant that it is a critique because it is their only hypothesis that is significantly not true (All the others are either significantly true or not significantly true, I think) ?

If not...

I recall testing for statistical significance is testing $\beta = 0$. Is the correlation being significant equivalent to statistical significance ($\beta \neq 0$) ?

It seems like they are instead saying that $\beta = 0$ is false from the calculated -28.56. I don't recall being able to conclude significance of a coefficient from its number. 28.56 may be far from 0 relative to 6 or 0.01, but it may be near relative to 90 or 1,000,000.

Answer 1

The "critique" in the linked slide show seems to represent an apparent discrepancy: a significant correlation between FICO and OtE (evidently a measure of "openness") while there is no significant relation between OtE and FICO in the multiple regression model. These findings are in the context of the authors having hypothesized a relation between FICO and E (a measure of "extraversion") that was not supported by the multiple regression. They did not hypothesize a relation of OtE to FICO.

The absolute values of the coefficients aren't the important issue in these significance tests. A correlation coefficient is restricted to the range [-1,1]; the observed correlation coefficient of -0.17 between OtE and FICO would be significantly different from zero, given the number of cases, if those variables had a joint normal distribution. So that's considered "significant."

The $\beta$ coefficients have no such restriction so that their magnitudes depend on the measurement scales. For example, a car with a certain fuel efficiency will have different values depending on whether you express that efficiency as miles per gallon, kilometers per liter, etc. What's important in evaluating significance of a regression coefficient is the ratio of the coefficient to its estimated standard error; in the ratio the measurement scale cancels out. Table 2 of the paper (not in the slideshow) reports that the standard error of the -11.79 E regression coefficient was 11.69, and for the -28.56 OtE regression coefficient was 16.91. Neither coefficient reached the ratio of about 2 between coefficient and standard error needed to reach statistical significance.

So how to deal with the apparent discrepancy between a "significant" correlation between FICO and OtE and a "non-significant" coefficient in multiple regression? This is explained as follows. A correlation coefficient just looks at two variables. A regression coefficient looks at the relation of a predictor variable to the outcome variable when all the other variables are taken into account. The reason the authors did a multiple regression was because they has some hypotheses to test about relations of certain variables to FICO, and knew that they had to take the effects of other variables into account to try to isolate the relations they were testing.

A potential problem with the paper is that OtE and E are significantly correlated with each other ($r = 0.27$ from Table 1, in slide show), and both have nominally negative correlations with FICO. The point of "taking other variables into account" is to try to correct for correlations between variables that are measuring different things. But if both OtE and E are related to some deeper personality characteristic, then they might be two ways of measuring the same thing and this multiple regression could end up as a type of over-correction. The multiple regression perhaps couldn't determine which of these 2 variables to credit with the relation to FICO, so that both multiple regression coefficients ended up with standard errors too large to be considered significant. When predictor variables are correlated this can even lead to changes in the signs of coefficients between single comparisons and multiple regression.

So at a superficial level the authors did nothing wrong. These coefficients were not "statistically significant" in the multiple regression. There is a good chance, however, that had they approached this problem in a more sophisticated way they might have found different results. Trying to examine that many variables (11) with so few cases (142) can lead to problems, which the authors did not seem to address. There can be better ways to deal with these issues, such as principal components regression, ridge regression, and LASSO, which the authors evidently did not explore.