Return to Answer

The "critique" in the linked slide show seems to represent an apparent discrepancy: a significant correlation between FICO and OtE (evidently a measure of "openness") while there is no significant relation between OtE and FICO in the multiple regression model. These findings are in the context of the authors having hypothesized a relation between FICO and E (a measure of "extraversion") that was not supported by the multiple regression. They did not hypothesize a relation of OtE to FICO.

The absolute values of the coefficients aren't the important issue in these significance tests. A correlation coefficient is restricted to the range [-1,1]; the observed correlation coefficient of -0.17 between OtE and FICO would be significantly different from zero, given the number of cases, if those variables had a joint normal distribution. So that's considered "significant."

The $\beta$ coefficients have no such restriction so that their magnitudes depend on the measurement scales. For example, a car with a certain fuel efficiency will have different values depending on whether you express that efficiency as miles per gallon, kilometers per liter, etc. What's important in evaluating significance of a regression coefficient is the ratio of the coefficient to its estimated standard error; in the ratio the measurement scale cancels out. Table 2 of the paper (not in the slideshow) reports that the standard error of the -11.79 E coefficient was 11.69, and for the -28.26 OtE coefficient was 16.91. Neither coefficient reached the ratio of about 2 between coefficient and standard error needed to reach statistical significance.

So how to deal with the apparent discrepancy between a "significant" correlation between FICO and OtE and a "non-significant" coefficient in multiple regression? This is explained as follows. A correlation coefficient just looks at two variables. A regression coefficient looks at the relation of a predictor variable to the outcome variable when all the other variables are taken into account. The reason the authors did a multiple regression was because they has some hypotheses to test about relations of certain variables to FICO, and knew that they had to take the effects of other variables into account to try to isolate the relations they were testing.

A potential problem with the paper is that OtE and E are significantly correlated with each other ($r = 0.27$ from Table 1, in slide show), and both have nominally negative correlations with FICO. The point of "taking other variables into account" is to try to correct for correlations between variables that are measuring different things. But if both OtE and E are related to some deeper personality characteristic, then they might be two ways of measuring the same thing and this multiple regression could end up as a type of over-correction. The multiple regression perhaps couldn't determine which of these 2 variables to credit with the relation to FICO, so that both multiple regression coefficients ended up with standard errors too large to be considered significant. When predictor variables are correlated this can even lead to changes in the signs of coefficients between single comparisons and multiple regression.

So at a superficial level the authors did nothing wrong. These coefficients were not "statistically significant" in the multiple regression. There is a good chance, however, that had they approached this problem in a more sophisticated way they might have found different results. Trying to examine that many variables (11) with so few cases (142) can lead to problems, which the authors did not seem to address. There can be better ways to deal with these issues, such as principal components regression, ridge regression, and LASSO, which the authors evidently did not explore.

The "critique" in the linked slide show seems to represent an apparent discrepancy: a significant correlation between FICO and OtE (evidently a measure of "openness") while there is no significant relation between OtE and FICO in the multiple regression model. These findings are in the context of the authors having hypothesized a relation between FICO and E (a measure of "extraversion") that was not supported by the multiple regression. They did not hypothesize a relation of OtE to FICO.

The absolute values of the coefficients aren't the important issue in these significance tests. A correlation coefficient is restricted to the range [-1,1]; the observed correlation coefficient of -0.17 between OtE and FICO would be significantly different from zero, given the number of cases, if those variables had a joint normal distribution. So that's considered "significant."

The $\beta$ coefficients have no such restriction so that their magnitudes depend on the measurement scales. For example, a car with a certain fuel efficiency will have different values depending on whether you express that efficiency as miles per gallon, kilometers per liter, etc. What's important in evaluating significance of a regression coefficient is the ratio of the coefficient to its estimated standard error; in the ratio the measurement scale cancels out. Table 2 of the paper (not in the slideshow) reports that the standard error of the -11.79 E regression coefficient was 11.69, and for the -28.56 OtE regression coefficient was 16.91. Neither coefficient reached the ratio of about 2 between coefficient and standard error needed to reach statistical significance.

So how to deal with the apparent discrepancy between a "significant" correlation between FICO and OtE and a "non-significant" coefficient in multiple regression? This is explained as follows. A correlation coefficient just looks at two variables. A regression coefficient looks at the relation of a predictor variable to the outcome variable when all the other variables are taken into account. The reason the authors did a multiple regression was because they has some hypotheses to test about relations of certain variables to FICO, and knew that they had to take the effects of other variables into account to try to isolate the relations they were testing.

A potential problem with the paper is that OtE and E are significantly correlated with each other ($r = 0.27$ from Table 1, in slide show), and both have nominally negative correlations with FICO. The point of "taking other variables into account" is to try to correct for correlations between variables that are measuring different things. But if both OtE and E are related to some deeper personality characteristic, then they might be two ways of measuring the same thing and this multiple regression could end up as a type of over-correction. The multiple regression perhaps couldn't determine which of these 2 variables to credit with the relation to FICO, so that both multiple regression coefficients ended up with standard errors too large to be considered significant. When predictor variables are correlated this can even lead to changes in the signs of coefficients between single comparisons and multiple regression.

So at a superficial level the authors did nothing wrong. These coefficients were not "statistically significant" in the multiple regression. There is a good chance, however, that had they approached this problem in a more sophisticated way they might have found different results. Trying to examine that many variables (11) with so few cases (142) can lead to problems, which the authors did not seem to address. There can be better ways to deal with these issues, such as principal components regression, ridge regression, and LASSO, which the authors evidently did not explore.

The confusion here has to do with correlation coefficients between FICO and each predictor variable separately, versus coefficients from multiple regression of FICO against all the predictor variables together.

A correlation coefficient just looks at two variables. A regression coefficient looks at the relation of a predictor variable to the outcome variable when all the other variables are taken into account. When predictor variables are correlated this can even lead to changes in the signs of coefficients between single comparisons and multiple regression.

Looking at the paper quickly (it's behind a pay wall, unfortunately), I think that what happened here was that two correlated variables, openness (OtE) and extraversion (E), both tend to have negative relations to FICO. The multiple regression effectively couldn't determine which of these 2 variables to credit with the relation to FICO, so that both multiple regression coefficients ended up with standard errors too large to be considered significant. The standard error of the E coefficient was 11.69, and for OtE 16.91; neither coefficient reached the ratio of about 2 between coefficient and standard error needed to reach nominal statistical significance.

So at a superficial level the authors did nothing wrong. These coefficients were not "statistically significant" in the multiple regression. There is a good chance, however, that had they approached this problem in a more sophisticated way they might have found different results. Trying to examine that many variables (11) with so few cases (142) can lead to problems, which the authors did not seem to address.

The "critique" in the linked slide show seems to represent an apparent discrepancy: a significant correlation between FICO and OtE (evidently a measure of "openness") while there is no significant relation between OtE and FICO in the multiple regression model. These findings are in the context of the authors having hypothesized a relation between FICO and E (a measure of "extraversion") that was not supported by the multiple regression. They did not hypothesize a relation of OtE to FICO.

The absolute values of the coefficients aren't the important issue in these significance tests. A correlation coefficient is restricted to the range [-1,1]; the observed correlation coefficient of -0.17 between OtE and FICO would be significantly different from zero, given the number of cases, if those variables had a joint normal distribution. So that's considered "significant."

The $\beta$ coefficients have no such restriction so that their magnitudes depend on the measurement scales. For example, a car with a certain fuel efficiency will have different values depending on whether you express that efficiency as miles per gallon, kilometers per liter, etc. What's important in evaluating significance of a regression coefficient is the ratio of the coefficient to its estimated standard error; in the ratio the measurement scale cancels out. Table 2 of the paper (not in the slideshow) reports that the standard error of the -11.79 E coefficient was 11.69, and for the -28.26 OtE coefficient was 16.91. Neither coefficient reached the ratio of about 2 between coefficient and standard error needed to reach statistical significance.

So how to deal with the apparent discrepancy between a "significant" correlation between FICO and OtE and a "non-significant" coefficient in multiple regression? This is explained as follows. A correlation coefficient just looks at two variables. A regression coefficient looks at the relation of a predictor variable to the outcome variable when all the other variables are taken into account. The reason the authors did a multiple regression was because they has some hypotheses to test about relations of certain variables to FICO, and knew that they had to take the effects of other variables into account to try to isolate the relations they were testing.

A potential problem with the paper is that OtE and E are significantly correlated with each other ($r = 0.27$ from Table 1, in slide show), and both have nominally negative correlations with FICO. The point of "taking other variables into account" is to try to correct for correlations between variables that are measuring different things. But if both OtE and E are related to some deeper personality characteristic, then they might be two ways of measuring the same thing and this multiple regression could end up as a type of over-correction. The multiple regression perhaps couldn't determine which of these 2 variables to credit with the relation to FICO, so that both multiple regression coefficients ended up with standard errors too large to be considered significant. When predictor variables are correlated this can even lead to changes in the signs of coefficients between single comparisons and multiple regression.

So at a superficial level the authors did nothing wrong. These coefficients were not "statistically significant" in the multiple regression. There is a good chance, however, that had they approached this problem in a more sophisticated way they might have found different results. Trying to examine that many variables (11) with so few cases (142) can lead to problems, which the authors did not seem to address. There can be better ways to deal with these issues, such as principal components regression, ridge regression, and LASSO, which the authors evidently did not explore.

The confusion here has to do with correlation coefficients between FICO and each predictor variable separately, versus coefficients from multiple regression of FICO against all the predictor variables together.

A correlation coefficient just looks at two variables. A regression coefficient looks at the relation of a predictor variable to the outcome variable when all the other variables are taken into account. When predictor variables are correlated this can even lead to changes in the signs of coefficients between single comparisons and multiple regression.

Looking at the paper quickly (it's behind a pay wall, unfortunately), I think that what happened here was that two correlated variables, openness (OtE) and extraversion (E), both tend to have negative relations to FICO. The multiple regression effectively couldn't determine which of these 2 variables to credit with the relation to FICO, so that both multiple regression coefficients ended up with standard errors too large to be considered significant. The standard error of the E coefficient was 11.69, and for OtE 16.91; neither coefficient reached the ratio of about 2 between coefficient and standard error needed to reach nominal statistical significance.

So at a superficial level the authors did nothing wrong. These coefficients were not "statistically significant" in the multiple regression. There is a good chance, however, that had they approached this problem in a more sophisticated way they might have found different results. Trying to examine that many variables (11) with so few cases (142) can lead to problems, which the authors did not seem to address.

The confusion here has to do with correlation coefficients between FICO and each predictor variable separately, versus coefficients from multiple regression of FICO against all the predictor variables together.

A correlation coefficient just looks at two variables. A regression coefficient looks at the relation of a predictor variable to the outcome variable when all the other variables are taken into account. When predictor variables are correlated this can even lead to changes in the signs of coefficients between single comparisons and multiple regression.

Looking at the paper quickly (it's behind a pay wall, unfortunately), I think that what happened here was that two correlated variables, openness (OtE) and extraversion (E), both tend to have negative relations to FICO. The multiple regression effectively couldn't determine which of these 2 variables to credit with the relation to FICO, so that both multiple regression coefficients ended up with standard errors too large to be considered significant. The standard error of the E coefficient was 11.69, and for OtE 16.91; neither coefficient reached the ratio of about 2 between coefficient and standard error needed to reach nominal statistical significance.

So at a superficial level the authors did nothing wrong. These coefficients were not "statistically significant" in the multiple regression. There is a good chance, however, that had they approached this problem in a more sophisticated way they might have found different results. Trying to examine that many variables (11) with so few cases (142) can lead to problems, which the authors did not seem to address.