0
$\begingroup$

I have a model which is based in a theoretical assumption and all variables can be justified in that model. I am applying it to several different clusters of data. The R square for the model is .528 and N = 44. I am including a snip of my SPSS report. It shows that most of the associations of the variables are not statistically significant. Can I say that this model explains the relationship in this case?

enter image description here

$\endgroup$
1

4 Answers 4

2
$\begingroup$

In ANOVA terminology, you would say that the "omnibus" test passed with a type 1 error rate (alpha) below whatever you feel the need to set it to. We draw this conclusion from the F-test, which tests the null hypothesis that all of the variables taken together lead to predictions that are better than a simple naïve prediction at the mean. So in short, yes it is likely that this model "explains" the relationship.

However, since there are clearly several variables that don't contribute to the overall fit of the model, you would now probably be interested in finding out which of the predictors to keep in your model. For example, it appears that "per_pro" is driving most of the fit of your model, so are the others important? If you remove some of the predictors, does this affect the significance of the others? I don't know what your hypothesis or theoretical assumptions are, but I would think the next step is to do some regression diagnostics along with some forward or backward selection procedures to determine if (and why) you should select this model as your final model.

I want to second Wayne's comment, and emphasize that checking for multicollinearity is a vital part of any diagnostic process when using these sorts of models. Depending on your goals, you may want to be able to interpret one or more of your predictors in terms of the influence they have on the dependent variable here. For example, you could interpret the "B" coefficient associated with "per_pro" as meaning approximately "With all other predictors held constant, a 1-unit increase in per_pro corresponds to a 1.052-unit increase in PerMemChng." However, this is not strictly true if per_pro is correlated with one or more of your other predictors, so be sure to check for this.

$\endgroup$
1
  • $\begingroup$ In particular, "If you remove some of the predictors, does this affect the significance of the others?" may indicate multicollinearity. @SteveC have you checked for this? It would not affect your predict-ability, but would throw off significance. $\endgroup$
    – Wayne
    Jul 6, 2016 at 15:27
1
$\begingroup$

I'm not sure that I would think about "explaining" in a binary absolute sense. Rather, I'd think about it in more relative terms.

The model that you have, which contains 7 predictors, is better at explaining the variance in your outcome variable than a model with no predictors (a model where all you know about your $y$ is its mean). That's what you can conclude on the basis of the significant F-test.

What is worth asking now, is whether your model, containing as it does 7 predictors, would be better at explaining the variance in your outcome variable than a model with only one predictor, per_pro (or, does a model with 7 predictors better explain your outcome than a model with 6 variables).

Of course, going forward, you may also want to keep in mind the fact that you have relatively few observations given the number of predictors. The lack of significant coefficients could well be due to a lack of power, rather than a lack of a relationship.

$\endgroup$
1
$\begingroup$

@Gabe is correct that your model significantly "explains" some of the variance in your dependent variable, and that per_pro deserves special attention. Part of your problem comes from the large numbers of predictor variables relative to the number of cases. If the dependent variable "PerMemChng" is continuous, then with 44 cases you should be evaluating no more than 4 or so predictor variables, not 7, in a standard linear regression like this (about 10 cases per predictor). Otherwise you risk overfitting, forming a model that "fits" your present data sample but that would perform poorly on another sample from the same population. If your predictors are correlated with each other, the large number of predictors may also make it difficult to demonstrate a true significance of any one.

I disagree, however, about using stepwise selection to choose a final model, as discussed on this Cross Validated Page among many others. The best approach is often to start with your knowledge of the subject matter and use that to select a number of predictors suitable for the number of cases you have available, or to accumulate more cases if you really want to consider all 7. Alternatively, you could use a method like ridge regression that would allow you to incorporate all 7 variables by "penalizing" the coefficients down to lower magnitudes than their ordinary linear-model coefficients, thus minimizing the risk of overfitting. That often works well with a moderate number of correlated predictors.

For future reference, if you are doing predictions from a properly fit (not overfit) model then you should not worry about whether a particular variable meets some test of significance, as noted for example on this page. Predictions are typically improved by including all appropriate variables in a model.

$\endgroup$
0
$\begingroup$

In machine learning, computer vision, people use every pixel as a feature. Therefore there are millions of features and most features will NOT be significant. (and people do not check significance before using it.)

But jointly with all possible features, the model can have high accuracy. So, if you are focusing on accuracy instead of interpretation, you can use it.

@EdM had a very insightful comment, that by doing such, it is very likely to over fit. The reason these models are still working is because in computer science world there are huge amount of data, for example billions of images in internet, all the models we can afford compute would more likely to under fit.

$\endgroup$
3
  • 1
    $\begingroup$ I do not mind anyone to down vote, but could you please explain the reason? I am very open to discuss academically and would not revenge... $\endgroup$
    – Haitao Du
    Jul 6, 2016 at 15:47
  • 1
    $\begingroup$ Wasn't my vote, but I would guess it had to do with the danger of overfitting in a case like this. You might fit a particular data set with high apparent accuracy, but the result might not generalize to other data sets. $\endgroup$
    – EdM
    Jul 6, 2016 at 15:54
  • $\begingroup$ @EdM Thank you very much for the feedback, I totally agree. The reason these types of model work is that they have huge amount of data, say 1 million images, in that case, it is less likely to over fit. See my answers here welcome for more suggestions. $\endgroup$
    – Haitao Du
    Jul 6, 2016 at 16:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.