4
$\begingroup$

1) You fit a multiple regression to examine the effect of a particular variable a worker in another department is interested in. The variable comes back insignificant, but your co-worker says that this is impossible as it is known to have an effect. What would you do?

2) You have 1000 variables and 100 observations. You would like to find the significant variables for a particular response. What would you do?

$\endgroup$
  • 2
    $\begingroup$ Could you delete this question and re-post it as two questions? $\endgroup$ – Peter Flom - Reinstate Monica Nov 13 '13 at 15:40
  • 3
    $\begingroup$ I do not want (nor do I intend) to chide, but I would like to encourage votes on questions that strike users as interesting enough to answer. Currently this question has five answers but no votes. Voting helps convey interest and helps other users sort content. Cheers. $\endgroup$ – cardinal Nov 13 '13 at 20:01
  • 1
    $\begingroup$ @cardinal - I think it's a good point to make, and wouldn't mind at all if it were made more often. If five people are moved to answer it should be attracting some kind of voting response (whether up or down, but the fact that it gets so many answers at all at least hints that some upvotes are probably in order). If there's a problem with the question that's holding people back from upvoting it, they should consider whether they could improve it with an edit. $\endgroup$ – Glen_b -Reinstate Monica Nov 13 '13 at 23:37
  • $\begingroup$ On (1) it is clearly not impossible to fail to reject the null, even when an effect is definitely present; not achieving significance is not the same as "there's no effect". If there's a priori knowledge about the presence, size or even sign of an effect, we should be using that knowledge. On (2) good options are very limited (I'd be inclined to say there are no good options), but some kind of dimension reduction or regularization would be required; any outside knowledge will be vital. $\endgroup$ – Glen_b -Reinstate Monica Nov 13 '13 at 23:43
2
$\begingroup$

Just some strayed thoughts.

1) You fit a multiple regression to examine the effect of a particular variable a worker in another department is interested in. The variable comes back insignificant, but your co-worker says that this is impossible as it is known to have an effect. What would you do?

Many reasons could have caused this:

  1. The study is underpowered.
  2. Even it's properly powered, type II error can still occur.
  3. A predictor or a set of predictors are collinear with your main outcome, inflating the variance, causing it to be non-significant. Similarly, you might have accidentally included a variable in the causal pathway.
  4. Your co-worker is wrong. For example, the coworker might have developed a strong impression on another analysis in which proper confounding variables were not included. While your model could have correctly include the confounding variables, and it turned out what your co-worker believes was just an illusion all along.
  5. Since bias can go both ways, it's also possible that your model is missing some important confounding variables.
  6. Your major predictor can be interaction with another predictor (e.g. sexes) in such a coincident way that the effects within each sex cancel out each other.
  7. You're examining a very specific subset of a large population.
  8. The operationalizaion of variables (or even research designs) can be different between your analysis and the analysis your colleague remembers. Sometimes just simple aggregation can change the performance of a predictor.
  9. Your have chosen to use a different reference group, which can cause the p-value of the set of dichotomous indicators to shift.
  10. The studies referenced by your co-worker may not be externally valid (aka not generalizable) to your studied population.

2) You have 1000 variables and 100 observations. You would like to find the significant variables for a particular response. What would you do?

This is a really broad question. Significant variables for a particular response is up for so many interpretations I'm not sure how to even suggest an answer.

Generally, don't run a regression model on them as is because the number of predictors exceeds the number of cases. I'd perhaps strategize by:

  1. Clarifying with the questioner on whether the model is for prediction or proof of a concept.
  2. Implement some content-based selection, e.g. evaluate if the predictor even makes any sense to be included, draft a conceptual framework or causal diagram to aid filtering of the 1000 variables etc.
  3. Then, include some routine check such as getting rid one of any two variables that are highly collinear, getting rid of variables with profound amount of missing or variables that only contain a constant. Mix in with some operational decision factors such as cost of collecting the variables and burdens on the respondents, etc.
  4. Suggest some statistical data reduction or variable selection techniques. Emphasize on their pros and cons as well as key assumptions.
  5. Emphasize on monitoring the type I error rate along the way and make necessary adjustment to avoid spurious findings.

Depends on the job/position associated with the interview, you may want to mention reading the technical manual of the data provider, consulting your peers, performing a literature review, etc. There really isn't any "golden answer" for these two questions.

$\endgroup$
1
$\begingroup$

(1) This doesn't quite make sense. "What would you do" is very general.

Often variables that are known to be significant/related to the outcome are included in a regression model even if they are found not to be significant (to increase confidence in that you are providing unbiased estimates for the other coefficients). So one answer would be you keep the variable in the model.

But if you're interested in the estimate of the variables of interest. That is a domain knowledge questions, rather than a statistical questions.

(2) Shrinkage methods = class of approach you might consider with p>>n. Ridge/Lasso most referenced.

$\endgroup$
1
$\begingroup$

1) This is a multiple regression. Presumably there are other variables. It may be the case that the specific variable in question has an effect on the response, but in the presence of the other variables its effect is not significant. I'd use partial least squares pairing the variable in question with the other variables to determine which other variables are "explaining away" the effect of the target variable and report my findings back to the contentious party, in addition to teaching them a little bit about multiple regression.

2) Investigate variable selection and dimensionality reduction methods. I'd start with lasso and PCA. If there's a large class imbalance in our small sample, I'd consider oversampling the minority case.

$\endgroup$
0
$\begingroup$

1) Why would you test it in the first place if you already knew that there was a relationship? Therefore, there must be something unique about what you are investigating. It could be that there is another predictor variable in the model that is making the effect of the predictor of interest non-significant. It could simply be Type II error.

What would I do? Report everything accurately and honestly, and discuss what may have happened.

2) I'd have to know more info. It sounds like the person is trying to data snoop and increase type I error rate.

$\endgroup$
-2
$\begingroup$

1) If you mean that the co-worker knows the variable has a large (or important) effect on the response and yet your analysis did not detect this important effect then you need to first ensure that your analysis methods are not in error. If the methods are sound, then you should ask and have answered why the data you were given is not in harmony with your co-worker's belief structure.

2) Multinomial Regression

$\endgroup$
  • 2
    $\begingroup$ How does multinomial regression help in this scenario? $\endgroup$ – assumednormal Nov 13 '13 at 20:54
  • $\begingroup$ @Max, after identifying and eliminating those variables that are not contributory to the response a multinomial regression can be performed as shown here. Then the coefficients can be examined to identify the most significant variables, thus providing an answer to the interviewer's question. $\endgroup$ – Steve Nov 13 '13 at 21:11
  • $\begingroup$ @Max, if my explanation and link is satisfactory to you can you kindly retract your down vote ? $\endgroup$ – Steve Nov 13 '13 at 22:36
  • 2
    $\begingroup$ @Steve it's still not remotely clear why you are suggesting multinomial regression and your link to wikipedia doesn't help to explain your reasoning. Nowhere in the problem set-up was it suggested that the response variable is categorical. Nor does multinomial regression have any inherent variable selection properties. Your answer to number 1 ("you should ask and have answered why the data you were given is not in harmony with your coworkers beliefs") isn't really a satisfactory response either, since this was essentially the question you were prompted with. $\endgroup$ – David Marx Nov 14 '13 at 0:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.