Just some strayed thoughts.
1) You fit a multiple regression to examine the effect of a particular
variable a worker in another department is interested in. The variable
comes back insignificant, but your co-worker says that this is
impossible as it is known to have an effect. What would you do?
Many reasons could have caused this:
- The study is underpowered.
- Even it's properly powered, type II error can still occur.
- A predictor or a set of predictors are collinear with your main outcome, inflating the variance, causing it to be non-significant. Similarly, you might have accidentally included a variable in the causal pathway.
- Your co-worker is wrong. For example, the coworker might have developed a strong impression on another analysis in which proper confounding variables were not included. While your model could have correctly include the confounding variables, and it turned out what your co-worker believes was just an illusion all along.
- Since bias can go both ways, it's also possible that your model is missing some important confounding variables.
- Your major predictor can be interaction with another predictor (e.g. sexes) in such a coincident way that the effects within each sex cancel out each other.
- You're examining a very specific subset of a large population.
- The operationalizaion of variables (or even research designs) can be different between your analysis and the analysis your colleague remembers. Sometimes just simple aggregation can change the performance of a predictor.
- Your have chosen to use a different reference group, which can cause the p-value of the set of dichotomous indicators to shift.
- The studies referenced by your co-worker may not be externally valid (aka not generalizable) to your studied population.
2) You have 1000 variables and 100 observations. You would like to
find the significant variables for a particular response. What would
This is a really broad question. Significant variables for a particular response is up for so many interpretations I'm not sure how to even suggest an answer.
Generally, don't run a regression model on them as is because the number of predictors exceeds the number of cases. I'd perhaps strategize by:
- Clarifying with the questioner on whether the model is for prediction or proof of a concept.
- Implement some content-based selection, e.g. evaluate if the predictor even makes any sense to be included, draft a conceptual framework or causal diagram to aid filtering of the 1000 variables etc.
- Then, include some routine check such as getting rid one of any two variables that are highly collinear, getting rid of variables with profound amount of missing or variables that only contain a constant. Mix in with some operational decision factors such as cost of collecting the variables and burdens on the respondents, etc.
- Suggest some statistical data reduction or variable selection techniques. Emphasize on their pros and cons as well as key assumptions.
- Emphasize on monitoring the type I error rate along the way and make necessary adjustment to avoid spurious findings.
Depends on the job/position associated with the interview, you may want to mention reading the technical manual of the data provider, consulting your peers, performing a literature review, etc. There really isn't any "golden answer" for these two questions.