# Tag Info

37

If you check the references below you'll find quite a bit of variation in the background, though there are some common elements. Those numbers are at least partly based on some comments from Fisher, where he said (while discussing a level of 1/20) It is convenient to take this point as a limit in judging whether a deviation is to be considered ...

32

Hypothesis testing versus parameter estimation Typically, hypotheses are framed in a binary way. I'll put directional hypotheses to one side, as they don't change the issue much. It is common, at least in psychology, to talk about hypotheses such as: the difference between group means is or is not zero; the correlation is or is not zero; the regression ...

19

I have to give a non-answer (same as here): "... surely, God loves the .06 nearly as much as the .05. Can there be any doubt that God views the strength of evidence for or against the null as a fairly continuous function of the magnitude of p?" (p.1277) Rosnow, R. L., & Rosenthal, R. (1989). Statistical procedures and the justification of ...

11

This answer is in two main parts: firstly, using linear interpolation, and secondly, using transformations for more accurate interpolation. The approaches discussed here are suitable for hand calculation when you have limited tables available, but if you're implementing a computer routine to produce p-values, there are much better approaches (if tedious when ...

10

Just to add to the existing answers (which are great, by the way). It is important to be aware that statistical significance is a function of sample size. When you get more and more data, you can find statistically significant differences wherever you look. When the amount of data is huge, even the tiniest effects can lead to statistical significance. This ...

9

It is not one of your control variables that is not significant but rather one level of that variable that is not significantly different from your baseline, as you point out. In this situation, I would not change anything and report coefficients for both levels. Recoding is an option, but unless there is a good reason to do it, I would not. You state ...

8

Statistically significant doesn't imply large $R^2$; with large $n$ even very tiny effects are distinguishable from chance; statistical significance is not practical significance. As for significance, when you only have a single predictor, either the p-value next to the variable (dChgs.nz) or the p-value for the $F$ for the overall regression - they're the ...

8

The long answer is "yes". There are few reasons to remove insignificant predictors and many reasons not to. As far as interpreting them you do so ignoring the $P$-value just as you might interpret other predictors: with confidence intervals for effects over interesting ranges of the predictor.

8

The concept of significance or hypothesis testing is not relevant for a whole population. Hypothesis testing is based on the assumption that you deal with a sample from a (usually) infinite population, and asks the question: what is the probability that we have drawn the sample by chance from a population that fulfills the assumptions of the null hypothesis? ...

8

Note that the Kolmogorov-Smirnov test statistic is very clearly defined in the immediately previous section: $$D_n=\sup_x|F_n(x)−F(x)|\,.$$ The reason they discuss $\sqrt{n}D_n$ in the next section is that the standard deviation of the distribution of $D_n$ goes down as $1/\sqrt n$, while $\sqrt{n}D_n$ converges in distribution as $n\to\infty$. Yes, the ...

8

These are two different phenomena: $t$-statistic Holding all else constant, if $N$ increases the $t$-value must increase as a simple matter of arithmetic. Consider the fraction in the denominator, $\hat\sigma/\sqrt{n}$, if $n$ gets bigger, then $\sqrt n$ will get bigger as well (albeit more slowly), because the square root is a monotonic ...

7

$\alpha$ and $\beta$ are related. I'll try to illustrate the point with a diagnostic test. Let's say that you have a diagnostic test that measures the level of a blood marker. It is known that people having a certain disease have lower levels of this marker compared to healthy people. It is immediately clear that you have to decide a cutoff value, below ...

7

There are two issues here: 1) If you're doing a formal hypothesis test (and if you're going as far as quoting a p-value in my book you already are), what is the formal rejection rule? When comparing test statistics to critical values, the critical value is in the rejection region. While this formality doesn't matter much when everything is continuous, it ...

7

You have gotten several good answers already. There are reasons to keep covariates and reasons to drop covariates. Statistical significance should not be a key factor, in the vast majority of cases. Covariates may be of such substantive importance that they have to be there. The effect size of a covariate may be high, even if it is not significant. The ...

7

You have almost performed what is usually called a power analysis. I say almost, because what you usually measure in a power calculation is not the mean p-value, but rather the probability that, given the sample size and the hypothesised mean difference, you would get a p-value lower than say 0.05. You can make small changes to your calculations in order ...

7

Mostly its that "it's been done that way in the past", but in some domains it is precisely because the authors are not drawing statistical inferences directly from the reported standard errors (even though, for the example paper it might be reasonable to do so). As an example, physics research papers often depict the standard errors related to (estimated) ...

6

Fisher uses p-values as a continuous measure of evidence against a null hypothesis? Perhaps. What convinces you of this? So a p-value of 0.06 would indicate that there is no difference and the null hypothesis is true? Not at all. How did you go from 'continuous measure of evidence against' to 'there is no difference'? In particular, Fisher would ...

6

I believe there is some underlying psychology for the 5%. I have to say I don't remember where I picked this up, but here's the exercise I used to do with every undergrad intro stats class. Imagine a stranger approaches you in a pub and tells you: "I have a biased coin that produces heads more often than tails. Would you like to buy one from me, so that ...

6

The Multiple $R^2$ is the square of the correlation between the response and the fitted values. It tells you how much of the total variance is explained by the model's prediction. The $R^2$ doesn't tell you whether the model is significant or not. Of course: if you want a good prediction model, your aim is to get a high $R^2$. In this case, your model ...

6

Are you interested in being able to say that one of the percentages is greater than the other? In the cases you want to do it, do they always add to 100%? In that case, it's easy - you compare one of the percentages to 50%; if it's bigger than 50% the complementary one is smaller than 50%. If you want to compare two proportions where there are other ...

6

In addition to @gung's answer, I'll try to provide an example of what the anova function actually tests. I hope this enables you to decide what tests are appropriate for the hypotheses you are interested in testing. Let's assume that you have an outcome $y$ and 3 predictor variables: $x_{1}$, $x_{2}$, and $x_{3}$. Now, if your logistic regression model ...

6

If there was a reasonable basis for suspecting your hypothesis might be true before you ran your study; and you ran a good study (e.g., you didn't induce any confounds); and your results were consistent with your hypothesis and statistically significant; then I think you are fine, as far as that goes. However, you shouldn't think that significance is all ...

6

Before reporting this and this and this and this, start by formulating what do you want to learn from you experimental data. The main problem with usual hypothesis tests (these tests we learn at school...) is not the binarity: the main problem is that these are tests for hypotheses which are not hypotheses of interest. See slide 13 here (download the pdf to ...

5

Adding to the good comments and answers so far, here are some reasons to include control variables even if not significant: 1) If you expected a large effect and you get a small one, that is important to know 2) Adding the control variable may affect the relationship between the other independent variables and the dependent variable And here is a reason ...

5

Is that possible? Yes. Stepwise regression pursues to maximize overall (joint) prediction by the variables left in the model while attempting to minimize their number. Because variables usually intercorrelate, their relations are complex and the significance level of the variable in the model after removing some other variables from it can change ...

5

The following article might be helpfull to you, as it describes how to evaluate if the effect of a given explanatory factor is invariant over persons, time, or organizations: Paternoster, R., Brame, R., Mazerolle, P., & Piquero, A. R. (1998). Using the Correct Statistical Test for the Equality of Regression Coefficients. Criminology, 36(4), 859–866. ...

5

While agreeing with @gung 's comment above, it might be possible to point you to some general ideas, please bear in mind that these aren't complete answers. A good book will help. The t-values are another effect size measure (like the coefficient) but they are on a standard scale, so that, according to some people you can compare them across variables, ...

5

A test procedure goes like this: (1) Define the sample space: 1024 outcomes of tossing a coin 10 times (2) State the null hypothesis: A fair coin; i.e. $\mathsf{H}$ & $\mathsf{T}$ equiprobable, tosses independent (3) Define a test statistic: You can use the sum of heads, or the number of runs, or whatever you like (4) Perform the experiment & ...

5

Often these different tests test subtly different hypotheses. For example, many tests that are used to check for normality/Gaussianity of a distribution actually look at very specific deviations from normality (e.g. no skewness) and tests differ with respect to what quantity they exactly look at. So I would start with finding out what the difference is ...

5

One useful insight is that there is really nothing specific about a covariate statistically speaking, see e.g. Help writing covariates into regression formula. Incidentally, it might explain why there is no covariate tag. Consequently, material here and elsewhere about non-significant terms in a linear model are relevant, as are the well known critics of ...

Only top voted, non community-wiki answers of a minimum length are eligible