Hypothesis testing assesses whether data support a given hypothesis rather than being an effect of random fluctuations or some other process described by an alternative hypothesis.
41
votes
8answers
9k views
Is normality testing 'essentially useless'?
A former colleague once argued to me as follows: "we usually apply normality tests to the results of processes that, under the null, generate random variables that are only asymptotically or nearly ...
8
votes
3answers
8k views
Significance of coefficients in linear regression: significant t-test vs non-significant F-statistic [duplicate]
I'm fitting a multiple linear regression model between 4 categorical variables (with 4 levels each) and a numerical output. My dataset has 43 observations.
R gives me the following p-values from the ...
41
votes
9answers
98k views
What is the meaning of p values and t values in statistical tests?
After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It ...
10
votes
3answers
911 views
What tests do I use to confirm that residuals are normally distributed?
I have some data which looks from plotting a graph of residuals vs time almost normal but I want to be sure. How can I test for normality of error residuals?
4
votes
1answer
2k views
How to interpret type I (sequential) ANOVA and MANOVA?
My primary question is how to interpret the output (coefficients, F, P) when conducting a Type I (sequential) ANOVA? My specific research problem is a bit more complex, so I will break my example into ...
13
votes
4answers
2k views
Statistical inference when the sample “is” the population
Imagine you have to do reporting on the numbers of candidates who yearly take a given test. It seems rather difficult to infer the observed % of success, for instance, on a wider population due to the ...
3
votes
1answer
3k views
How can I calculate margin of error in a NPS (Net Promoter Score) result?
I'll let Wikipedia explain how NPS is calculated:
The Net Promoter Score is obtained by asking customers a single
question on a 0 to 10 rating scale, where 10 is "extremely likely" and
0 is ...
5
votes
1answer
5k views
Why Levene test of equality of variances rather than F ratio?
SPSS uses the Levene test to evaluate homogeneity of variances in the independent group t-test procedure.
Why is the Levene test better than a simple F ratio of the ratio of the variances of the ...
32
votes
2answers
2k views
What are examples where a “naive bootstrap” fails?
Suppose I have a set of sample data from an unknown or complex distribution, and I want to perform some inference on a statistic $T$ of the data. My default inclination is to just generate a bunch of ...
26
votes
3answers
2k views
F and t statistics in a regression
In a multiple linear regression, why is it possible to have a highly significant F statistic (p<.001) but have very high p-values on all the regressor's t tests?
In my model, there are 10 ...
11
votes
7answers
2k views
How to test hypothesis of no group differences?
Imagine you have a study with two groups (e.g., males and females) looking at a numeric dependent variable (e.g., intelligence test scores) and you have the hypothesis that there are no group ...
16
votes
6answers
604 views
Effect size as the hypothesis for significance testing
Today, at the Cross Validated Journal Club (why weren't you there?), @mbq asked:
Do you think we (modern data scientists) know what significance means? And how it relates to our confidence in our ...
6
votes
8answers
477 views
Testing random variate generation algorithms
Which methods are used for testing random variate generation algorithms?
9
votes
5answers
4k views
Appropriate normality tests for small samples
Title says enough... so far, I've been using Shapiro-Wilk's statistic in order to test normality assumptions in small samples. Could you, please, recommend another technique?
3
votes
5answers
3k views
Can chi square be used to compare proportions?
I've read that the chi square test is useful to see if a sample is significantly different from a set of expected values.
For example, here is a table of results of a survey regarding people's ...
16
votes
6answers
1k views
How can I test the fairness of a d20?
How can I test the fairness of a twenty sided die (d20)? Obviously I would be comparing the distribution of values against a uniform distribution. I vaguely remember using a Chi-square test in ...
13
votes
2answers
10k views
Chi-Squared always a one-sided test?
A published article contains these 2 sentences:
"Moreover, misreporting may be caused by the application of incorrect rules or by a lack of knowledge of the statistical test. For example, the total ...
10
votes
2answers
1k views
What test can I use to compare slopes from two or more regression models?
I would like to test the difference in response of two variables to one predictor. Here is a minimal reproducible example.
...
6
votes
3answers
844 views
When to use Fisher and Neyman-Pearson framework?
I've been reading a lot lately about the differences between Fisher's method of hypothesis testing and the Neyman-Pearson school of thought.
My question is, ignoring philosophical objections for a ...
4
votes
1answer
1k views
How do I compare correlation coefficients of the same variables across different groups?
I am testing the correlation between Variable A and Variable B by gender and marital status.
So I have the coefficients for males and females and single, married and other.
I want to compare the ...
33
votes
2answers
1k views
What are good references containing arguments against null hypothesis significance testing?
In the last few years I've read a number of papers arguing against the use of null hypothesis significance testing in science, but didn't think to keep a persistent list. A colleague recently asked me ...
39
votes
10answers
6k views
Are large data sets inappropriate for hypothesis testing?
In a recent article of Amstat News, the authors (Mark van der Laan and Sherri Rose) stated that "We know that for large enough sample sizes, every study—including ones in which the null hypothesis of ...
18
votes
3answers
674 views
Justification of one-tailed hypothesis testing
I understand two-tailed hypothesis testing. You have $H_0 : \theta = \theta_0$ (vs. $H_1 = \neg H_0 : \theta \ne \theta_0$). The p-value is the probability that $\theta$ generates data at least as ...
10
votes
3answers
374 views
Testing data against a known distribution
I asked a question similar to this a while ago, and the general answer was "your question is too vague". So let me try again with a little more detail...
I have written a program which generates ...
16
votes
5answers
730 views
Test for finite variance?
Is it possible to test for finiteness (or existence) of the variance of a random variable given a sample? As a null, either {the variance exists and is finite} or {the variance does not exist/is ...
13
votes
2answers
1k views
Testing significance of peaks in spectral density
We sometimes use spectral density plot to analyze periodicity in time series. Normally we analyze the plot by visual inspection and then try to draw a conclusion about the periodicity. But has the ...
6
votes
1answer
2k views
On univariate outlier tests (or: Dixon Q versus Grubbs)
In (most of) the analytical chemistry literature, the standard test for detecting outliers in univariate data (e.g. a sequence of measurements of some parameter) is Dixon's Q test. Invariably, all the ...
11
votes
4answers
949 views
What to do when the means of two samples are significantly different but the difference seems too small to matter
I have two samples ($n \approx 70$ in both cases). The means differ by about twice the pooled std. dev. The resulting $T$ value is approximately 10. Whilst it's great to know that I have conclusively ...
8
votes
1answer
3k views
Is Kolmogorov-Smirnov test valid with discrete distributions?
I'm comparing a sample and checking whether it distributes as some, discrete, distribution. However, I'm not enterily sure that Kolmogorov-Smirnov applies. Wikipedia seems to imply it does not. If it ...
4
votes
2answers
2k views
How to test whether a sample of data fits the family of Gamma distribution?
I have a sample of data which was generated from a continuous random variable X. And from the histogram I draw using R, I guess that maybe the distribution of X obeys a certain Gamma distribution. But ...
1
vote
1answer
438 views
Testing for moderation with continuous vs. categorical moderators
I am testing an interaction effect where X and Y are continuous variable and M (Moderator) is a categorical variable (effects coding +1, -1).
I have no clue about how to do a post-hoc probing of ...
5
votes
2answers
932 views
Hypothesis testing and total variation distance vs. Kullback-Leibler divergence
In my research I have run into the following general problem: I have two distributions $P$ and $Q$ over the same domain, and a large (but finite) number of samples from those distributions. Samples ...
0
votes
2answers
1k views
How can you have a non-significant multiple regression model w/ significant predictors? [duplicate]
Possible Duplicate:
Not-significant F but a significant coefficient in multiple linear regression
How can a regression be significant yet all predictors be non-significant?
Significance of ...
20
votes
2answers
948 views
Understanding p-value
I know that there are lots of materials explaining p-value. However the concept is not easy to grasp firmly without further clarification.
Here is the definition of p-value from Wikipedia:
The ...
29
votes
6answers
2k views
Motivation for Kolmogorov distance between distributions
There are many ways to measure how similar two probability distributions are. Among methods which are popular (in different circles) are:
the Kolmogorov distance: the sup-distance between the ...
17
votes
5answers
552 views
Is my weatherman accurate?
A question which bothered me for some time, which I don't know how to address:
Every day, my weatherman gives a percentage chance of rain (let's assume its calculated to 9000 digits and he has never ...
15
votes
7answers
2k views
Is it possible to prove a null hypothesis?
As the question states - Is it possible to prove the null hypothesis? From my (limited) understanding of hypothesis, the answer is no but I can't come up with a rigorous explanation for it. Does the ...
12
votes
7answers
1k views
Which one is the null hypothesis? Conflict between science theory, logic and statistics?
I am having difficulties understanding the underlying logic in setting the null hypothesis. In this answer the obviously generally accepted proposition is stated that the null hypothesis is the ...
9
votes
3answers
2k views
Robust t-test for mean
I am trying to test the null $E[X] = 0$, against the local alternative $E[X] > 0$, for a random variable $X$, subject to mild to medium skew and kurtosis of the random variable. Following ...
7
votes
4answers
770 views
How to specify the null hypothesis in hypothesis testing
What is a good rule of thumb for how to choose the question for the null hypothesis.
For instance, if I want to check if the hypothesis B is true, should I use B as the null, B as the alternative ...
16
votes
4answers
5k views
Correct formatting (capitalization, italicization, hyphenation) of “p-value”?
I realize this is pedantic and trite, but as a researcher in a field outside of statistics, with limited formal education in statistics, I always wonder if I'm writing "p-value" correctly. ...
7
votes
2answers
9k views
What is the NULL hypothesis for interaction in a two-way ANOVA?
Let's say we have two factors (A and B), each with two levels (A1, A2 and B1, B2) and a response variable (y).
The when performing a two way ANOVA of the type:
...
20
votes
6answers
655 views
What is the connection between credible regions and Bayesian hypothesis tests?
In frequentist statistics, there is a close connection between confidence intervals and tests. Using inference about $\mu$ in the $\rm N(\mu,\sigma^2)$ distribution as an example, the $1-\alpha$ ...
4
votes
3answers
335 views
p-value with multimodal PDF of a test statistic
I have opened a thread about p-value under the title "Understanding p-value" and gotten two answers and some comments. I think my questions in the thread is somewhat diverse and want to clarify my ...
9
votes
3answers
1k views
Relation between confidence interval and testing statistical hypothesis for t-test
It is well known that confidence intervals and testing statistical hypothesis are strongly related. My questions is focused on comparison of means for two groups based on a numerical variable. Let's ...
7
votes
3answers
374 views
Using computer simulations to better understand statistical concepts at the graduate level
Hi I'm taking a graduate course in Statistics and we've been covering Test statistics, and other concepts.
However, I am often able to apply the formulas and develop a sort-of intuition on how stuff ...
5
votes
3answers
2k views
Checking if two Poisson samples have the same mean
This is an elementary question, but I wasn't able to find the answer. I have two measurements: n1 events in time t1 and n2 events in time t2, both produced (say) by Poisson processes with ...
5
votes
6answers
1k views
What ways are there to show two analytical methods are equivalent?
I have two different analytical methods that can measure the concentration of a particular molecule in a matrix (for instance measure the amount of salt in water)
The two methods are different, and ...
3
votes
1answer
3k views
Statistical test for difference between two odds ratios?
I am attempting to compare two diagnostic odds ratios (DORs). I would like to know of a statistical test which will allow me to do this. Please help! Thank you!
1
vote
0answers
101 views
What hypothesis for A is better than B?
I am still confused about Hypothesis testing. How does one set up the null hypothesis $H_0$ and the alternative hypothesis $H_a$? I have read a post here that doesn't give it much credit, as far as ...
