# Tagged Questions

Statistical significance refers to the probability that, if, in the population from which this sample were drawn the true effect were 0 (or some hypothesized value) a test statistic as extreme or more extreme than the one gotten in the sample could have occurred.

2k views

### How to test the statistical significance for categorical variable in linear regression?

If in a linear regression I have categorical variable... how do I know the stastical signifance of the categorical variable? Let's say the factor $X_1$ has 10 levels... there will be 10 different ...
4k views

### How can I calculate margin of error in a NPS (Net Promoter Score) result?

I'll let Wikipedia explain how NPS is calculated: The Net Promoter Score is obtained by asking customers a single question on a 0 to 10 rating scale, where 10 is "extremely likely" and 0 is ...
3k views

### Do we need a global test before post hoc tests?

I often hear that post hoc tests after an ANOVA can only be used if the ANOVA itself was significant. However, post hoc tests adjust p values to keep the global type I error rate at 5%, don't they? So ...
62k views

### How to interpret and report eta squared / partial eta squared in statistically significant and non-significant analyses?

I have data that has eta squared values and partial eta squared values calculated as a measure of effect size for group mean differences. What is the difference between eta squared and partial eta ...
1k views

### Test a significant difference between two slope values

The data I have are a regression slope value of y~time, a standard error, an n value and a p value, for a particular species in two different areas. I want to check whether the the regression slope ...
2k views

### Regarding p-values, why 1% and 5%? Why not 6% or 10%?

Regarding p-values, I am wondering why 1% and 5% seem to be the gold standard for "statistical significance". Why not other values, like 6% or 10%? Is there a fundamental mathematical reason for ...
637 views

### How do I find values not given in (/interpolate in) statistical tables?

Often people use programs to obtain p-values, but sometimes - for whatever reason - it may be necessary to obtain a critical value from a set of tables. Given a statistical table with a limited ...
3k views

### Cross-correlation significance in R

How do you tell if the correlations at different lags obtained from the cross-correlation (ccf function) of two time series are significant.
459 views

### Should covariates that are not statistically significant be 'kept in' when creating a model?

I have several covariates in my calculation for a model, and not all of them are statistically significant. Should I remove those that are not? This question discusses the phenomenon, but does not ...
1k views

### What are good references containing arguments against null hypothesis significance testing?

In the last few years I've read a number of papers arguing against the use of null hypothesis significance testing in science, but didn't think to keep a persistent list. A colleague recently asked me ...
472 views

### Checking whether accuracy improvement is significant

Suppose I have an algorithm that classifies things into two categories. I can measure the accuracy of the algorithm on say 1000 test things -- suppose 80% of the things are classified correctly. Lets ...
3k views

### Either quadratic or interaction term is significant in isolation, but neither are together

As part of an assignment, I had to fit a model with two predictor variables. I then had to draw a plot of the models' residuals against one of the included predictors and make changes based on that. ...
2k views

### What is the difference between “testing of hypothesis” and “test of significance”?

Is there a difference between the phrases "testing of hypothesis" and "test of significance" or are they the same? After a detailed answer from @Micheal Lew, I have one confusion that nowadays ...
442 views

### How to assess effect of intervention in one state versus another using annual case fatality rate?

I am a beginner in statistics with just basic knowledge. I have these data: cases, deaths and CFR (Case Fatality Rate-deaths per 100 cases) of a disease for 17 years (1994-2010) from 2 neighbouring ...
633 views

### How to test for significant differences in means across four within subject conditions where dependent variable is a proportion?

I have a dataset with four within-subject conditions. Participants gave binary answers to 8 questions in each condition, and these were averaged to form a proportion for each condition. I would like ...
2k views

### Comparing two classifier accuracy results for statistical significance with t-test

I want to compare the accuracy of two classifiers for statistical significance. Both classifiers are run on the same data set. This leads me to believe I should be using a one sample t-test from what ...
7k views

### Threshold for correlation coefficient to indicate statistical significance of a correlation in a correlation matrix

I have computed a correlation matrix of a data set which contains 455 data points, each data point containing 14 characteristics. So the dimension of the correlation matrix is 14 x 14. I was ...
360 views

### Significance tests

I have two sample means from two different samples. I purport that the true population means are identical. $H_{0}$: $\mu_{1}=\mu_{2}$ and $H_{1}$: $\mu_{1}\ne\mu_{2}$ What significance test would I ...
1k views

### Calculating the $p$-value of an $F$- statistic

I am trying to implement an algorithm for calculating $p$-values of $F$-tests and I need this method to be highly precise. It is easy to implement this with $z$- or $t$- tables, however I don't know ...
952 views

### How to test for equality of means for proportions? And two-sided or one-sided?

I have the following problem: two groups A and B. The proportions of married and single are ($p_1$) and ($p_2$) respectively, and the standard deviations ($s_1$) and ($s_2$). The sample consists of ...
2k views

### How can you have a non-significant multiple regression model w/ significant predictors? [duplicate]

Possible Duplicate: Not-significant F but a significant coefficient in multiple linear regression How can a regression be significant yet all predictors be non-significant? Significance of ...
4k views

### Comparing and contrasting, p-values, significance levels and type I error

I was wondering if anybody could give a concise rundown as to the definitions and uses of p-values, significance level and type I error. I understand that p-values are defined as "the probability of ...
1k views

### How do I compare bootstrapped regression slopes?

Let us assume I have two data sets with n observations of data pairs of independent variable x and dependent variable y each. Let us further assume I want to generate a distribution of regression ...
2k views

### Highlighting significant results from non-parametric multiple comparisons on boxplots

I have box plots of 13 groups that I show in one plot. The groups have unbalanced populations and are not normally distributed. I want to show which pairs are statistically similar (i.e., have ...
2k views

### Significance test on the difference of Spearman's correlation coefficient

(Thanks a lot for the quick responses! I did a poor job of asking the question, so let me retry.) I do not know how to find out whether or not the difference between two Spearman's correlations is ...
1k views

### Determining sample size

If I have a population and want to draw samples from there, what are the steps to determine the number of samples to draw from the population? In other words, if I have population N and want to draw k ...
4k views

### Examples of studies using p < 0.001, p < 0.0001 or even lower p-values?

I come from the social sciences, where p < 0.05 is pretty much the norm, with p < 0.1 and p < 0.01 also showing up, but I was wondering: what fields of study, if any, use lower p-values as a ...
108 views

### Calculating when significance will be reached in A/B test ($\chi^2$ test of the difference between two proportions)

I'm a marketer trying to understand the math involved in A/B testing. I've come across a tool that estimates how many more samples are required to reach significance for a test that's already begun. ...
483 views

### Is it possible to have a variable significant in multiple regression but not significant in stepwise regression?

I have run a stepwise regression and found that some of the selected variables are not significant yet in a multiple regression with all variables included in the model those variables were ...
2k views

### Do you reject the null hypothesis when $p < \alpha$ or $p \leq \alpha$?

This is clearly just a matter of definition or convention, and of almost no practical importance. If $\alpha$ is set to its traditional value of 0.05, is a $p$ value of 0.0500000000000... considered ...
208 views

### Multiple comparisons for parameter significance in a sparse high dimensional regression model

I have a regularized linear regression model with a large number of parameters (~100,000) that has been optimized using a sparse fitting algorithm (coordinate descent with early stopping). The model ...
181 views

### What test should I use to determine if a policy change had a statistically significant impact on website registrations?

A client's website was operating under a certain policy for membership sign ups for over a year. At the start of October 2012 the client implemented a new policy for sign ups that was supposed to ...
4k views

### Correcting p values for multiple tests where tests are correlated (genetics)

I have p values from a lot of tests and would like to know whether there is actually something significant after correcting for multiple testing. The complication: my tests are not independent. The ...
5k views

### How to test group differences on a five point variable?

I have a series of observations that fall into bins (or "scores"); that is, the data can be 0, 1, 2, 3 or 4. There are two groups of such data, control and treated. I know the number of individuals ...
1k views

### How to interpret the margin of error in a poll?

Recently the media reported on a political poll that stated that "46% of Republican voters in Mississippi think that interracial marriage should be illegal". One example story (of many around the ...
1k views

### How to assess whether a coin tossed 900 times and comes up heads 490 times is biased?

A coin is tossed 900 times and heads appeared 490 times. Does the result support the hypothesis that the coin is unbiased?
264 views

### Evaluate statistical significance of difference between outcomes of tests

I have 400 patients and I apply 4 tests to all of them. This gives me a number of positive and negative for each test. I want to compare the tests 2 by 2 to find out if there is any statistically ...
394 views

### Comparing smoke and hormones level in two groups of people. Which test?

I'm totally not a pro in statistics so i hope the question is not too simple! I have two groups of patients. One group are smokers, the second are non-smokers. I want to compare the level of some ...
374 views

### Significance test for differences in baseline and remeasurement rates between two groups

I need help in knowing which test to perform with this analysis. I have two groups (lets say group A and B). A study was performed with initial rates of heart failures in group A and B (i.e, ...
114 views

### Randomisation/permutation test for paired vectors in R

I'm not an expert, so forgive me if some of the terminology is a little clumsy. Happy to provide more information where required. I have two vectors of 50 paired numeric values in R. I want to ...
306 views

### Multiple t-tests vs. one-way ANOVA

I'm working on a classification problem and I have a very high F1 baseline of 85%. I have trained three classification models and I want to know which one is the best. How can I do so? I tried two ...
512 views

### Statistical significance of a survey/poll?

Let's pretend I'm conducting a poll/survey. It's a simple yes/no poll (i.e. everyone only gives 1 of 2 answers). I have asked N people so far, and X of them have said "yes". I would like to stop ...
43 views

### Chi-square analysis

I am trying to determine association between my independent variables (categorical, nominal) and dependent variables (categorical, nominal). What other statistical analysis I can use aside from ...
4k views

### How can one do an MCMC hypothesis test on a mixed effect regression model with random slopes?

The library languageR provides a method (pvals.fnc) to do MCMC significance testing of the fixed effects in a mixed effect regression model fit using lmer. However, pvals.fnc gives an error when the ...
2k views

### Why is “statistically significant” not enough?

I have completed my data analysis and got "statistically significant results" which is consistent with my hypothesis. However, a student in statistics told me this is a premature conclusion. Why? Is ...
786 views

### Safely determining sample size for A/B testing

I am a software engineer looking to build an A/B testing tool. I don't have a solid stats background but have been doing quite a bit of reading over the last few days. I am following the methodology ...
1k views

### What is a good introduction to statistical hypothesis testing for computer scientists?

I was recently exposed to some statistical hypothesis testing methods (e.g. Friedman test) at work, and I would like to increase my knowledge on the topic. Can you suggest a good introduction to ...
745 views

### Sequential hypothesis testing in basic science

I'm a pharmacologist and, in my experience, almost all papers in basic biomedical research use Student's t-test (either to support inference or to conform to expectations...). A couple of years ago it ...