Statistical significance refers to the probability that, if, in the population from which this sample were drawn the true effect were 0 (or some hypothesized value) a test statistic as extreme or more extreme than the one gotten in the sample could have occurred.

learn more… | top users | synonyms (2)

0
votes
0answers
8 views

What study would this be?

http://www.npr.org/sections/health-shots/2016/01/27/464552087/teaching-parenting-skills-at-doctor-visits-helps-childrens-behavior According to this report by NPR, what's the study design. Is it a ...
2
votes
1answer
12 views

Why do we use the chi-squared test to test independence of 2 categorical variables?

I sometimes use the chi-squared test to test the independence of categorical variables. I know how to do the test but don't know the underlying concepts and details. So, I was wondering about the ...
2
votes
0answers
8 views

Computations of Durbin-Watson quantiles

For some educational reasons I am currently trying to re-compute Durbin and Watsons (1951, Paper II) critical values of the lower and upper bounds of their Test-Statistic (dL and dH), and also table ...
0
votes
0answers
22 views

Comparing classification accuracies with different numbers of features

I have a classification problem in which I’m trying to assign a label to each of a collection of inputs x_i. I’m comparing two different feature representations of the inputs, trying to determine ...
1
vote
0answers
12 views

Determine if ranks of a group are different

I have some data that the response variable is color of potato chip. It has a large positive skew. There are 7 potato varieties and 8 different locations in this study. I want to rank the potato ...
0
votes
0answers
9 views

Compare accuracy of a new light meter

I'm performing a study where i want to test the accuracy of a new light meter (A) againsta a standard calibrated and validated light meter (B) as a reference. I have measure different light ...
0
votes
0answers
5 views

What is the approach for analyzing split tests (as in AB test) with more than two variations?

Is there a framework for this? First for how to determine required sample sizes, but then how to evaluate outcome (multiple comparisons between variations etc)?
1
vote
1answer
14 views

Equivalent of significance in population data? [duplicate]

I know that significance tests don't make sense on population data. But how then can I figure out if effect sizes are meaningful? Is it purely a judgement call? Background: I have a census for an ...
0
votes
0answers
35 views

Statistics help required [on hold]

I am doing a project in A/B testing.I want statistical approach to evaluate my work. A/B testing is comparing two versions of a web page/ taglines /landing pages to see which one performs better. You ...
1
vote
1answer
62 views

Statistical significance in time series (R)

Stats newb here, I have to determine if two time series are really different instead of being part of the same population with noise in the samples. The data is a comparison between two algorithms ...
0
votes
1answer
14 views

Fundamental question about applying Anova

I have a fundamental question about applying ANOVA. Suppose we are testing 4 drugs which reduce blood pressure. Our experiment contains 4 drugs + 1 Placebo. We give these to 3 types of patients ...
0
votes
0answers
15 views

Calculate statistical significance in natural language processing

I have a task to say whether the difference in performance between two systems is statistically significant. The task is similar to sentiment analysis. I have sentences and I need to classify them ...
1
vote
0answers
17 views

Individual versus group-wise significance in ARDL context

In an ARDL model approach, what is one supposed to do if the F-bound test shows insignificance while some variables have significant long run and short run coefficients (the error correction term is ...
1
vote
0answers
8 views

test significance for frequencies

I have frequency data for six datasets (3 control samples and 3 treated samples) consisting of values for physical stiffness, when I plot the frequencies against stiffness it looks like the treated ...
0
votes
0answers
20 views

Question about confidence interval

I got curious while I was reading about confidence interval again. Let's say I want to find out about where the person's income stands in distribution by getting confidence interval. Assuming a a 20 ...
0
votes
0answers
12 views

How to test statistical significance to prove that a particular technique estimates a variable better than another technique?

Suppose the following is the scenario: I am estimating the measure of a construct. I have Subject Matter Experts (SME) who have manually (through observation) estimated the measures. Suppose I have ...
1
vote
2answers
40 views

Appropriate test for comparison of 50+ groups

I am currently working with a data set containing several hundreds of thousands of instances for which I am trying to find the most appropriate analysis. The goal is to determine whether there are ...
3
votes
1answer
40 views

How do I know if my result is statistically significant?

I wanted to test the effectiveness of 3 different messages I sent to users(respondents) to take a survey. In total, I sent out 6000 messages (3 sets of 2000 users). For message A, I received a ...
0
votes
0answers
3 views

How to test the significance of nucleotide diversities and fixation indices between populations ?

I have four populations with >3Mio Snps and calculated nucleotide diversity as implemented in vcftools in window sizes and then took the mean of it. Now I like to know whether the differences in mean ...
1
vote
0answers
13 views

Controlling for White Matter when looking at a sub-section of White Matter in the Brain - Valid Statistically?

I am interested in other people's opinion on the following matter. I have just come across a brain imaging paper that looks at differences in the volume a sub-section of white matter (SS) in the ...
0
votes
0answers
13 views

Determining minimum required sample size for control (for purposes of measuring lift)

SO in a sales setting where people can either buy or not buy (i.e. binary response), say I have a treatment that I know (or believe strongly) to be effective. I want to give it to as many people as ...
0
votes
0answers
10 views

Relationship between central moments and dependent variable

I was given the task to determine the effects of several statistics on (electricity) derivative prices. Suggestions were variance, skewness, kurtosis to start with. I have a pretty good idea how to ...
0
votes
3answers
69 views

Why do we do hypothesis testing on estimates of linear regression?

I was reading about linear regression and what I understood is that once we minimize OLS equation we get the beta parameters. Its just like solving a normal equation to get the unknowns. Then why do ...
0
votes
0answers
12 views

How do I test the significance of my correlation coefficients for various NONLINEAR regressions?

I have a set of 25 points and I used a TI-84 calculator to calculate the linear, quadratic, logistic, exponential, and power regression models for this data set. I have, likewise, obtained the r^2 ...
0
votes
0answers
19 views

How to test the significant difference of two confusion matrices? [duplicate]

Is there a test for testing the difference between two confusion matrices?
0
votes
1answer
20 views

Test significance of time dummy

I have a logit model $y = \alpha + age + educ + male + year + u$ where y takes the value one if an individual exits unemployment. When calculating the marginal effects of, say, male, after the ...
0
votes
0answers
14 views

How many percent in classification accuracy could be considered as significant

I have 2 classification methods to classify 6 classes. The method 1 achieved the accuracy = A1%, and the method 2 achieved A2%, with A2 > A1. How many percentage point between A2 and A1 could be ...
11
votes
0answers
114 views

Has anyone besides Egon Pearson accessed Gosset's 1904 paper?

Has anyone besides Egon Pearson accessed William Sealy Gosset's 1904 report "The Application of the 'Law of Error' to the Work of the Brewery"? I guess it's Guinness property, but given its historical ...
0
votes
0answers
10 views

Chi squared test on multiple survey responses?

I have completed a questionnaire using a variety of categories (i.e. some questions are yes/no, some are a different categories etc.). I want to find out whether there is a relationship between age ...
1
vote
0answers
13 views

What is way to measure relationship between independent categorical variable on dependent variable (categorical/continuous)? [duplicate]

If both dependent and independent are continuous then we can find correlation and we know which column can be more influential. But if independent variable is categorical, how will i know how much it ...
0
votes
0answers
14 views

significance test for a threshold

I have two samples one of which is significantly larger than the other. I have performed the same operation for both the data sets and I find that the two samples exhibit similar behavior at different ...
1
vote
1answer
48 views

Multiple regression in R results different to simple linear regression? [duplicate]

Hi I am having trouble acquiring the final results for presentation. The results from a multiple regression are different to my results in a simple linear regression. For example, the multiple ...
0
votes
1answer
18 views

What test should I use if:

I'm trying not to have a seizure among the bunch of numbers and data I have. My experiment data consists of three non-equal, non-parametric groups, and I need to compare those groups to state if ...
2
votes
2answers
78 views

Why does the parameter variance change when control variables are added to a regression model?

If I add a control variable to my regression, this changes the variances of the parameter estimates. Why is this the case? Is this because SSE increases and therefore the SSR decreases? So basically ...
0
votes
0answers
5 views

Compare related time-series count data before and after intervention

For a university library we did an intervention to create more free seating spots for students by introducing a way for students to temporarily give up their spot while away. They did this with an ...
1
vote
1answer
19 views

binomial test for two data populations

I have a test that has 39 questions. 20 of them have 18 possible answers and 19 of them have 3 possible answers. I built a QA system that answers 19 of the 20 and 18 of the 19 correctly. I want to ...
0
votes
0answers
15 views

A/B Testing and statistical significance [duplicate]

I often read that when you are running a website conversion a/b test, if you see a 5% significance level before you hit your sample size, you should discount it, and that the more often you compare ...
1
vote
1answer
32 views

Reporting of Heterogeneity in Metaanalysis

The background is a forest plot of a meta-analysis which is reported to have been calculated with a "random effects model". The pooled effect is reported as a standardized mean difference (SMD). ...
8
votes
2answers
118 views

Why is high positive kurtosis problematic for hypothesis tests?

I've heard (sorry cannot provide a link to a text, something I have been told) that a high positive kurtosis of residuals can be problematic for accurate hypothesis tests and confidence intervals (and ...
0
votes
0answers
20 views

What statistical test should I use to compare the binary outcome of two groups (with pre- and post- test)?

I have two groups of people: experiment group and control group. People in each group accepted pre-test and post-test. The outcome of the tests is binary: "good" or "bad". In other words, each people ...
0
votes
0answers
24 views

Which statistical tests are reasonable with this time of day data set?

I'm not really sure which statistical test is appropriate for my data. data: I have behavioral data (being active, resting, eating ...) with a time stamp. Data was taken between 6-18 h and is not ...
0
votes
0answers
75 views

Can I relax the independence assumption for a Chi-Squared test on observational email data?

I'm trying to determine if there is a difference in response rates between email subject lines with the purpose of generating sound hypotheses for A/B testing email campaigns. The data I'm using is ...
1
vote
1answer
38 views

How to determine when the difference between two goodness of fit values is significant?

I have a data series that I'm trying to fit to a model. I'm trying several types of models (exponential, linear, logarithmic). In order to assess which one it fits best, I use a Residual Sum of ...
0
votes
0answers
13 views

Assessing significance / confidence of a crossvalidated performance measure

I have a prediction model $P$ and I use some performance measure $I$ to measure $P$'s accuracy. The distribution of $I$ is unknown (it's a custom metric, which is somehow similar to the precision ...
0
votes
0answers
14 views

Chi squared null-hypothesis test. Long tail

I have the following data. ...
0
votes
0answers
7 views

Significant change over time between categories

I am trying to establish whether change over time is significant at an archaeological site. Basically, there are seven different time periods (lets call it A-G) and in each time period there are six ...
3
votes
1answer
30 views

Clustering and A/B testing

My question is the following: Let's imagine I've defined clusters in my data (different segments of customers) and I run an A/B test. Can I compare the performances of the different clusters on the ...
1
vote
1answer
28 views

Check population randomness in an A/B test?

I am fairly new to A/B testing and I don't really understand some points that my colleagues make. On our website users are randomly assigned to A or B when their session start (by randomly assigned, ...
0
votes
0answers
16 views

Test to test for significant differences in attrition rates

Suppose I have two populations and I want to compare the attrition rates between them to see if there is a significant difference. Attrition rates are tallied up monthly. For each individual in the ...
0
votes
0answers
38 views

If my goal is to show very low correlation, how should I check for statistical significance?

I am working on a research project in social sciences. My null hypothesis is that there is a high correlation between two variables and my alternative hypothesis is that there is very low to no ...