Statistical significance refers to the probability that, if, in the population from which this sample were drawn the true effect were 0 (or some hypothesized value) a test statistic as extreme or more extreme than the one gotten in the sample could have occurred.

learn more… | top users | synonyms (2)

0
votes
0answers
9 views

Calculating a significant difference

I have a series of results for groups of students achieving a certain level and I need to highlight where the results for different groups are significantly high or low. For example: ...
1
vote
1answer
27 views

Testing the statistical significance of regression coefficients in a logistic regression

Are only the p-values relevant when testing the regression coefficients of a logistic regression? Does the z-value of a coefficient give any further information about the significance of the ...
0
votes
0answers
22 views

Maximum Likelihood estimator for family of binomial distributions

For the below example, I am considering Heads as a success and Tails as a failure, when I toss a coin. (Ex: The first row in the the below tables says, when I tossed the coin 10 times I got 3 ...
0
votes
0answers
9 views

Estimating the false discovery rate under the F-test statistics

We have food as a factor that can take in 3 values and 2 test scores as dependent variables. Basically, there are $k$ students who consume one of these food types before taking both tests. We want to ...
0
votes
0answers
8 views

Statistics question regarding sample size and test statistic [on hold]

Does Increasing the sample size makes the standard error smaller which in turn makes the test statistic larger (more extreme)?
2
votes
1answer
16 views

Is the significance of difference in slopes equivalent to the significance of the slope of the difference of two series?

Say you have an independent variable, $x$, and two dependent variables $y_1$ and $y_2$. I want to calculate whether these two variables have a significantly different slope. I can do it by calculating ...
0
votes
0answers
11 views

pair-wise comparison for two sets of data where each point contains multiple values

I have two sets of data of the same subjects with 8 time points in each set.The second set is a repeated measure of the same 5 subjects. There are 5 subjects polled per time point. Is it correct to do ...
0
votes
0answers
17 views

One-way ANOVA: Concluding one true differences without any post-hoc tests?

Let's say that I run a one-way ANOVA with 3+ levels and that it comes out significant. Can I then, from this result alone, conclude that at least the two sample means that differs the most in my data ...
1
vote
1answer
20 views

Diebold-Mariano test for predictive accuracy

I am using the Diebold-Mariano test in the forecast package of R to test the predictive accuracy. In particular, I want to ...
0
votes
0answers
3 views

Proof of equivalence between results obtained from alternative event study approaches

Consider two common approaches to an event study in the case of an asset return time series: 1. The event window model We define an estimation window and a distinct event window. We fit a model of ...
0
votes
0answers
16 views

Change point analysis along an environmental gradient

I am looking at environmental data - habitat scores and stream conductivity (water quality measure) and looking to do some change point analysis in a set of 178 observations. As opposed to a more ...
2
votes
1answer
22 views

Adjusted R-squared: number of terms or independent variables?

When applying a multiple linear regression, does the adjusted R-squared value depend on the number of independent variables in the model or the number of terms? Specifically, I'm concerned that adding ...
0
votes
0answers
11 views

Significance test for percentages

I have a test in which a person can choose to be in one of three chambers, and I want to compare two different population of people according to the time they've spent in each location. What is the ...
0
votes
0answers
11 views

What does the output of the sigtest function (from influence.me) mean?

Okay, I've read the documentation here but am still unsure about how to interpret the results of the sigtest function - ...
1
vote
0answers
25 views

Testing hypothesis when each data instance is a vector

The experiment involved studying population body velocity changes during a marathon . Velocity of each person in control group and experiment group were measured at multiple instances during marathon. ...
0
votes
0answers
16 views

To be significant or to be stabile, what's more scientfically important?

Recently I discovered the techniques related to cross validation. Basically you can split up your data in n groups and then run your model on one part of the data and assess prediction reliability on ...
0
votes
0answers
20 views

how to calculate Kolmogorov-Smirnov test for one sample in R [migrated]

I have a matrix in which I would like to calculate Kolmogorov-Smirnov test on each column separately. Let's say my matrix is like below ...
3
votes
2answers
99 views

What does an error in ANOVA indicate?

When I came across ANOVA, the instructor talked about df(Error), ss(Error), etc. What do these error terms indicate? Do error terms differ for two-way ANOVA with dependent and independent variables? ...
0
votes
0answers
36 views

Statistical comparison between two curves

I have two time course experiments of the same study, and I want to determine if the time course curve is statistically significant or not. Let's say I treated a group of people with drug A, measure ...
0
votes
1answer
39 views

Very stupid stats question about significance values

I have a very basic stats about significance values and p-value. Say I perform a test using 0.05 as the significnace level. Using this p < 0.05, I obtain say 20 000 significant points. Is it valid ...
1
vote
1answer
20 views

Statistical Significance of a Subset

I have looked around and haven't been able to find an answer to this question. Seems like an easy question, so maybe it's just a matter of looking in the wrong place. I have a data set representing ...
1
vote
0answers
21 views

Test for significant difference in smoking rate across age groups

I need to compute if there are significant differences in smoking rate across 4 age groups. I can assume a simple random sample from ages 15-55yrs, (made-up details below). I am wondering what type ...
0
votes
0answers
5 views

Treatment outcome and risk factor for failure?

I have to assess outome of a specific surgical treatment and the possible risk factors for failure of the treatment on a retrospective cohort. Size is 50 with long and varied follow up. May I ask the ...
1
vote
0answers
13 views

Understanding Results of ANOVA

I ran a two-factor ANOVA, where I'm looking at the time it takes to type a word. The data was split up by what type of task it took place in (6 tasks) and where the word was (outside/middle). For each ...
0
votes
0answers
5 views

Accuracy results on subsamples

Given a binary classification task, let us suppose that a classifier achieves an accuracy result of $x\%$ on a sample of size $N$. What would be a way to calculate the conditional probability that ...
11
votes
3answers
327 views

Are smaller p-values more convincing?

I've been reading up on p-values, type 1 error rates, significance levels, power calculations, effect sizes and the Fisher vs N-P debate. This has left me feeling a bit overwhelmed. I apologise for ...
0
votes
0answers
14 views

Method or Algorithm to produce near-accurate probability selection

Are there any known algorithms or methods to accurately identify, using random occurrences, the most probable next occurrence(s) as a pre-determined length of number set, by providing an already ...
0
votes
1answer
11 views

Finding the significance of certain questions in a survey

My company performs a Go/No-Go questionnaire to determine whether or not to go after a particular opportunity. In this questionnaire is a series of 14 yes/no questions. We have accumulated a ...
0
votes
0answers
16 views

Evaluating significance for difference of $R^2$

I have a dataset with five variables. The first three are the independent variables, the fourth is the dependent variable and the fifth is a factor variable related to the row (the fifth variable ...
0
votes
0answers
19 views

Comparing 2 insignificant p-values

I've run two t-tests, and the p-value of each is a=0.3 and b=...
2
votes
2answers
47 views

Meaning of significance level and critical values for kolmogorov smirnov goodness of fit test

I'm new to using goodness of fit tests and am encountering what I guess is a beginners misunderstanding of how the tables for critical values and different significance levels have to be read. I have ...
0
votes
0answers
12 views

interaction term not in expected direction

I am conducting a hierarchical regression analysis to test interaction effects. The main variable and the moderator are related to the dependent variable significantly and in expected direction. ...
0
votes
0answers
21 views

predict sales using naive bayes and handle sparse data problem

Problem I am trying to use naive bayes for ranking products in a search application. I would like to predict the sales of a given product given the search keyword and the category. the current formula ...
1
vote
0answers
45 views

Significance in beta regression and glm binomial

While performing betaregression using betareg R package I noticed that the terms in my model are often significant, even with very small sample sizes. I tried the same model using glm with binomial ...
1
vote
0answers
32 views

Understanding forecast horizon for Diebold-Mariano tests

I have a problem understanding the parameter horizon of the function dm.test {forecast} in my particular setting. Background: My goal is to forecast energy consumption for individual households. The ...
2
votes
1answer
27 views

How can permutation test be used for assessing the prediction capability of a model?

I have a set of real labels $(y_1, y_2, ...,y_n)$ and predictions $(\hat{y}_1, \hat{y}_2, ...,\hat{y}_n)$ produced by my model. My supervisor has told me to assess the significance of the predictions ...
0
votes
1answer
28 views

statistical significance help

How would I calculate statistical significance for these test results please? Really appreciate any help! Survey ...
0
votes
0answers
12 views

Significant change through different levels

I am not good with stats so this might seem obvious. I have seven levels in an archaeological site and I would like to test whether there is a significant difference in artefact length between levels. ...
3
votes
2answers
79 views

Compare two binomials with unequal groups of trials

I'm attempting to determine if store B is 'better' than store A. Let's say that the two stores are identical in every way except for the sales assistants. I'd like to determine if one store makes more ...
0
votes
0answers
15 views

How do I determine the minimum sample size needed to evaluate the performance of a binary classifier with a statistically significant result?

I’m trying to determine the minimum sample size(s) needed to evaluate the performance of a binary classifier with a statistically significant result. I'm not trying to compare different classifiers ...
1
vote
0answers
27 views

How to determine the significance of an interaction?

My question is simple: How do you determine the overall significance of an interaction (i.e. the marginal effect of $X$ on $Y$ for different values of $Z$)? But the background is a bit ...
0
votes
0answers
32 views

Significance of logistic regression coefficients

Let's say I fit logistic regression with 2 predictors (x1, x2) and binary outcome (y). When I fit two separate regression y~x1 and y~x2 coefficients for predictors are significantly different from 0, ...
0
votes
1answer
59 views

Do the different levels of treatment in a completely randomized design come from different population?

Suppose in a completely randomized design , a treatment A have four levels. When we check whether the means of four levels are statistically significant ,i.e., $$H_0:\mu_1=\mu_2=\mu_3=\mu_4$$ ...
0
votes
0answers
7 views

FWER is a probability, where does “rate” comes in

I was wondering where does the "rate" in FWER coming from, is this simple the name given to the probability.
9
votes
1answer
136 views

On Fisher's exact test: What test would have been appropriate if the lady hadn't known the number of milk-first cups?

In the famous lady tasting tea experiment by RA Fisher, the lady is informed of how many milk-first/tea-first cups there are (4 for each out of 8 cups). This respects the fixed marginal total ...
4
votes
1answer
50 views

Is there a basic definition of Significant result and Non-significant result?

Is there a basic or widely accepted definition of Significant Result and Non-Significant Result? Seems that every where I look they want to get into problem solving without defining what those terms ...
1
vote
0answers
23 views

Bias from stratified sampling

Due to a lack of significance and the large size of the dataset (which had binomial responses with 20,000 responses out of a sample of 15,000,000) my peer has used random sampling to reduce the amount ...
-1
votes
0answers
12 views

AI battle (or experiment with 2 or 3 element sample space) - what number of experiments to do to get win rate with choosed statistical significance?

I make script that runs Wesnoth game in 1vs1 AI mode several times. Both sides play on mostly symmetrical starting condition. Each game ends with player1 win or ...
0
votes
0answers
16 views

Significance testing for comparing precision, recall and f-measure with no true negatives?

I would be interested in finding out confidence intervals for precision, recall, and f-measures when evaluating a NLP (natural language processing) tagger, and in comparing two or more taggers and ...
3
votes
0answers
67 views

Regularization vs dropping insignificant features

Maybe I did not formulate my question properly. I would like to know how the variable selection differs between regularization and reverse-variable selection based on significance values. In short, I ...