I'm using ttest for two populations, I know that we can reject the null hypothesis when the p-value is less that 0.1 (with 90% confidence). but does it make sense to get a pvalue of say .17 and then say the two means are different with 83% confidence?


A p-value is used to assess the strength of the evidence against the Null hypothesis. A smaller p-value corresponds to stronger evidence against the Null hypothesis.

Let me clarify some details regarding Confidence: Confidence is not 1-p-value. Confidence is 1-Alpha, where alpha is a predetermined threshold to determine if a difference is significant.

Let's suppose we choose an Alpha = 0.05. If we perform the test and get a p-value=0.001, we can conclude: "There is sufficient evidence to reject the Null hypothesis". Confidence doesn't really mean anything when interpreting the results of the test, and it certainly does not mean that confidence is 1-0.001=0.999.

If you have a p-value of 0.17, this does not mean you have 83% confidence in the result. It simply means there is a 17% chance of obtaining the sample results assuming the Null hypothesis is true.

Confidence is less involved when discussing hypothesis testing, but more meaningful when discussing a confidence interval. Let me know if this helps answer your question. I'm happy to help further if needed.

  • $\begingroup$ Thanks, so when performing a ttest can I set a confidence level then? $\endgroup$
    – HHH
    Mar 4 '19 at 18:26
  • $\begingroup$ No, I wouldn't call it a confidence level. We don't have a level of confidence in the p-value; the p-value is what it is, with no uncertainty. When performing a hypothesis test, you set the threshold for concluding significance, say at 0.05 or 0.1. If the p-value is less than the threshold, then you reject the Null hypothesis. You cannot make a statement like "I am 95% confident the null hypothesis is false". The validity of the null hypothesis is judged by the p-value primarily. $\endgroup$
    – JLG
    Mar 4 '19 at 21:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.