Skip to main content
added 519 characters in body
Source Link
DWin
  • 7.8k
  • 23
  • 35

The KS test is premised on testing the "sameness" of two independent samples from a continuous distribution (as the help page states). If that is the case then the probability of ties should be astonishingly small (also stated). The test statistic is the maximum distance between the ECDF's of the two samples. The p-value is the probability of seeing a test statistic as high or higher than the one observed if the two samples were drawn from the same distribution. (It is not the "probability that var1 = var2". And furthermore, 1-p_value is NOT the that probability either.) High p-values say you cannot claim statistical support for a difference, but low p-values are not evidence of sameness. Low p-values can occur with low sample sizes (as your example provides) or the presence of interesting but small differences, e.g. superimposed oscillatory disturbances. If you are working with situations with large numbers of ties it suggests you may need to use a test that more closely fits your data situation.

My explanation of why ties were a violation of assumptions was not a claim that ties invalidated the results. The statistical properties of the KS test in practice are relatively resistant or robust to failure of that assumption. The main problem with the KS test as I see is that it is excessively general and as a consequence is under-powered to identify meaningful differences of an interesting nature. The KS test is a very general test and has rather low power for more specific hypotheses.

On the other hand, I also see the KS-test (or the "even more powerful" Anderson Darling or Lillefors(sp?) test) used to test "normality" in situations where such a test is completely unwarranted, such as test for the normality of variables being used as predictors in a regression model before the fit. One might legitimately want to be testing the normality of the residuals since that is what is assumed in the modeling theory. Even then modest departures from normality of the residuals do not generally challenge the validity of the results. Persons would be better of using robust methods to check for important impact of "non-normality" on conclusions about statistical significance.

Perhaps you should consult with a local statistician? It might assist you in defining the statistical question a bit more precisely and therefore have a better chance of identifying a difference if one actually exists. That would be avoidance of a "type II error": failing to support a conclusion of difference when such a difference is present.

The KS test is premised on testing the "sameness" of two independent samples from a continuous distribution (as the help page states). If that is the case then the probability of ties should be astonishingly small (also stated). The test statistic is the maximum distance between the ECDF's of the two samples. The p-value is the probability of seeing a test statistic as high or higher than the one observed if the two samples were drawn from the same distribution. (It is not the "probability that var1 = var2". And furthermore, 1-p_value is NOT the that probability either.) High p-values say you cannot claim statistical support for a difference, but low p-values are not evidence of sameness. Low p-values can occur with low sample sizes (as your example provides) or the presence of interesting but small differences, e.g. superimposed oscillatory disturbances. If you are working with situations with large numbers of ties it suggests you may need to use a test that more closely fits your data situation.

My explanation of why ties were a violation of assumptions was not a claim that ties invalidated the results. The statistical properties of the KS test in practice are relatively resistant or robust to failure of that assumption. The main problem with the KS test as I see is that it is excessively general and as a consequence is under-powered to identify meaningful differences of an interesting nature. The KS test is a very general test and has rather low power for more specific hypotheses.

Perhaps you should consult with a local statistician? It might assist you in defining the statistical question a bit more precisely and therefore have a better chance of identifying a difference if one actually exists. That would be avoidance of a "type II error": failing to support a conclusion of difference when such a difference is present.

The KS test is premised on testing the "sameness" of two independent samples from a continuous distribution (as the help page states). If that is the case then the probability of ties should be astonishingly small (also stated). The test statistic is the maximum distance between the ECDF's of the two samples. The p-value is the probability of seeing a test statistic as high or higher than the one observed if the two samples were drawn from the same distribution. (It is not the "probability that var1 = var2". And furthermore, 1-p_value is NOT the that probability either.) High p-values say you cannot claim statistical support for a difference, but low p-values are not evidence of sameness. Low p-values can occur with low sample sizes (as your example provides) or the presence of interesting but small differences, e.g. superimposed oscillatory disturbances. If you are working with situations with large numbers of ties it suggests you may need to use a test that more closely fits your data situation.

My explanation of why ties were a violation of assumptions was not a claim that ties invalidated the results. The statistical properties of the KS test in practice are relatively resistant or robust to failure of that assumption. The main problem with the KS test as I see is that it is excessively general and as a consequence is under-powered to identify meaningful differences of an interesting nature. The KS test is a very general test and has rather low power for more specific hypotheses.

On the other hand, I also see the KS-test (or the "even more powerful" Anderson Darling or Lillefors(sp?) test) used to test "normality" in situations where such a test is completely unwarranted, such as test for the normality of variables being used as predictors in a regression model before the fit. One might legitimately want to be testing the normality of the residuals since that is what is assumed in the modeling theory. Even then modest departures from normality of the residuals do not generally challenge the validity of the results. Persons would be better of using robust methods to check for important impact of "non-normality" on conclusions about statistical significance.

Perhaps you should consult with a local statistician? It might assist you in defining the statistical question a bit more precisely and therefore have a better chance of identifying a difference if one actually exists. That would be avoidance of a "type II error": failing to support a conclusion of difference when such a difference is present.

added 149 characters in body
Source Link
DWin
  • 7.8k
  • 23
  • 35

The KS test is premised on testing the "sameness" of two independent samples from a continuous distribution (as the help page states). If that is the case then the probability of ties should be astonishingly small (also stated). The test statistic is the maximum distance between the ECDF's of the two samples. The p-value is the probability of seeing a test statistic as high or higher than the one observed if the two samples were drawn from the same distribution. (It is not the "probability that var1 = var2". And furthermore, 1-p_value is NOT the that probability either.) High p-values say you cannot claim statistical support for a difference, but low p-values are not evidence of sameness. Low p-values can occur with low sample sizes (as your example provides) or the presence of interesting but small differences, e.g. superimposed oscillatory disturbances. If you are working with situations with large numbers of ties it suggests you may need to use a test that more closely fits your data situation.

My explanation of why ties were a violation of assumptions was not a claim that ties invalidated the results. The statistical properties of the KS test in practice are relatively resistant or robust to failure of that assumption. The main problem with the KS test as I see is that it is excessively general and as a consequence is under-powered to identify meaningful differences of an interesting nature. The KS test is a very general test and has rather low power for more specific hypotheses.

Perhaps you should consult with a local statistician? It might assist you in defining the statistical question a bit more precisely and therefore have a better chance of identifying a difference if one actually exists. That would be avoidance of a "type II error": failing to support a conclusion of difference when such a difference is present.

The KS test is premised on testing the "sameness" of two independent samples from a continuous distribution. If that is the case then the probability of ties should be astonishingly small. The p-value is the probability of seeing a test statistic as high or higher than the one observed if the two samples were drawn from the same distribution. (It is not the "probability that var1 = var2". And furthermore, 1-p_value is NOT the that probability either.) High p-values say you cannot claim statistical support for a difference, but low p-values are not evidence of sameness. Low p-values can occur with low sample sizes or the presence of interesting but small differences, e.g. superimposed oscillatory disturbances. If you are working with situations with large numbers of ties it suggests you may need to use a test that more closely fits your data situation.

My explanation of why ties were a violation of assumptions was not a claim that ties invalidated the results. The statistical properties of the KS test in practice are relatively resistant or robust to failure of that assumption. The main problem with the KS test as I see is that it is excessively general and as a consequence is under-powered to identify meaningful differences of an interesting nature. The KS test is a very general test and has rather low power for more specific hypotheses.

Perhaps you should consult with a local statistician? It might assist you in defining the statistical question a bit more precisely and therefore have a better chance of identifying a difference if one actually exists. That would be avoidance of a "type II error": failing to support a conclusion of difference when such a difference is present.

The KS test is premised on testing the "sameness" of two independent samples from a continuous distribution (as the help page states). If that is the case then the probability of ties should be astonishingly small (also stated). The test statistic is the maximum distance between the ECDF's of the two samples. The p-value is the probability of seeing a test statistic as high or higher than the one observed if the two samples were drawn from the same distribution. (It is not the "probability that var1 = var2". And furthermore, 1-p_value is NOT the that probability either.) High p-values say you cannot claim statistical support for a difference, but low p-values are not evidence of sameness. Low p-values can occur with low sample sizes (as your example provides) or the presence of interesting but small differences, e.g. superimposed oscillatory disturbances. If you are working with situations with large numbers of ties it suggests you may need to use a test that more closely fits your data situation.

My explanation of why ties were a violation of assumptions was not a claim that ties invalidated the results. The statistical properties of the KS test in practice are relatively resistant or robust to failure of that assumption. The main problem with the KS test as I see is that it is excessively general and as a consequence is under-powered to identify meaningful differences of an interesting nature. The KS test is a very general test and has rather low power for more specific hypotheses.

Perhaps you should consult with a local statistician? It might assist you in defining the statistical question a bit more precisely and therefore have a better chance of identifying a difference if one actually exists. That would be avoidance of a "type II error": failing to support a conclusion of difference when such a difference is present.

added 253 characters in body
Source Link
DWin
  • 7.8k
  • 23
  • 35

The KS test is premised on testing the "sameness" of two independent samples from a continuous distribution. If that is the case then the probability of ties should be astonishingly small. The p-value is the probability of seeing a test statistic as high or higher than the one observed if the two samples were drawn from the same distribution. (It is not the "probability that var1 = var2". And furthermore, 1-p_value is NOT the that probability either.) High p-values say you cannot claim statistical support for a difference, but low p-values are not evidence of sameness. Low p-values can occur with low sample sizes or the presence of interesting but small differences, e.g. superimposed oscillatory disturbances. If you are working with situations with large numbers of ties it suggests you may need to use a test that more closely fits your data situation.

My explanation of why ties were a violation of assumptions was not a claim that ties invalidated the results. The statistical properties of the KS test in practice are relatively resistant or robust to failure of that assumption. The main problem with the KS test as I see is that it is excessively general and as a consequence is under-powered to identify meaningful differences of an interesting nature. The KS test is a very general test and has rather low power for more specific hypotheses.

Perhaps you should consult with a local statistician? It might assist you in defining the statistical question a bit more precisely and therefore have a better chance of identifying a difference if one actually exists. That would be avoidance of a "type II error": failing to support a conclusion of difference when such a difference is present.

The KS test is premised on testing the "sameness" of two independent samples from a continuous distribution. If that is the case then the probability of ties should be astonishingly small. The p-value is the probability of seeing a test statistic as high or higher than the one observed if the two samples were drawn from the same distribution. (It is not the "probability that var1 = var2".) High p-values say you cannot claim statistical support for a difference. If you are working with situations with large numbers of ties it suggests you may need to use a test that more closely fits your data situation.

My explanation of why ties were a violation of assumptions was not a claim that ties invalidated the results. The statistical properties of the KS test in practice are relatively resistant or robust to failure of that assumption. The main problem with the KS test as I see is that it is excessively general and as a consequence is under-powered to identify meaningful differences of an interesting nature. The KS test is a very general test and has rather low power for more specific hypotheses.

Perhaps you should consult with a local statistician? It might assist you in defining the statistical question a bit more precisely and therefore have a better chance of identifying a difference if one actually exists. That would be avoidance of a "type II error": failing to support a conclusion of difference when such a difference is present.

The KS test is premised on testing the "sameness" of two independent samples from a continuous distribution. If that is the case then the probability of ties should be astonishingly small. The p-value is the probability of seeing a test statistic as high or higher than the one observed if the two samples were drawn from the same distribution. (It is not the "probability that var1 = var2". And furthermore, 1-p_value is NOT the that probability either.) High p-values say you cannot claim statistical support for a difference, but low p-values are not evidence of sameness. Low p-values can occur with low sample sizes or the presence of interesting but small differences, e.g. superimposed oscillatory disturbances. If you are working with situations with large numbers of ties it suggests you may need to use a test that more closely fits your data situation.

My explanation of why ties were a violation of assumptions was not a claim that ties invalidated the results. The statistical properties of the KS test in practice are relatively resistant or robust to failure of that assumption. The main problem with the KS test as I see is that it is excessively general and as a consequence is under-powered to identify meaningful differences of an interesting nature. The KS test is a very general test and has rather low power for more specific hypotheses.

Perhaps you should consult with a local statistician? It might assist you in defining the statistical question a bit more precisely and therefore have a better chance of identifying a difference if one actually exists. That would be avoidance of a "type II error": failing to support a conclusion of difference when such a difference is present.

added 706 characters in body
Source Link
DWin
  • 7.8k
  • 23
  • 35
Loading
added 3 characters in body
Source Link
DWin
  • 7.8k
  • 23
  • 35
Loading
Post Migrated Here from stackoverflow.com (revisions)
Source Link
DWin
  • 7.8k
  • 23
  • 35
Loading