1
$\begingroup$

I have a dataset consisting of 1000 samples (rows). Each sample consists of 300 categorical data points (columns). The process of generating these data points has a known probability of generating an erroneous result, $pErr$.

Let $pErr = 0.1$ for this example. As each of the 300 columns consists of 1000 categorical data points, the number of data points, which can be considered as errors in each column, is $1000 * pErr = 100$. Consider this value $Expected$.

I need to know the lowest number of errors per column, which is considered statistically significant. I run $\chi^2$ tests, starting with $Observed = Expected + 1$, and increasing $Observed$ by $1$, until the test statistic is greater than the $\chi^2$ value, for one degree of freedom. In this example, an $Observed$ value of 119 is statistically significant: test statistic = 4.01 > $\chi^2$ = 3.84.

I then examine each of the 300 columns for categorical data points occurring less than 119 times, and consider them to be errors.

Question: Do I need to apply a multiple comparisons correction to the analysis? If so, where?

Edit: Each data point (each column for each row) is independent. I am not interested in any overall significance or testing. I am only interested in each column as an entity. The number of errors for each column are considered separately from any other column.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.