Can we rank P-values from chi-square tests by magnitude (similar to correlation coefficient)

Question

A, B, C are categorical variables. A is a dependent variable against which I would like to find if B and C have any relationship.

Suppose P-value from a Chi-square test between categorical attribute A and B comes out to be 4%, and that between A and C comes out to be 1%.

• Is it correct to say C has higher impact on A than B?
• Should P-value be treated as binary (either significant as per predefined significance level or not significant)?
• Do we know the "direction" of impact from P-value (e.g. the way we can interpret the correlation coefficient, ~1 -> higher positive correlation &
  ~-1 higher negative correlation)?
• Consequently, can we rank a list of categorical variables based on the P- values with the dependent variables?

Kindly also suggest me if there are alternative ways of achieving the above objectives.

Please let me know if I need to be more clear in the question.

Answer 1

The answer to the questions in bullet points is no. And there is another approach, you need probably to fit some model. You didn't give much detail or context, so only: look into logistic or loglinear models, maybe.

Then your bulleted points:

• No, p-value does not measure strengt of effect, only strength of
  evidence. You need some measure of effect size.
• No, do not treat as binary. The numerical value keeps more
  information.
• No, p-value do not indicate direction of effect, in any way. You
  need some measure of effect size.
• No.

If you want more concrete proposals, we need much more context.

Answer 2

Is it correct to say C has higher impact on A than B?

No. The p-value is a measure of the strength of evidence against the null hypothesis (statistical independence) in favour of the alternative hypothesis (statistical dependence). It is not a measure of the strength of the dependence itself. This distinction is often captured in the contrasting of "statistical significance versus practical significance".

Should P-value be treated as binary (either significant as per predefined significance level or not significant)?

Not unless you have to. The p-value captures more information than its reduction to the binary categories of "significant" or "non-significant", so it is best to keep and use the original p-value if possible. In binary decision problems it is necessary to reduce things to a binary decision, so in this case the p-value is indeed treated as a binary, but even here it is useful to retain the value as contextual information. Here I recommend you read Wasserstein and Lazar (2016) (the ASA statement on p-values); this statement discusses the limitations of p-values and cautions against their reduction to binary classification for "significance".

Do we know the "direction" of impact from p-value (e.g. the way we can interpret the correlation coefficient, ~1 -> higher positive correlation & ~-1 higher negative correlation)?

No. The test-statistic in the chi-squared test is based on the sum-of-squared deviations from expected value over categories, which means that both positive and negative correlation of underlying variables will tend to manifest in larger values of the test-statistic (i.e., be more conducive to the alternative hypothesis). The p-value is computed as a probability based on the null hypothesis when the underlying variables are independent. The p-value does not show the direction of any underlying correlation between the variables. (For this purpose the sample correlation matrix would be the most useful object.)

Consequently, can we rank a list of categorical variables based on the p-values with the dependent variables?

It is unclear what exactly you are proposing here, but it does not sound useful. In general, there are dangers that arise from comparing p-values across different hypothesis tests. P-values taken across different hypothesis tests do not always obey sensible ranking desiderata, so they should be compared only with caution and skepticism.