1
$\begingroup$

When refuting two causal models, model 1 has a bigger p-value and an estimated effect closer to the new effect (compared to model 2). Both can't be refuted because they have a p-value above 0.05.

Is it safe to assume that model 1 will arguably outperform model 2 when applied to new data?

Basically, what I want to know is if refuting methods can also be used to measure the performance of a model (e.g. the closer p-value is to 1, the better is the expected performance of the model).

$\endgroup$
2
  • 1
    $\begingroup$ P values tell you nothing about out of sample performance or about causality. Your title seems to hint your question is about causal inference but the question itself mentions p values (statistical inference) and out of sample performance (which makes me think prediction). What exactly are you doing? $\endgroup$ May 7, 2023 at 22:17
  • $\begingroup$ @DemetriPananos I'm using the python module DoWhy for causal inference. This tools has refutation methods built in that present p-values. I just want to know if those refutation methods can be used to measure the performance of a causal model over another. (It could be based on other indicator like the difference between new effect and estimated effect). $\endgroup$
    – Rui Lima
    May 7, 2023 at 22:34

1 Answer 1

0
$\begingroup$

The answer to your question is no, the p-value or the proximity of the estimated effect to the null hypothesis (in this case, I assume it's zero or no effect) are not good metrics for predicting how well a model will perform on new data.

The p-value is a measure of the strength of evidence in support of a statistical hypothesis, not a measure of the prediction accuracy of a model. A smaller p-value suggests that we have stronger evidence to reject the null hypothesis, not that our model will necessarily perform better on unseen data.

In the same vein, the estimated effect size being closer to zero does not imply that a model will perform better on new data. It simply means that, according to the data you used to estimate the effect size, the effect is close to zero.

When you want to assess how well a model is likely to perform on new data, you are concerned with the model's predictive accuracy or generalizability, not the statistical significance of its parameters.

Model comparison based on predictive performance is typically done through cross-validation or using a separate validation dataset. Metrics such as mean squared error (for regression problems) or accuracy, precision, recall, or AUC-ROC (for classification problems) are often used.

In essence, while p-values and effect sizes are useful for understanding the properties of your model and the relationships in your data, they do not directly provide information about the predictive performance of your model.

Moreover, be cautious about the use of a p-value threshold (like 0.05) to determine the validity of a model or effect. The p-value is a continuous measure of evidence, and the choice of a particular threshold like 0.05 is somewhat arbitrary and can lead to dichotomization of results, which is a topic of ongoing debate in the statistics community.

$\endgroup$
2
  • 1
    $\begingroup$ OP is asking about refuting causal models. Can you edit to explain how p-values are or are not helpful for causal modeling? $\endgroup$
    – Sycorax
    May 13, 2023 at 14:01
  • 3
    $\begingroup$ One of several AI content detectors rates this as mainly AI generated. Can you please confirm that this was not mainly sourced from an Ai response. At a very minimum you need to attribute material that is Ai-didact generated. $\endgroup$ May 15, 2023 at 3:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.