Comparison of two test metrics

Question

I'm trying to compare two test metrics (Metric A and Metric B) to determine which one better predicts a delta value, which represents a Euclidean difference. I am unsure how to determining which metric is better at predicting the delta and how to statistically validate this difference. Here's the relevant details and my approach so far. Any suggestions would be greatly appreciated.

Metrics and Delta Values:

• Metrics Range: Both metrics nominally range from 0 to 100, but one metric can produce negative values.
• Delta Values: Always equal to or greater than 0.
• Special Condition: When either metric equals 100, the delta will be 0.

Current Approach:

1. Correlation Analysis:

   • I performed a correlation analysis between Metric A and Metric B, and they correlate highly with each other. (~0.95)

2. Linear Regression:

   • I have performed linear regressions between Metric A and Delta, and Metric B and Delta.
   • I compared the R² values from these regressions to evaluate predictive power.

3. Heteroscedasticity Issue:

   • The residual plots from the regressions show a high degree of heteroscedasticity. This seems like a big concern. I'm wondering how to deal with this.
   • I think the heteroscedasticity is expected because, as the metric values decrease, the deltas tend to become greater.

Objective:

• Primary Goal: Determine which metric (Metric A or Metric B) better predicts the delta value.
• Statistical Significance: Check if the difference in predictive power between the two metrics is statistically significant.

Proposed Solution:

• Likelihood Ratio Test:
  • I have read that calculating the Likelihood Ratio Test Statistic and its p-value could be useful for this comparison.
  • This approach might help in determining the statistical significance of the differences in predictive power.

Likelihood Ratio Test Statistic: 50.555228536421964
P-value: 1.1585811876258776e-12

Answer 1

You need to precisely define "better". Once you do this, the solution is probably simple.

If "better" means "more correlated with delta" then do the correlations.

If "better" means "more likely to be within a certain distance of delta" then you can find the differences (or, perhaps, ratios) and find out.

If you have some other meaning of "better" then state it and maybe someone here can help (or maybe it will be just as obvious as the above cases). For some of these, there will be tests of the difference that have analytic solutions, for others, you might have to simulate to get the p-value. However, since the two metrics correlate very highly with each other, you will (probably) have a small difference and thus need a large sample to find that a difference is significant.