1
$\begingroup$

I am trying to compare different evaluation metrics to assess performance of LLM-based solutions. To do so, I was planning to compute different metrics values (continuous outcome) and compare these values with a human judgement (binary outcome - correct or not correct).

Consider the example:

question = "Who is the author of Harry Potter?"
answer = "J.K. Rowling"
generated_answer = "The author of Harry is Joanne Rowling"

Suppose you have 2 different functions that compute a continuous score:

  1. Precision (word-based)
  2. Cosine similarity between 2 embeddings (answer and generated_answer)

From the 2 functions above, suppose we get these scores:

precision = 1/7 = 0.1428
cosine = 0.8

For that same example, we have a binary value coming from a human judgement that indicates the generated answer is correct.

correctness = 1

Suppose we have this for $n$ examples, it means we have $n$ continuous values for precision and $n$ continuous values for cosine. We also have $n$ binary values for correctness. I was thinking to using correlations (Spearman or Kendall) between each of these metrics and the correctness to determine which one is more aligned with it. However, that means I have to measure a correlation between continuous and binary values, which I am not sure is valid. My goal is to get a sense of which evaluation function is more aligned with the human judgement. Any thoughts?

$\endgroup$

1 Answer 1

0
$\begingroup$

Correlational Approach

If you are just looking for correlations between continuous and binary variables, then this is specifically what the point biserial correlation is for. The point biserial correlation is just a simplification of the normal Pearson correlation. The formula for it is:

$$ r_{pb} = \frac{(\bar{y}_1 - \bar{y}_0)\sqrt{PQ}}{sd_y} $$

where $\bar{y}$ is the mean of the continuous variable for each group, $P$ is the proportion in each group and $Q$ is just $1-P$. From Cohen et al., 2002, we can see how to estimate it. Below is a table from the book (p.30):

enter image description here

We would just plug in the values in the table to our formula like so:

$$ r_{pb} = \frac{(66.0 - 69.5)\sqrt{(.429)(.571)}}{2.45} = -.707 $$

Note that the $sd$ used here is the standard deviation without a Bessel correction, so it uses $n$ rather than $n-1$.

Regression Approach

However, if you believe there may be some relationship between all three variables (two continuous and one binary), and there is some kind of explanatory mechanism behind it, then you may need a regression-based approach, where you simply enter each of these variables in depending on what their functional relationship is.

For example, we may have reason to believe (I don't actually know in your case) that the cosine function values and precision predicts correctness. Then you can fit this as a logistic regression, where the two continuous variables are entered as predictors and correctness is used as the dependent variable:

$$ \text{correctness} = \beta_0 + \beta_1 \text{precision} + \beta_2 \text{cosine} + \epsilon $$

$\endgroup$
4
  • $\begingroup$ Thanks for the answer! I like the idea related with using the regression. I think I'd be tempted to test using separate logistic regression (1 variable) for each metric. That would kind of capture the relationship between the metric score and the log(odds) of being correct. Once I have regressions for all metrics, I was thinking using exp(Beta) to compare their impact on correctness. Assuming here I normalize all the scores. $\endgroup$
    – stecaron
    Commented Nov 29, 2023 at 19:52
  • $\begingroup$ Regarding Point-Biserial correlation, I've seen it, but I was a bit skeptical when I saw the assumptions defined here: resources.nu.edu/statsresources/…) - 1)no outliers, 2)normally distributed and 3)homogeneity in the variance of both groups. $\endgroup$
    – stecaron
    Commented Nov 29, 2023 at 19:54
  • $\begingroup$ Another avenue I was thinking, which may results in loosing some information, but at the same time is quite simple would be to find a threshold for each metric. Then using that threshold, that would gives me sort of accuracy on the capaciy of the metric to separate correct versus incorrect groups. $\endgroup$
    – stecaron
    Commented Nov 29, 2023 at 19:57
  • $\begingroup$ Those assumptions are the same as any linear regression you would use, the importance of which largely depends on the data you have. I don't think testing the relationship with the variables in isolation (one model for each predictor) is the best way. Fitting all of the variables into one regression controls for the influence of the other predictors rather than just the influence of one, thus you get a complete picture of the actual relationships involved. There is also the issue of inflation of false positive errors by running many regressions instead of just one. $\endgroup$ Commented Nov 29, 2023 at 20:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.