2
$\begingroup$

I'm new in statistics and would be so grateful if you give me some insights:

I have two big tables - results of work Model_1 and Model_2. I created and calculated statistics - something like Precision = True Positives/(True Positives + False Positives) and suppose that the first model is better that the second one because the proportion in Model_1 is better that in Model_2.

How to prove it statistically? I have an idea to use bootstrap (or just random samples from my two initial samples), calculate the metric there over and over again and see if the distribution of that metric is normal. Is it fine?

Then, if normality is proved it possible to use T-Test (or Mann-Whitney if not) not for mean but for a custom proportion? Theoretically yes, but reading the manual for standard functions:

scipy.stats.ttest_ind(...)[source] Calculate the T-test for the means of two independent samples of scores.

seems like it is all about MEANs.

Improve this question
$\endgroup$
4
  • $\begingroup$ If you're able to use SciPy, it has both the T-Test and Mann-Whitney. docs.scipy.org/doc/scipy/reference/generated/… and docs.scipy.org/doc/scipy/reference/generated/… $\endgroup$
    – Matthew0898
    Commented Apr 11, 2021 at 15:16
  • $\begingroup$ yes, thanks, but I guess before using T-Test or MW, I need to check the normality of distribution. Of which distribution? I can, as I wrote, generate more samples from my initial two (they are tables of 200K and 60K rows), so I can take randomly 1000 rows from each of them, calculate the metric and see if the distribution is normal. That is all that comes in my head - and I wonder is it correct. And then, T-Test, is it applied to a custom metric, NOT the mean-value? $\endgroup$
    – TRP
    Commented Apr 11, 2021 at 15:23
  • $\begingroup$ My apologies, I tried to edit the comment before migration. Perhaps scipy.stats.ttest_ind_from_stats is what you're looking for. $\endgroup$
    – Matthew0898
    Commented Apr 11, 2021 at 15:26
  • $\begingroup$ Thanks! Yes, that is is really looks like my case. The last thing - how could I get the std. deviations? I have a ratio in first sample, the ratio in second sample. That's it. $\endgroup$
    – TRP
    Commented Apr 11, 2021 at 15:41

2 Answers 2

Reset to default
2
$\begingroup$

This sounds like you want to use the McNemar test which is intended for paired nominal (boolean) data.

There seems to be a Python implementation in the statsmodel library. See also this question.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi! Thanks, the article would be very useful BUT - my models were tested on different samples. And I can't create a contingency table =( $\endgroup$
    – TRP
    Commented Apr 11, 2021 at 15:31
2
$\begingroup$

What you need is probably a 2-sample T test. In python you can use ttest_ind_from_stats from scipy.stats

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.