2
$\begingroup$

I have some data where I observe something at a certain frequency each month. Each time I observe, I count a value of 1 if true or 0 if false. The data looks like this:

Date, freq, count_of_true
2021-12-01, 10, 4
2021-11-01, 20, 14
2021-10-01, 50, 30

I want to use the monthly count_of_true as an independent variable to check for correlation with a un-described dependent variable. But, count_of_true is a function of frequency (and other factors).

I could check correlation using a new variable: true_rate = count_of_true / freq. But true_rate does not take into account magnitude.

Is there a known way to create a variable that considers frequency and magnitude at the same time? For example, some popular formula which might use the Z-score of frequency variable as a weight which scales magnitude variable?

Improve this question
$\endgroup$
3
  • $\begingroup$ The questi9on is too abstract. With some context, maybe ... $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jul 18, 2022 at 20:00
  • $\begingroup$ by frequency, you mean the number of measurements? $\endgroup$
    – rep_ho
    Commented Jul 19, 2022 at 13:05
  • $\begingroup$ @rep_ho yes number of measurements $\endgroup$
    – Frank
    Commented Jul 19, 2022 at 13:50

1 Answer 1

Reset to default
1
$\begingroup$

I think one way would assign a Beta(count_of_true, count_of_false). I think that it would work because it get in account both the rate of trues as well the number of samples. And is possible to assess the confidence in those estimates. Another way would be use some kind of smoothing, for example:

https://lazyprogrammer.me/probability-smoothing-for-natural-language-processing/

Improve this answer
$\endgroup$
4
  • $\begingroup$ I'll check out the link and get back. The function beta is really what I am asking for. Is there any well-known beta function? $\endgroup$
    – Frank
    Commented Jul 19, 2022 at 13:52
  • $\begingroup$ The regular Beta distribution: en.wikipedia.org/wiki/Beta_distribution $\endgroup$
    – Allan
    Commented Jul 19, 2022 at 13:54
  • $\begingroup$ It looks like beta is a function of three parameters, 2 shape parameters and an $x \in {0, 1}$. Are you saying count_of_true and count_of_false are the shape parameters and $x$ would be the frequency? $\endgroup$
    – Frank
    Commented Jul 19, 2022 at 14:22
  • $\begingroup$ The count of true and count of false are the shape parameters, and you will get a function of those two. And with this function you can assess the shape, the width and another stuff from this curve. It won't be a single number, it will be a function which you can make inference. $\endgroup$
    – Allan
    Commented Jul 19, 2022 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.